The term decimate originally comes from 5th century Greek. In hoplite warfare, a loss of 10% was considered a staggering defeat.

I haven't looked it up, but the story of the evil Roman army killing one in ten of their own soldiers sounds a lot like a false etymology (possibly even perpetuated in later antiquity). We don't actually know very much at all about Romans from the 5th century BC... they had no empire at that point... certainly not enough of an army to sacrifice one in ten of them!
The story smacks of later mythologized slander.