This is a curious word and does not mean today what it meant to the Ancient Romans who coined the term. Merriam-Webster defines decimate as:

DECIMATE; transitive verb
1: to select by lot and kill every tenth man of
2: to exact a tax of 10 percent from
3a : to reduce drastically especially in number
b : to cause great destruction or harm to

It is 3a and b that is more commonly used today however, it is definition one where the word came from. Specifically the Roman military practice of decimation in which mutinous or cowardly units were sentenced to be decimated. This was never a common punishment, most extreme punishments are only effective if they are both extreme and rare. There are a few instances in recorded sources of a unit being decimated one of which is the Legio III Augusta which was decimated in A.D. 18 after allowing one of its subunits to be annihilated in battle.

The question is, would this punishment really be effective? I think so as the fear of a repeat would tend to keep the soldiers in a mutinous or cowardly unit in line. There is an old saying that goes something like “soldiers should fear their leaders more than the enemy.” A punishment like decimation certainly qualifies as something inspiring fear. If the enemy does not kill them, they can be sure that their own people will and the best way to ensure continued life is to fight as hard as possible.