You would expect an article about cowbells at Modern Farmer to explain how farmers use bells to keep track of cows. And that it does, but that's just the beginning. The cowbell went from purely livestock use to a musical instrument because it evoked the pastoral sound of farmland.
Both Gustav Mahler and Richard Strauss wrote cowbells into new symphonies in 1904. One can imagine the discussion in cultural salons about which composer stole the idea from the other. Both men were familiar with German farming customs and the seasonal "cow parades" through town, and both wanted to create that feeling in their compositions.
Once the cowbell was in the orchestra, it stayed. Over the next hundred years, it has been heard (herd?) in plenty of popular songs, with its biggest moment of glory coming in 2000 on Saturday Night Live, when Christopher Walken demanded "More cowbell!" from Will Ferrell, which became a rallying cry for those who love the sound as well as those who just think it's funny. Read the entire story at Modern Farmer.