I honestly don't think ivory would be hard enough to withstand constant grinding. Most cultures would rather use stone for milling.

The drop spindle whorl theory gets the most points. Religious artifact would be next.

Although, if you do put a cord pull through the hole, it might be the decorative cover of some kind of urn that possible stored grain or oil. The size mentioned makes sense.