(vimeo link)
If we don't have visual cues to guide us, people tend to walk in circles. Many theories have been put forth for why this is so, but experiments that control for variables such as right-handedness, brain-side dominance, and more strength on one side come up with the same results: we tend to go in circles. Read more about it at NPR. http://www.npr.org/blogs/krulwich/2010/11/03/131050832/a-mystery-why-can-t-we-walk-straight -via Metafilter
There are a lot of forces at work, between muscles and tendons and joints and your footwear and the ground -- and many types of friction opposing those forces at various points, in asymmetrical ways. I suspect even your joints provide different friction, depending on wear and tear. On top of that, there is certainly a physiological factor related to handedness; for instance, cars with manual gearboxes have notoriously asymmetrical settings for the clutch pedal (intended to be operated with the left foot) compared to the brake and acceleration pedals (intended to be operated with the right foot), specifically in order to compensate for this asymmetry in our behavior. And that's without even touching hemisphere dominance or other more subtle factors.
Given all of the above, it would be quite remarkable if we WERE able to walk in a straight line without external assistance.