Whenever this topic comes up, privacy is usually one of the first things that leaps to mind. Google are careful to address this explicitly:
We’re really excited to make Latitude and your location more useful to you, but we definitely understand that your privacy is important. Just as before, Google Location History is entirely opt-in only and your location history is available privately to you and nobody else. Additionally, you may be asked to periodically re-enter your password when opening any Location History page, even if you’re signed in to your Google Account already (just to make sure you’re really you). Of course, you may always delete any or all of your location history in the Manage History tab or disable Location History at any time.