Have the same issue! It does have voice recognition, however that will only stop someone from making a phone call to one of your contacts or adding an event to your calendar.
Bigger issue I find is that anyone can cast to your Google Home, meaning they can change songs, higher/lower the volume... anyone know if there is a way of stopping this?