BTW the official rules is that's it's OK
http://www.cbsa-asfc.gc.ca/publications/pub/bsf5082-eng.htmlIdentification documents required
When you enter Canada, a border services officer may ask to see your passport and a valid visa (if you are arriving from a country from which one is required). If you are a United States citizen (U.S.), you do not need a passport to enter Canada; however, you should carry proof of your citizenship such as a birth certificate, a certificate of citizenship or naturalization or a Certificate of Indian Status, as well as a photo ID. If you are a permanent resident of the U.S., you must bring your permanent resident card with you.
All travellers, including U.S. citizens, are encouraged to visit the U.S. Customs and Border Protection's Web site at
www.cbp.gov for information on the U.S. Western Hemisphere Travel Initiative and its traveller requirements to enter or return to the United States.
I can also tell you from experience that's it's OK. However that's was like 2+ years ago so things may have changed