You are more likely to skin your knee than to have a crash landing on top of your head, but still, I think it’s a good idea to always wear a bicycle helmet. It’s really not that annoying and it could save you from brain damage or death!
Helmets made for the USA must meet the US Consumer Product Safety Commission standard, so look for a CPSC sticker on your helmet. Most every helmet from a major manufacturer (Bell, Giro, POC, etc.) will have met this safety standard.