No, it doesn't sound ridiculous. I know how you feel. I'll be 20 next month and I'm also petite and I look younger than I am, so I have the exact same fear. I rarely go anywhere by myself, actually. I don't really feel safe when I'm alone.
As for feeling safer (for both of us), I completely...