If you have concerns about the city and can still choose, you might want to get a room in the Hilton or Hyatt, as those both have walkways connected to the convention center (I think the Sheraton does too, looking at an overhead picture, but I don't remember seeing it in person). The Hilton's is also fully enclosed, so weather as well as going between the hotel and convention center wouldn't be a concern. For getting around Baltimore, I'm guessing you're likely coming in either at the airport or train station; the light rail system (which I have found to be clean and safe) connects to both (though I don't know what things at the airport are like; I've gone through Penn Station the years I've been) and has several stops right next to the convention center.
If you're worried about just the number of congoers, I haven't found it to be too crowded the years I've been, though of course how crowded is too crowded varies from person to person. There have also been plenty of con staff around, including convention security and medical personnel.
If you're worried about some subset of the fandom, as I gather you may be, well, in my personal experience, the convention seems to make even the drastic rifts among us melt away for the duration, all of us coming together as bronies. That said, there's plenty that goes on I haven't experienced, and while I'd expect it to be at least almost entirely fun and voluntary, I can't personally vouch for everything. I will say I think it's extremely unlikely that you'll be in any actual danger within the convention areas, and if you feel uncomfortable, there are spaces you can get away. There was an incident last year at an offsite gathering, but you can readily avoid those if you're nervous.
Overall, if money and time and such aren't significant problems, I would recommend going. If it goes well for you, you'll have overcome your fear and had a great time that you might otherwise have missed; if it doesn't, then depending on the details you'll either have a better idea what to do next time or have confirmed that big conventions aren't really a good choice for you (and they aren't for some people) and you can direct your resources elsewhere.
Whatever you decide, good luck.