It's safer than say downtown San Francisco, Market Street.
I recommend once there, hold the map so that the bay (water) is to your left, the old fort is behind you, Emanicapation Park to your right, (and all the rest of the stuff uphill) and the U.S. Post office (yellow and white building, with flags and fenced lot behind) almost in front of you.
Quickest route, straight ahead and go on the sidewalk. Now turn towards the water and walk till you get to the main road.
There is a Taxi stand to your right, that's where you can catch a taxi back to the pier.
Turn right and walk along the water front
Stay close to the buildings, (watching for traffic). You will cross a couple small streets (all two lane). The HRC is upstairs in the Palm Passage. Close to the waterfront. You may have to hunt for the stairs up. The gift shop is usually open on two sides and has the HRC T-shirts on the walls.
You could instead go from Emancipation Park, cross in front of the Post Office and walk down Main Street. It is more crowed and there are barkers out for some of the shops.
Enjoy your visit!
Sandy