If you're celebrating your birthday while there, wear it.
If you're not, if you just want to do it because it's there to be done, well, wear it if you REALLY want to. But people might very well say "Happy Birthday" and some even do ask when it is, or they'll say "oh it's my b'day too, isn't it great to be a 'whatever'-day baby", etc. If you don't want that, don't wear it.
Even if it IS your birthday, if you don't want that, don't wear it.
I don't love that sort of stuff, so even if I'm there on my b'day I don't tend to wear it. (though I totally did at
Disneyland when it was my actual b'day during the free birthday YOAMD time) DH and I waited to celebrate an anniversary until we were at Disneyland, and I made it two days with the button before I couldn't stand it anymore. DH loved it, and kept wearing his. DH loves wearing his b'day button. He LOVES saying HB to people wearing their b'day buttons. It makes him sad when people get freaked out when he says it; if you don't want people to notice it, don't wear it...that's his view.
Disney says if you're celebrating a birthday, get the b'day button.