I watch MLP, I play the fan-made games, I watch the music videos, I listen to the music, I watch analytical videos, I read the fan wikis, I look at the art, I even create some art of my own. I know who is and who isn't considered "horse famous" in various circles.
I consume official MLP media, and a huge chunk of that's fan made MLP stuff. I adore the show and will even write long-winded blurbs about my own thoughts and opinions about the show and the fandom surrounding it.
But I personally do not call myself a brony, (or pegasister) because I do not feel comfortable identifying myself as such.
If other people want to use the word that's fine. Some people genuinely don't know the toxic stigma that surrounds brony culture, or they use it just because it makes things less confusing for themselves and those around them.
But I don't think that other people should force that label onto you just because "you like G4 ponies so of course you're a brony!!! xDDDDDD"
It's basically the same reason I don't call myself an Otaku either, despite my love for anime.
So...if you feel comfortable with the word, go ahead and use it!
But if you're not comfortable, that's okay too!
Your brother sounds really mean though