I say you have nothin' to worry about.
I remember joining during the hiatus between Season 2 and Season 3, catching up about a month before Season 3 aired and watched the episodes as they came out since, and I was worried that THAT
felt too late!
(I even had some guy who knew my pals at the ol' lunch table tell me "Sorry you joined during such a bad time". Bluh. Pessimism.)
I'd say it's never too late to join the community, even if it may feel like it in certain circumstances (i.e. after the show ends, but that's no excuse not to enjoy it and the fandom~