I propose that this is because they are the only ones with a conduit (their horn) to channel their powers. I porpose that if an earth pony or a pegasus had some mean of conducting their magic (IE a wand or a staff) that they too could cast spells.
By that logic unicorns should be able to control weather and grow plants, but they can't.
My idea is that unicorns, earth ponies, and pegasi each have an inherent connection with a particular domain of science - physics, biology, and chemistry respectively.
The bulk of unicorn magic seems to revolve around manipulation of physics: gravity, teleportation, and transfiguration. Meanwhile pegasi are the ones dealing with the chemistry of weather manipulation and earth ponies have mastery over biological practices like farming and medicine.
Notice how unicorns don't seem to be able to use magic to heal wounds or cure disease? When Rainbow Dash broke her wings Twilight didn't say "I'll fix that for you! *poof*" She went to a hospital and recieved normal medical care....from a staff off all earth pony nurses. Yeah, the doctor was a unicorn, but it's the nurses, not the doctors that do most of the hands-on care. The doctors are the brains behind it.