I already feel handicapped enough by playing a non-human spellcaster and losing out on a feat and a skill... Don't kick me while I'm down and prevent me from holding a magic staff and an off-hand item like all other spellcasters!

Also, mechanically changing Magic Staves from Medium to Small probably wouldn't effect anything else, besides being a small buff to small-sized spellcasters.
Food for thought: If small-sized PCs can use a katana in one hand (or both), let's fix magic staves too

Side Note: Don't worry, no one will dual wield magic staves, as they cannot be dual wielded by default, including by larger PCs.