Should you avoid training legs in order to excel at hand balancing?
As always, it’s really up to you and what your priorities are. Hand balancing is of course an upper body dominated discipline, so carrying extra weight in your legs will not do you any favors. It’s typical for hand balancers who specialize to not perform any kind of leg training apart from flexibility work.

On the other hand, your hand balancing is more dependent on your skill and precision. If you know how to stack your joints and maintain the equilibrium, the weight of your legs should be irrelevant.
Skills like press to handstand(especially from Seated) and planche May be affected more from the change in weight distribution however. That doesn’t mean the skills are unattainable by any means, but it does mean that you might have to work a bit harder and get stronger in order to achieve them.

As for my own practice, there was a period when I did significantly less leg training because hand balancing was my top priority. It wasn’t because getting skinnier legs was going to help my hand balance, it was more about the priorities of where I wanted to spend my time and energy.
In general I do a significant amount of leg training because I’m training disciplines other than handstands that require it. I also find leg training useful for other reasons.

Ultimately, you should do what you enjoy doing. Practicing your handstands is going to have a much more profound impact on your skill progression than not training your legs, so make sure you prioritize the right things.