There are some really cool looks that can be achieved with the bare chest appearances for males, unfortunately it's only available for certain armor types. In the interest of giving players more customization for their characters, I think it would be cool to allow this for all armor types.
If the concern is to allow players to be able to easily identify what to expect from potential threats, my counterpoint to that would be that you can already mask what your armor type is using any number of robes.