Regardless of size.
Meaning a halfling and a half-giant can hit you just as hard as one another.
Suggestion 1: make unarmed damage vary by size.
1 for tiny (Imp, fey)
1d2 for small (halfling, gnome, etc)
1d3 for medium
1d4 for large
Suggestion 2: give large monks a damage dice increase to represent their larger fists. The D&D 3.5 SRD entry for monks shows their damage dice shifting dramatically

Tiny monks should also receive a damage dice decrease, not that I think ANYONE would ever use such.