50 Lore is really not a negative investment. You get access to many things you would not normally get access too, most builds I see passed around these days include getting 50 Lore to get WOF, because it's just that powerful.
The classes that get even more skill points are fine, Discpline, SpellCraft, Tumble, and lore are basically the four most taken skill points because of the power they provide. Lore even more so for the honestly not large investment it takes to hit 50 Lore.
Unless your specifically doing some niche build like sailing, there isn't a ton of options that you can invest your skill points into that give you the same mileage as putting them into lore. Scrolls don't take up that much inventory, and they weigh nothing, so even for a dex build it's pretty stanard.
Scrolls of Greater Restoration and WoF are considered "Starter PVP staples" because of how commonly used they are. It's not a difficult benchmark to hit.
It's also ridiculous that it's so easy to reach and completely shuts down entire classes as a hard counter due to the most recent changes to sequencers.
Conjurers now literally do not have a way to protect their summons anymore. The ONE option we had to make people have to take 1 additional round to blatantly dismiss our conjurations has been removed, because apparently taking two rounds to completely delete a potent tool was just too long for some people.
My levle 30 Fighter heavily invested in Lore, and it's given her access to protections defenses and things she simply would not have without it.
Investing in Lore isn't a cost, it's a literal investment that just expands your ability to do things, and gives you more tools at your disposal that you would not normally have without it.
It also lets you just absolutely shut down an entire playstyle of characters, that already were struggling for any form of usefulness.