In terms of the Boreal Keep, imagine a different scenario.
Imagine if the Arcane Tower at Mayfields was as it is now, except that there was no way to climb into it, or to access the internal portals unless you were an access holder or you were in party with an access holder.
You've got a property with many rooms, full of mages, who are unable to be removed or even accessed by outsiders without their approval, who can do functionally whatever they like.
Sure, you can attack them outside of the tower.
But how are you going to spy on them when they ward the tower?
How are you going to break in to gather information on their plans, or interrupt their meetings and events?
Imagine if the Red Wizards took over such a place and decided that slavery, undead, and general misery were the order of the day, and started flexing outwardly to disrupt politics and oppose other groups, working from their unassailable tower.
In short time, folks would be calling for staff to adjust the mechanics, because it's not right that a group can be in a position to take hostile actions without needing to deal with repercussions.
The Boreal Keep is like that for Andunor, regardless of the current tenants actions and intentions.