I think these outside areas should be biddable/purchaseable by outside and independent groups that are not associated with a settlement, currently Darrowdeep, Ice hold, Castle Gloom etc require you to bid on behalf of a settlement, but some of these areas are incredibly vastly underutilized, it would probably be better if people could bid on these structures on behalf of a faction, to create a space to be utilized for said faction.
Also obviously of course settlements could outbid said faction or group if they really wanted to, but an option should exist for this.