Regarding Recent Performance - Impending Server Split

OOC General Discussion

Moderators: Forum Moderators, Active DMs

Post Reply
User avatar
Irongron
Server Owner/Creative Lead
Server Owner/Creative Lead
Posts: 4666
Joined: Tue Sep 09, 2014 7:13 pm

Regarding Recent Performance - Impending Server Split

Post by Irongron » Fri Mar 27, 2020 10:27 am

As players may have noticed we have been experiencing increasing instability on the servers, after discussion with Mithreas and Morderon, we have determined the only sure-fire way to really address this is by splitting the servers once again.

We have great hardware, but Arelith is a complex engine running in an old game, and player numbers, are by far the highest factor in performance. While I love seeing numbers of 100+, I need to really consider the enjoyment of players first and foremost, and anything over around 75 players will always cause issues. I don't mind being above these numbers for short periods, but we're increasingly hovering around 100 for much of the day, especially for Cities & Planes.

My thoughts are currently as follows, but may change.

Underdark will once again be placed on its own server.
Cordor, along with Baator, and the Shadow Plane will be on a second.
All ship and island content, including Sencliff, will be moved to Distant Shores
Surface server will continue to include all mainland content, including the new city.

Should it be required, if numbers continue to grow, the nation of Guldorand too, may end up on its own server.

I can't give an exact ETA on this, as splitting servers is a really time-consuming process, both for myself and Morderon. Separating the Underdark will be quite straightforward, but moving the ship system to DS will require some code adjustments to the shipping system which will likely be rather complex.

Further to this I've asked the team to look at merging the portal page and -playerlist function into a single list. Sadly this is due to the fact that the last time the Underdark was on its own server there was a significiant problem with players there using the player list to metagame the presence of surface PCs, and using it to send threatening tells, or to justify searching transition areas for the interlopers. I hope that our playerbase has matured since then, of course, but nevertheless a combined playerlist is likely to everyone's advantage.

I can't say I'm wild about splitting the servers, there are a number of disadvantages, but as I said above, it really is our best possible method to improve performance for all of our players.

CookieMonster
Arelith Supporter
Arelith Supporter
Posts: 488
Joined: Mon Sep 08, 2014 4:41 pm
Location: England

Re: Regarding Recent Performance - Impending Server Split

Post by CookieMonster » Fri Mar 27, 2020 11:29 am

Hopefully this isn't to much of a bother for the Team. But we are of course grateful to any time and effort you put into our continued enjoyment. Thanks guys.

Though, I imagine that a lot of the traffic is as a result of the Corvid 19 Pandemic. Many countires are in complete lockdown. The UK for example as of Tuesday has restricted travel to any location outside of your home to essential movements only.

Stay safe everyone.
DM Spyre wrote:Someone has watched too much Dragonball Super.

User avatar
In Sorrow We Trust
Contributor
Contributor
Posts: 1051
Joined: Tue Nov 19, 2019 7:10 am

Re: Regarding Recent Performance - Impending Server Split

Post by In Sorrow We Trust » Fri Mar 27, 2020 2:18 pm

+1. This is exactly what I was suggesting be done a while back.

Vittoria Veleno
catch me if you can


User avatar
Bunny
Posts: 106
Joined: Mon Sep 02, 2019 3:24 am

Re: Regarding Recent Performance - Impending Server Split

Post by Bunny » Fri Mar 27, 2020 3:35 pm

Great time to add more ships for player ownership

User avatar
Ork
Arelith Gold Supporter
Arelith Gold Supporter
Posts: 2488
Joined: Sun Nov 02, 2014 8:30 pm

Re: Regarding Recent Performance - Impending Server Split

Post by Ork » Fri Mar 27, 2020 3:38 pm

I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.

good man of god
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 175
Joined: Thu Jan 15, 2015 12:26 pm
Location: United Kingdom

Re: Regarding Recent Performance - Impending Server Split

Post by good man of god » Fri Mar 27, 2020 3:44 pm

Ork wrote:
Fri Mar 27, 2020 3:38 pm
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.

Previous:
Tornius Daressin
Matthew Daressin-Gravelle
Asvusha (Blake Lynk)
Sebastian Webster

Current:
Inkwell Thornwhisper


Void
Posts: 1600
Joined: Sat Mar 24, 2018 7:03 pm

Re: Regarding Recent Performance - Impending Server Split

Post by Void » Fri Mar 27, 2020 3:49 pm

Would be nice to have ability to transfer henchment through transitions, given that number of leyline regions increases once again.
It would be also nice if beamdog did a multicore update for the game logic, but that's likely a pipe dream at this point.
Another forum ban, here we go again.

Good Character
Posts: 898
Joined: Mon Feb 10, 2020 11:37 pm

Re: Regarding Recent Performance - Impending Server Split

Post by Good Character » Fri Mar 27, 2020 3:54 pm

Hard but strong decision. How are the server costs going look like now?

Xerah
Posts: 2036
Joined: Wed Sep 23, 2015 5:39 pm

Re: Regarding Recent Performance - Impending Server Split

Post by Xerah » Fri Mar 27, 2020 3:57 pm

Please refrain from requesting new features in this thread. This project is a huge undertaking and it comes off as so dismissive of the time and work involved when you jump right into asking for new things. Thanks
Katernin Bersk, Chancellor of Divination; Kerri Amblecrown, Paladin of Milil; Xull'kacha Auvry'rae, Redcap Fey-pacted; Sadia yr Thuravya el Bhirax, Priestess of Umberlee; Lissa Whitehorn, Archmage of Artifice

User avatar
Bulldog711
Posts: 35
Joined: Tue Mar 10, 2015 10:34 am

Re: Regarding Recent Performance - Impending Server Split

Post by Bulldog711 » Fri Mar 27, 2020 4:07 pm

Just a random thought here for the possible future splits: make Surface/mainland broken up into north and south sections, or something similar. A good line of that transition might be somewhere in Arelith Forest but I do not know the full stretch of content behind the scenes so it might be better shifted elsewhere. It could alleviate the stress on the servers and make it flow easier on the transitions. Just giving my 2 coppers and I'm sure we all greatly appreciate the efforts yall are making to have the game run smoother.

User avatar
Flower Power
Posts: 493
Joined: Sat Nov 16, 2019 8:02 am

Re: Regarding Recent Performance - Impending Server Split

Post by Flower Power » Fri Mar 27, 2020 5:49 pm

I've been saying that throwing the UD off onto its own server (or, at least, onto the Skal server) would improve overall stability for years, sooooo.

+1
what would fred rogers do?

Nekonecro
Posts: 358
Joined: Sun Sep 14, 2014 6:52 pm

Re: Regarding Recent Performance - Impending Server Split

Post by Nekonecro » Fri Mar 27, 2020 7:20 pm

Would like to know how a merged player list will affect things like yoinking and scrying?
Usually one would open the list with P and then send a tell to the person with -scry or -yoink.
Will there be issues if it's all in one list for cross server scry/yoink?

User avatar
Zahlfire
Arelith Platinum Supporter
Arelith Platinum Supporter
Posts: 104
Joined: Fri Jul 13, 2018 1:56 am
Location: Gatlinburg
Contact:

Re: Regarding Recent Performance - Impending Server Split

Post by Zahlfire » Fri Mar 27, 2020 9:11 pm

Ork wrote:
Fri Mar 27, 2020 3:38 pm
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.
The GrumpyCat wrote:
Sat Feb 29, 2020 2:48 am
Perhaps the takeaway is not to go 'Ugh. Lesbians... we should cull them.'
Currently plays: N/A

Who to PM and When

Feel free to PM me with Lore questions!

AstralUniverse
Posts: 2723
Joined: Sun Dec 15, 2019 2:54 pm

Re: Regarding Recent Performance - Impending Server Split

Post by AstralUniverse » Fri Mar 27, 2020 9:35 pm

Ork wrote:
Fri Mar 27, 2020 3:38 pm
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.
1000%
Irongron wrote:
Fri Mar 27, 2020 10:27 am
My thoughts are currently as follows, but may change.

Underdark will once again be placed on its own server.
Cordor, along with Baator, and the Shadow Plane will be on a second.
All ship and island content, including Sencliff, will be moved to Distant Shores
Surface server will continue to include all mainland content, including the new city.
Just wondering, how about switching between Baator + Shadowplane, and the new city, and having two cities in one server and the entire surface content in the other? Would it be balanced in terms of content size and player traffic?
Svrtr wrote:

I've spoken with Kenji and warpriest will be allowed to take elemental avatar so keep this in mind too


User avatar
DM Rex
Posts: 806
Joined: Tue Mar 10, 2020 11:13 pm

Re: Regarding Recent Performance - Impending Server Split

Post by DM Rex » Fri Mar 27, 2020 9:38 pm

As crazy as it is, most people don't hangout in hell. They hangout in the cities.
Hence why it's presently presented as the proposed split above.

The two largest being still - Andunor and Cordor.

User avatar
LichBait
Posts: 422
Joined: Thu Jul 07, 2016 9:50 pm
Location: US EST Timezone

Re: Regarding Recent Performance - Impending Server Split

Post by LichBait » Fri Mar 27, 2020 9:39 pm

<3 To this. Thanks for all you do.

Current
Ayiesha Dahyarif
Ilphaeryl Xun'viir


User avatar
thimblegiant
Arelith Silver Supporter
Arelith Silver Supporter
Posts: 89
Joined: Sat Jan 12, 2019 1:24 pm

Re: Regarding Recent Performance - Impending Server Split

Post by thimblegiant » Sat Mar 28, 2020 11:10 am

Wish there was some way we could chip in to assist, but I'm not sure what to do other than to offer encouragement and thanks.

I was just on the C&P server (about noon GMT), and it was having performance issues, so much so it wasn't really playable. There were only 41 players, and my ping time was solid. I wasn't rubber-banding for the most part, but there were 30 sec delays rippling through every few minutes (open a shop - lockup for 30 sec, people popping, etc). The server was up just shy of 13 hours.

Given that presumably throwing larger hardware at it doesn't help, I would guess there is resource loss occurring from within the game. Probably not just memory leaks, but other finite resources that aren't being cleaned up properly. In that case it makes sense it's only a matter of time before a server reboot is required, which seems to be the current solution. With more players, it only exacerbates the problem. I know last night there was heavy use, with 100+ players on C&P for quite awhile. It also doesn't seem likely that bandwidth is really an issue because of the experiences I was just describing above, with only a small number of current players.

Maybe a server split is better in the long run anyway, given it will allow for a longer period between forced reboots, even if the numbers do drop off again in the future.

I would love to work on the engine to fix these issues, too bad it's not open source.

If there *is* anything we can do, please ask. I'm sure many would be willing to step up, but I know it's not like you need help moving a couch or something.

*Edit1: I got to thinking after posting this that memory fragmentation could also be another culprit, which may make sense why large delays are being introduced after awhile. No way to know without getting at the source, but I wonder if certain scripts could also be more problematic. Did I read the Arelith was open source? If so - where is it maintained? Wouldn't mind poking around a bit. Been meaning to do more with NWN anyway, this is as good of an excuse as any.

*Edit2: Okay - feel free to ignore my naive ramblings. I began to dig back into the various pieces and there is a lot to consume. See you in about a year (or two) after I get it all sorted. This looks to be a complex beast, and that's just the tools and scripting, I can only imagine what the engine looks like (moot point anyway). Still, willing to dive in and learn scripting. Personally really like this game and would love to be able to try to support it.
Last edited by thimblegiant on Sat Mar 28, 2020 12:59 pm, edited 2 times in total.

User avatar
Kuma
Arelith Supporter
Arelith Supporter
Posts: 2188
Joined: Mon Sep 15, 2014 5:05 pm
Location: Melbourne

Re: Regarding Recent Performance - Impending Server Split

Post by Kuma » Sat Mar 28, 2020 11:20 am

Nekonecro wrote:
Fri Mar 27, 2020 7:20 pm
Would like to know how a merged player list will affect things like yoinking and scrying?
Usually one would open the list with P and then send a tell to the person with -scry or -yoink.
Will there be issues if it's all in one list for cross server scry/yoink?
i imagine this won't impact the in-game default playerlist, which hypothetically allows for the metagaming described but no more than "theyre in the ud" or "theyre on surface" is as it is, and with four possible servers to be on it convolutes this method further. so i can't see the default P-playerlist being impacted, just the command and site.

i hope im wrong, because if im not im going to scream

House Freth: Reference Information
House Claddath: Reference Information
"What's a heretic?": a guide to religious schism terminology

Irongron wrote:

4. No full screen images of the NWN gnome model (might frighten the children)


Best Rich Face
Posts: 53
Joined: Wed Jan 22, 2020 4:34 am

Re: Regarding Recent Performance - Impending Server Split

Post by Best Rich Face » Sat Mar 28, 2020 5:51 pm

Is it planned to reintroduce a server transition into the DS death area as part of this split?

I have to say, being unable to move cross server for capture RP is really awful as it is, and having the main pirate headquarters on DS will probably exacerbate this in the case of say, Cordor navy or Guldorand navy captive RP either way while fighting pirates.

Post Reply