Page 1 of 1

Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 10:27 am
by Irongron
As players may have noticed we have been experiencing increasing instability on the servers, after discussion with Mithreas and Morderon, we have determined the only sure-fire way to really address this is by splitting the servers once again.

We have great hardware, but Arelith is a complex engine running in an old game, and player numbers, are by far the highest factor in performance. While I love seeing numbers of 100+, I need to really consider the enjoyment of players first and foremost, and anything over around 75 players will always cause issues. I don't mind being above these numbers for short periods, but we're increasingly hovering around 100 for much of the day, especially for Cities & Planes.

My thoughts are currently as follows, but may change.

Underdark will once again be placed on its own server.
Cordor, along with Baator, and the Shadow Plane will be on a second.
All ship and island content, including Sencliff, will be moved to Distant Shores
Surface server will continue to include all mainland content, including the new city.

Should it be required, if numbers continue to grow, the nation of Guldorand too, may end up on its own server.

I can't give an exact ETA on this, as splitting servers is a really time-consuming process, both for myself and Morderon. Separating the Underdark will be quite straightforward, but moving the ship system to DS will require some code adjustments to the shipping system which will likely be rather complex.

Further to this I've asked the team to look at merging the portal page and -playerlist function into a single list. Sadly this is due to the fact that the last time the Underdark was on its own server there was a significiant problem with players there using the player list to metagame the presence of surface PCs, and using it to send threatening tells, or to justify searching transition areas for the interlopers. I hope that our playerbase has matured since then, of course, but nevertheless a combined playerlist is likely to everyone's advantage.

I can't say I'm wild about splitting the servers, there are a number of disadvantages, but as I said above, it really is our best possible method to improve performance for all of our players.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 11:29 am
by CookieMonster
Hopefully this isn't to much of a bother for the Team. But we are of course grateful to any time and effort you put into our continued enjoyment. Thanks guys.

Though, I imagine that a lot of the traffic is as a result of the Corvid 19 Pandemic. Many countires are in complete lockdown. The UK for example as of Tuesday has restricted travel to any location outside of your home to essential movements only.

Stay safe everyone.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 2:18 pm
by In Sorrow We Trust
+1. This is exactly what I was suggesting be done a while back.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 3:35 pm
by Bunny
Great time to add more ships for player ownership

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 3:38 pm
by Ork
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 3:44 pm
by good man of god
Ork wrote:
Fri Mar 27, 2020 3:38 pm
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 3:49 pm
by Void
Would be nice to have ability to transfer henchment through transitions, given that number of leyline regions increases once again.
It would be also nice if beamdog did a multicore update for the game logic, but that's likely a pipe dream at this point.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 3:54 pm
by Good Character
Hard but strong decision. How are the server costs going look like now?

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 3:57 pm
by Xerah
Please refrain from requesting new features in this thread. This project is a huge undertaking and it comes off as so dismissive of the time and work involved when you jump right into asking for new things. Thanks

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 4:07 pm
by Bulldog711
Just a random thought here for the possible future splits: make Surface/mainland broken up into north and south sections, or something similar. A good line of that transition might be somewhere in Arelith Forest but I do not know the full stretch of content behind the scenes so it might be better shifted elsewhere. It could alleviate the stress on the servers and make it flow easier on the transitions. Just giving my 2 coppers and I'm sure we all greatly appreciate the efforts yall are making to have the game run smoother.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 5:49 pm
by Flower Power
I've been saying that throwing the UD off onto its own server (or, at least, onto the Skal server) would improve overall stability for years, sooooo.

+1

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 7:20 pm
by Nekonecro
Would like to know how a merged player list will affect things like yoinking and scrying?
Usually one would open the list with P and then send a tell to the person with -scry or -yoink.
Will there be issues if it's all in one list for cross server scry/yoink?

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 9:11 pm
by Zahlfire
Ork wrote:
Fri Mar 27, 2020 3:38 pm
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 9:35 pm
by AstralUniverse
Ork wrote:
Fri Mar 27, 2020 3:38 pm
I think this is a hard decision but the right one IG. We appreciate your work to keep this beautiful game afloat.
1000%
Irongron wrote:
Fri Mar 27, 2020 10:27 am
My thoughts are currently as follows, but may change.

Underdark will once again be placed on its own server.
Cordor, along with Baator, and the Shadow Plane will be on a second.
All ship and island content, including Sencliff, will be moved to Distant Shores
Surface server will continue to include all mainland content, including the new city.
Just wondering, how about switching between Baator + Shadowplane, and the new city, and having two cities in one server and the entire surface content in the other? Would it be balanced in terms of content size and player traffic?

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 9:38 pm
by DM Rex
As crazy as it is, most people don't hangout in hell. They hangout in the cities.
Hence why it's presently presented as the proposed split above.

The two largest being still - Andunor and Cordor.

Re: Regarding Recent Performance - Impending Server Split

Posted: Fri Mar 27, 2020 9:39 pm
by LichBait
<3 To this. Thanks for all you do.

Re: Regarding Recent Performance - Impending Server Split

Posted: Sat Mar 28, 2020 11:10 am
by thimblegiant
Wish there was some way we could chip in to assist, but I'm not sure what to do other than to offer encouragement and thanks.

I was just on the C&P server (about noon GMT), and it was having performance issues, so much so it wasn't really playable. There were only 41 players, and my ping time was solid. I wasn't rubber-banding for the most part, but there were 30 sec delays rippling through every few minutes (open a shop - lockup for 30 sec, people popping, etc). The server was up just shy of 13 hours.

Given that presumably throwing larger hardware at it doesn't help, I would guess there is resource loss occurring from within the game. Probably not just memory leaks, but other finite resources that aren't being cleaned up properly. In that case it makes sense it's only a matter of time before a server reboot is required, which seems to be the current solution. With more players, it only exacerbates the problem. I know last night there was heavy use, with 100+ players on C&P for quite awhile. It also doesn't seem likely that bandwidth is really an issue because of the experiences I was just describing above, with only a small number of current players.

Maybe a server split is better in the long run anyway, given it will allow for a longer period between forced reboots, even if the numbers do drop off again in the future.

I would love to work on the engine to fix these issues, too bad it's not open source.

If there *is* anything we can do, please ask. I'm sure many would be willing to step up, but I know it's not like you need help moving a couch or something.

*Edit1: I got to thinking after posting this that memory fragmentation could also be another culprit, which may make sense why large delays are being introduced after awhile. No way to know without getting at the source, but I wonder if certain scripts could also be more problematic. Did I read the Arelith was open source? If so - where is it maintained? Wouldn't mind poking around a bit. Been meaning to do more with NWN anyway, this is as good of an excuse as any.

*Edit2: Okay - feel free to ignore my naive ramblings. I began to dig back into the various pieces and there is a lot to consume. See you in about a year (or two) after I get it all sorted. This looks to be a complex beast, and that's just the tools and scripting, I can only imagine what the engine looks like (moot point anyway). Still, willing to dive in and learn scripting. Personally really like this game and would love to be able to try to support it.

Re: Regarding Recent Performance - Impending Server Split

Posted: Sat Mar 28, 2020 11:20 am
by Kuma
Nekonecro wrote:
Fri Mar 27, 2020 7:20 pm
Would like to know how a merged player list will affect things like yoinking and scrying?
Usually one would open the list with P and then send a tell to the person with -scry or -yoink.
Will there be issues if it's all in one list for cross server scry/yoink?
i imagine this won't impact the in-game default playerlist, which hypothetically allows for the metagaming described but no more than "theyre in the ud" or "theyre on surface" is as it is, and with four possible servers to be on it convolutes this method further. so i can't see the default P-playerlist being impacted, just the command and site.

i hope im wrong, because if im not im going to scream

Re: Regarding Recent Performance - Impending Server Split

Posted: Sat Mar 28, 2020 5:51 pm
by Best Rich Face
Is it planned to reintroduce a server transition into the DS death area as part of this split?

I have to say, being unable to move cross server for capture RP is really awful as it is, and having the main pirate headquarters on DS will probably exacerbate this in the case of say, Cordor navy or Guldorand navy captive RP either way while fighting pirates.