Arelith

Hi,

Disclaimer: Mandatory reminder that the Staff are volunteers doing marvels with an old outdated engine with no full access to the code. That type of thread is not meant to have unfair/entitled demands toward the team. I dare open such a topic all while fully acknowledging the mind boggling complexity of coding, pipeline organization, and QA testing. So for the love of everything holy, let's just... be nice?

So I'm sure it has come to everyone's attention (or has it? Maybe that's just me making stuff up, in which case please correct me), that there's oftentimes a very significant lag between updates deployment and the update thread. A non exhaustive list of things that happened within a month:

- An update is announced for PGCC but the actual effective update to be experienced by players will then take up to three more days before it's online.
- An update is announced for PGCC, the content is live on PGCC at the moment of the update (or close enough), but the announced material doesn't work, even if announced on Update.
- An update is announced for live and PGCC, but the announced content isn't correctly deployed, or deployed at all.

While all this topic is not the end of the world and can be disregarded on a basis of "be patient and stop whining", I feel that while this line of thinking surely has merit, maybe on the devside there's also steps that can be taken to limit (if not nullify) that from happening. "Updates lag" creates confusion (is it on or not?), uncertainty (should I trust the update thread?), or breach of trust (I level up trusting the update thread and end up in deadends due to misleading info OR something is announced as fixed/operational but it is not). Again, not the end of the world but can have sideeffect that are negative so there's that.

I obviously have no insight in the current pipeline of arelithian devs (and lifting the veil a little bit on your organizational processes/softwares, pipelines etc would be tremendously interesting, winkwink), but maybe a way for the update thread to be always in syncs with server-reboot would be to have devs that want to post in the update thread have their posting "queued" and the next person who resets servers/PGCC must post for others in the update thread/discord the queued update logs.

And if a staff member reads this, is there a dedicated QA member in the team? If not, how would you feel about 'recruiting' a member of the community with the explicit mission to check each updates before they are deployed (not the code, just the playerside of it+repro steps if issues)? Maybe there would be volunteers for that. Maybe that's a terrible idea. Just throwing that out there.

I will leave it there. If anyone feels like chiming in, please go ahead!

All of us are volunteers and live in different timezones. Sometimes an update will be submitted and approved but a bug in it was missed until after announcing. Sometimes an update will be announced and our server launching application will decide to die on us. Sometimes it will get announced as next reset and everyone will assume it is immediately in the game. Sometimes something in real life comes up in between announcing and implementing an update. Sometimes we just make a mistake and announce something too early. There are lots of reasons why this isn't going to be a perfect process. We are not a company and do this in our spare time.

I get that but more often than not an update to internal communication processes/softwares, or pipelines, can go a long way in reducing that kind of issue. Or, as mentioned, having an appointed person making sure that the hard work of devs is brought to the playerbase in an orderly fashion (syncing with reboots, testing, etc). That would mean marginally more time between patches, sure, and maybe that would make the process too heavy for timezone-scattered devs, I don't know, again I don't have insights about the current organization.

It would be unfair/unreasonable to expect professional pipelines, you guys don't have the time for it (albeit you make it up with your skillzzz, I have no doubt about your wizardry level). Frankly I'm not sure what's the solution could be but there's bound to have time-efficient ways to improve this "lag" without it being a strain on you guys. At least it's in that spirit that I started this thread.

To give a bit of insight:

The last batch of updates had a few hiccups that we encountered and some updates to the live servers were announced too soon. This can happen from time to time, but it is not the end of the world.

When something is exclusively being tested on the PGCC that utilizes hak files (2da or json), we have to alter our build process. Normal functionality is that it goes from our GitHub repos through TeamCity where it builds it and then the admins package it, but with PGCC exclusive testing, we go onto what we call a “Hak hold”. What this effectively means is that we alter a step in TeamCity which stops propagating those hak files to the live server and only to the PGCC. We then have to alter those jobs again when we want them to go to the live servers.

Currently we only have two people who handle that alteration. I am going to take some time to learn the back end of it so we have three. However, even with three, sometimes people will think that something was shipped when it was stuck on the PGCC only. Which is what happened.

Here’s a quick look into how our work is processed:

1) Either we have a detailed design thread on the forums created with discussion or we pick up bug reports.

2) We do the work and submit it through GitHub where the admins then approve its merge across.

3) We create an announcement piece in a separate Discord channel. Once it’s merged and staged for the next reset, it should be announced.

So there is organization. There is a plan. Stuff just happens from time to time. There really isn’t much more that we can improve upon in the sense of design, other than paying more attention to when announcements are to go out.

We could leave that exclusively to the admins, but they are not always around. So, it comes down to when it’s staged and someone grabbing it. We try our best.

Hey Spyre,

Thanks a lot for the "behind the curtain" info, it's really really interesting. I appreciate it a lot.
And again, we both said it's not the end of the world and "stuff happens" so we're definitely on the same page here. I never suggested/hinted at that there was no organization, that would be incredibly dumb (and utterly disrespectful) to even consider it.

Don't misunderstand me, I definitely root for the dev team and have utmost respect for your time, work, and commitment. However there has been a decent amount of hiccups in the past month (or a string of bad luck) and well, that's also what the feedback part of the forum is for.

Keep up the good work!