Arelith

Could we please rollback character files + database? Otherwise there is a massive imbalance for how this goes down.

Situation 1: Your character is a merchant with a stall. They sell lots of big ticket items and earn lots of gold! The gold is in their bank, where it is safe. This stuff happens, rollback, and they still have their gold.

Situation 2: Your character bought those big ticket items. You don't have the items, you don't have the gold in your account. WBTHTGA I guess.

in the meantime i'd approach those merchants you purchased from and see if you can get the item you put money down for

We rollback depending on the necessity and severity of impact. Rolling back everything is a lot longer of a process when the impacted parts is just character inventories.

We understand there is always going to be some level of frustration but I made the call on how this was handled. And, I put getting the servers back up into a stable status with only characters modified in that time to be rolled back a priority.

MalKalz wrote: Mon Mar 18, 2024 2:33 am

We rollback depending on the necessity and severity of impact. Rolling back everything is a lot longer of a process when the impacted parts is just character inventories.

We understand there is always going to be some level of frustration but I made the call on how this was handled. And, I put getting the servers back up into a stable status with only characters modified in that time to be rolled back a priority.

from what i understand of what happened, and how the rollback was implemented, isnt it possible that some of the weapons that were dweomered made it thru? Wouldn't they be around if someone made such a weapon, then put it in their bank, it would still exist, no? I bet there's at least one

highly relevant
https://www.youtube.com/watch?v=-tFSujz6z6s

But given the nature of the rollback, would it not be appropriate to consider doling out a square 500xHD gp and 500xHD xp to all players? or something like that? Most higher level players (myself included) lost way more than that

Quizmar wrote: Mon Mar 18, 2024 5:43 am

MalKalz wrote: Mon Mar 18, 2024 2:33 am

We rollback depending on the necessity and severity of impact. Rolling back everything is a lot longer of a process when the impacted parts is just character inventories.

We understand there is always going to be some level of frustration but I made the call on how this was handled. And, I put getting the servers back up into a stable status with only characters modified in that time to be rolled back a priority.

from what i understand of what happened, and how the rollback was implemented, isnt it possible that some of the weapons that were dweomered made it thru? Wouldn't they be around if someone made such a weapon, then put it in their bank, it would still exist, no? I bet there's at least one

highly relevant
https://www.youtube.com/watch?v=-tFSujz6z6s

Given how widespread the reasoning for this rollback was, if any players were harbouring copies of the weapons in other mediums and did not come forward, the DM team would take extremely harsh sanctions against said player. It would not be difficult to spot an individual using such an item.

Dreams wrote: Mon Mar 18, 2024 1:35 am

Could we please rollback character files + database? Otherwise there is a massive imbalance for how this goes down.

Situation 1: Your character is a merchant with a stall. They sell lots of big ticket items and earn lots of gold! The gold is in their bank, where it is safe. This stuff happens, rollback, and they still have their gold.

Situation 2: Your character bought those big ticket items. You don't have the items, you don't have the gold in your account. WBTHTGA I guess.

Yep. I bought a sword for 220-230k and also a gem with ghost. I lost gold and items after rollback. On a new PC 250k~ its a huge amount of gold.

I am going to be frank and blunt here.
The course of action to get the servers operational as soon as possible seems hurried and thoughtless.

This:
[]Created an imbalance of duplicate/missing items. So some people may have lost a lot of goodies that they took from the chest, and some others could find themselves waking up with an extra set of armors/weapons/etc.
[*]Doesn't actually safely dispose of all the impossible +20 weapons which is the main reason that the server had to be rolled back, as any player that decided to save one in the chest can keep it.

Personally I lost a good chunk of gold, and I wouldn't mind that if I felt this was handled competently and responsibly, or if I could at least petition to get back what I lost. Neither of those are happening, and I already heard of people rolling their characters in frustration for what happened. 10k worth of pity experience is not really going to cover what some people lost.

This should have been a full, consistent rollback. Server and vault.

Just from what i've heard in the limited circles im apart of:
One person had been given a gift of 100 addy from another that rolled.
One person had bought two gravity orbs.
In both instances what happened was essentially deleted with the rollback, just speaking for myself here but i'd rather the extra time be taken so things dont go side-ways. and whats more, is that if someone that did abuse whatever happened decided to put the item(s) in a chest the rollback wouldn't have even cleared it?

Just adding what i believed that many would actually missed out on.
The database rollback will have to include -EVERYTHING- which means, Fixtures, Documents, Elections or simply just -ANYTHING-

That's actually pretty horrifying to those who do not keep a copy IRL of what they wrote on these fixtures / documents that are filed in bookshelves IG, or perhaps even remembered where they placed something...
I believe that's one of the reason perhaps that the database is sparred from it but not the Character.

Granted...some may debate that the rollback should include shops, bank, storages. But...I am not the expert here, i don't know if any of those are tied to anything esle.

Edit : And it's not that i'm not impacted as well because I've lost a parrot chalice and other stuff.
Well...just going to keep finding a new one.

It's pretty horrifying right now if someone took a notebook out of a bookshelf yesterday and completely lost it. Or had a fixture they picked up from the ground.

In any instance of how we proceeded, we would have offended someone.

Full database rollbacks are complex - they will hit everything and impact everyone. Yes, your bank would have rolled back, but at the same time the awards people earned from rolling would be gone (someone receiving their first ever major would lose it - and yes this would have happened).

You also have no idea how tables are structured in the database. It’s not like a bank has its own separate table from the PC table, it’s combined - rolling back a single table from a database is a lot of work and reverting it all means anyone new that was created would have been wiped. So they would lose a lot more too.

Calling this decision hurried and thoughtles? I gave it proper analysis in what time I had to be as least impactful as I could. I considered finding ways to help those impacted, but I don’t have the resources or time to work on every single character. It isn’t fair to me and it isn’t fair to anyone I wouldn’t be able to help.

Compensation? There was never compensation before for issues - you had issues and outages and you received nothing but a server being brought back up and you impacted somehow. However, I started to implement requirements and mandates for changes to ensure that they were never deployed on weekends to avoiding impacting most players free time. Sometimes things get in depending on who is wanting things in and they are unaware of other changes they bring with them - which is what happened here. But, I have always tried to strive to be as fair as possible to ensure players time doesn’t feel wasted.

Gold? I considered it. EXP? Requires more implementation than time that I had.

Also compensation shouldn’t be looked for all the time. It’s a sense of entitlement that I am not fond of - if a server has an issue, I’m immediately flooded with compensation inquiries and award requests for outages. It shouldn’t be a thing but it is.

No one even considers the time that the volunteer staff gives up in these moments and they do not receive any compensation. They lose time with the family, they lose their own free time to play Arelith if they do, they aren’t paid anything and they don’t often receive a simple thank you. And no, please don’t post it now that it’s mentioned.

I’m getting tired of firefighting for the amount of crap that is thrown at us for trying to make the server available and you less impacted. It’s really not worth it these days, and it’s burning myself out and others that have to do it. But we do it because we care, so there is that.

Either way, we’ll see what generosity I have in my time and effort to see if there’s something more I can do. If it’s needed.

So here is no reason to wait for returning gold or items? And just keep playing and trying to achieve again your things?

MalKalz wrote: Mon Mar 18, 2024 9:55 am

I’m getting tired of firefighting for the amount of crap that is thrown at us for trying to make the server available and you less impacted. It’s really not worth it these days, and it’s burning myself out and others that have to do it. But we do it because we care, so there is that.

I think amidst people being vocally annoyed it's important that you hear something positive, MalKaz.

I really appreciate how quickly you dropped whatever you were doing to get the server fixed. For as long as I've been playing Arelith you have consistently done this whenever there was an issue, and that is amazing.

Thanks for your commitment and effort, without which we probably wouldn't have a working server right now.

Jencent wrote: Mon Mar 18, 2024 9:59 am

So here is no reason to wait for returning gold or items? And just keep playing and trying to achieve again your things?

Not at this time. I do not have the cycles to look over every single persons case and validate through what logs we can use.

MalKalz wrote: Mon Mar 18, 2024 11:08 am

Jencent wrote: Mon Mar 18, 2024 9:59 am

So here is no reason to wait for returning gold or items? And just keep playing and trying to achieve again your things?

Not at this time. I do not have the cycles to look over every single persons case and validate through what logs we can use.

Perfect moment to me to spend almost all gold of PC for a new coe item...

Just like "Bad things happen to Good adventurers", inconvenient rollbacks can happen to unlucky players. And as much as it sucks to lose something cool/expensive, you got to remember that you are playing on a server that is hosted, developed and maintained by someone else on their free time and expense.

If I pay money for a service and experience severe disruptions I'd feel entitled to compensation. But I don't pay a cent to play on Arelith, and the amount of enjoyment I have wrangled out of it for the past four years would likely have cost me hundreds of euros elsewhere.

Expecting, even demanding, compensation for rollbacks is simply preposterous.

Wethrinea wrote: Mon Mar 18, 2024 11:30 am

Just like "Bad things happen to Good adventurers", inconvenient rollbacks can happen to unlucky players. And as much as it sucks to lose something cool/expensive, you got to remember that you are playing on a server that is hosted, developed and maintained by someone else on their free time and expense.

If I pay money for a service and experience severe disruptions I'd feel entitled to compensation. But I don't pay a cent to play on Arelith, and the amount of enjoyment I have wrangled out of it for the past four years would likely have cost me hundreds of euros elsewhere.

Expecting, even demanding, compensation for rollbacks is simply preposterous.

AMEN <3

I vote everyone gets refunded their subscription fees and all of the money spent here on micro transactions this month.

I will be awaiting my check for 0 dollars and 0 cents with baited breath.

Alright Mal, fair enough.
I retract the part of 'hurried and thoughtless'.

And true, I don't know how Arelith handles its DB or what backflips you guys would have had to perform to roll back fully. Surely, some people could've lost their first Major, some people could have gotten a free award reroll - it definitely could have happened, but it would have been at least consistent and dispassionate for everybody across the board, which I think is ultimately as fair as any method can be.

I'd also like to point - generally - that the whole of Arelith is a bit of a community effort. Not just on part of the Staff - a lot of players make practical second jobs to consistently make events, build scenes, create spectacles, and equally spend inordinate amounts of time doing tech help, giving mechanical advice, building resources, sites, maps, or updating the wiki. The privilege of playing in Arelith is only thus because we built this city (on Rock and Roll) over literal decades - some people may not be paying with their wallets, but certainly it'd be equally preposterous to pretend that they aren't paying at all to keep the server alive.

The flak for something that is largely an accident or demands for compensation are outright misplaced, even if I can understand the frustration that spawns those complaints.

Aeryeris wrote: Mon Mar 18, 2024 10:20 am

I think amidst people being vocally annoyed it's important that you hear something positive, MalKaz.

I really appreciate how quickly you dropped whatever you were doing to get the server fixed. For as long as I've been playing Arelith you have consistently done this whenever there was an issue, and that is amazing.

Thanks for your commitment and effort, without which we probably wouldn't have a working server right now.

I very much appreciate this answer, thank you.

Due to a technical issue Spyre, Backlands and others gave up their Sunday to work for several hours on getting the sever back up, and playable. Arelith handles a large volume of players, and has a highly complex build structure.

Despite being up 24/7 and being constantly updated with new features and improvements we very rarely suffer any kind of downtime or server issues, because we have some massively qualified people working behind the scenes - to the extent that players often don't even realise that issues have been addressed. Just last week we were affected by a security breach owing to an exploit discovered in the Team City service, and reported in the press. It was picked up by us almost immediately, and Hoodoo dropped everything to get us functioning and safe again before the servers would next need to be reset. Every single day we merge a number of updates into the live repo - sometimes they are big announced changes to the world, as I, Twohand and the writing team have been making these last weeks, while working pretty much full time, more often they are optimizations or groundwork for future updates that players don't even see. Across our staff discord channels, GitHub, Team City, Database tools and in game, the work to keep this running is genuinely constant. A great many commercial titles don't see this level of round the clock effort, or benefit from having such a large dedicated staff.

Among it all I have been getting pretty sick of the level of rudeness expressed by some towards our staff over stuff they don't like.

We saw it recently with ship changes, which basically cost us the developer working on it (at least for now), it's happening, of course, with the summon changes, it's happened here with what was just extremely painful DB editing so we'd at least have a server to play on (in the process of which sadly some folk lost pretend ingots) oh, and barely a DM event goes by without my receiving at least one complaint from a player who thinks the DMs somehow aren't being fair with them.

It's getting hard to find volunteer staff for Arelith when they sometimes get treated so awfully, and whatever work they do so swiftly being thrown back in their faces.

Personally? I was skeptical of the 10k XP reward they put on every account following this. It wasn't their fault, and developers shouldn't need to feel that they're obliged to somehow compensate players when it was they who did all the work. It certainly didn't seem to be appreciated, and will likely set such absurd expectations in future.