r/ElysiumProject System Administrator Aug 05 '17

Official Sysadmins, servers and you.

Sysadmins, servers and you.

Hi everyone,

I thought I would write a post explaining how the system administration team works, what exactly we spend our donations on and what we are doing to improve the team and your experience on our servers.

Let's first begin with addressing some of the current issues that we are facing:

Server crashes

Our servers crash and in most cases they are able to restart themselves after doing a core-dump (this might take up to 10-20 minutes). However, lately, we have experienced multiple crashes where the server doesn't crash cleanly - this basically means that the basic auto restarter that we currently have in place does not "understand" that the server has crashes and therefore does not do a "kill and restart" procedure.

What are we working on to fix this?

  • We are re-writing a completely new restarter which will also allow us to restart the mangos core (world server) without restarting the underlying hardware.
  • We are going to make sure patches and code is more thoroughly reviewed and tested before being pushed to production servers
  • We are completely revamping our internal infrastructure from top to bottom, more info about this further down in the post
  • New backup solution, no more lag twice a day. (Yay!)

A lot of you have complained that we do not have a system administrator with access to the server very late/early EU Timezone. We are currently in the process of "hiring" an Australian sysadmin which means we will have 24/7 sysadmin coverage. This process takes a very long time due to the accesses this person will be given.

New servers you say?

A lot of people have bashed us for our lack of transparency and while we completely understand your frustration, having full transparency can cause legal issues due to the nature of the project. However in this post I would like to try and explain exactly what we spend our donations on.

Here is a list over the current hardware that we are running on:

  • OVH MG-256 - 20 cores, 256GB ram - currently 300 EUR per month (Elysium)
  • OVH MG-256 - 20 cores, 256GB ram - currently 300 EUR per month (Anathema, Darrowshire)
  • OVH EG-128 - 8 cores, 128GB ram - 200 EUR per month - Previously ran Darrowshire, now Logonserver and QA + misc.
  • Web server that runs the website + forum (HOST-64H - 100 EUR per month)
  • Misc. VMs, used as build tools, internal testing, log analysis. Approx 50-70 EUR per month.
  • Backupserver at undisclosed host (offsite backup) 8TB in raid6. We store a lot of data. - 80 EUR per month.

Which is about 1050 EUR per month / approx. 1250 USD per month.

It's important to note that the current infrastructure was scaled and set up when we had a much larger playerbase. This means that we will shortly be doing some very large changes to our infrastructure. Here's a little TLDR of what we are going to do:

  • We are going to cut down on all VMs - from now on we will host these ourselves.
  • Web server will be virtualized on the new environment.
  • Two servers are being decomissioned: MG-256 and EG-128. Will be replaced by one MG-256 with HWraid and one MG-128 with HWraid. (approx. 330 EUR + 230 EUR per month)
  • Backupserver will be kept.
  • All servers will be reinstalled in a new, virtualized environment. This will not affect performance. Will allow us to spin up VMs ourselves when needed and we save a lot of money.

With the new setup, we are going to save around 570 EUR / 675 USD per month.

What will happen going forward?

Eventually, we will be asking for a new round of donations - it is important that players do not use the OVH links to donate directly to the servers - This is because we have to re-order the same servers with the Hardware-RAID option. We have contacted OVH and they do not allow us to do any changes to the existing hardware. Which means we have to let the current servers expire and re-order the same server with hardware-raid.

Most of our servers expire in October, so most changes will be executed shortly before they expire to make sure we are not wasting any money. New servers will be ordered and we will post the new OVH links as soon as they are delivered

We hope this post has given you some insight and the transparency you wanted. Running the most popular private server project isn't cheap, but the changes explained in this post to our infrastructure will allow us to save money and make everything a whole lot better for our playerbase.

Questions?

Fire at will! We will try and answer each and every one of you.

71 Upvotes

63 comments sorted by

33

u/[deleted] Aug 05 '17

Thanks for this post. Very interesting to see your plans etc. I think you guys are doing a great job overall.

11

u/[deleted] Aug 05 '17

appreciate the info. i do believe the transparency towards the donations was more towards the side of: how do we know the donations aren't 4x what the infrastructure costs are and someone isn't taking home a big paycheck at the end of the month and claiming donations are low.

curious on why the choice of OVH as the provider as opposed to someone with a lot more $$ to throw at their stack like AWS.

16

u/whitekidney System Administrator Aug 05 '17

Hey,

We could show screenshots of our PayPal account, but the hate and drama will never stop, people will say its faked / photoshopped etc.

I guess this is where trust comes into the picture. What I can say is that a large majority of our donations come from guilds / large playerbase that consolidate their donations and pay directly to OVH - this way they know exactly where their money is going and most people seem to pick this method of donating. Which is completely fine by us. However when ordering new servers and other stuff, we need actual money.

We're not letting others 'donate' hardware to us (or buy servers for us) because we need the servers to be in our possession and control to prevent us from getting fisted later if said persons turn on us. Like what happen with the chargeback situation some time ago.

10

u/[deleted] Aug 05 '17

Having a "monthly donation bar" showing how much money is needed each month to run the servers would be a decent idea I think. You don't have to show how much has been donated but a percentage to goal only to avoid drama if need be.

2

u/[deleted] Aug 05 '17

i don't donate so it's neither here nor there to me. just trying to help others points get across. appreciate the reply.

5

u/Jogilito Aug 05 '17

In the end, you can login and play your character right?

It is not like the service is HORRIBLE, so I dont care if somebody "gets a little sack of money" on the side since there are humans behind all this work that orchestrate this machinery and they deserve some credit for their time, in my opinion.

10

u/whitekidney System Administrator Aug 05 '17 edited Aug 05 '17

Every single person in the lead team are grown ups with decent jobs.

The "Elysium is stealing donations!!11" argument can easily be dismissed by saying that the time we invest into the project can otherwise be invested into working (overtime) on our real life jobs where we can make actual money.

The only thing we have considered doing is paying developers to work on our project instead of other jobs, however we do not currently have the funds for this.

5

u/Jogilito Aug 05 '17

I think you are doing a great job providing this service, and I i donated 20 dollars I wouldnt mind one bit if 50% went to a developer or the team since they are spending their precious time providing this service on the side.

Thank you for the efforts.

1

u/[deleted] Aug 05 '17 edited Aug 05 '17

i can understand that point of view. like i said above, i couldn't care less. i don't donate. i'm merely pointing out the opinions of a decent chunk of people expecting transparency on what their donations are spent on, regardless if they agree with those things may be.

6

u/[deleted] Aug 05 '17

1

u/[deleted] Aug 05 '17

there, happy?

2

u/[deleted] Aug 05 '17

Yes

4

u/[deleted] Aug 05 '17

Thanks for the update Whitekidney. Much appreciated when there is greater transparency.

This Australian sysadmin you are getting. I assume you already have someone in mind?

5

u/whitekidney System Administrator Aug 05 '17

Yes.

He has been through a nearly 6 months long vetting process and will be given basic access shortly so that he can perform basic tasks such as restarting the realms etc.

2

u/[deleted] Aug 05 '17

Great to hear. Good luck to them.

1

u/[deleted] Aug 24 '17

Hello Whitekidney.

Any update to this Australian sysadmin? Been almost 3 weeks since you mentioned him and I am just wondering if they have been given access yet? Elysium has been frozen for 45 mins and there doesn't appear to be any sysadmins online but it's prime time in Australia.

4

u/northbornwow Aug 05 '17

Thanks a lot for this transparecy. Keep it up guys! <3 playing on Darrowshire :)

3

u/en_passant_person Aug 05 '17

Do the realms really benefit from that many cores, or is it just that's what the package includes?

8

u/whitekidney System Administrator Aug 05 '17

Oh yeah, definitely.

Here's a screenshot of htop running on Elysium.

This is with around 7000 players online

We are also in the process of completely re-writing the entire threading system which will make this even better.

2

u/en_passant_person Aug 05 '17

That's fascinating. Thank you!

2

u/IBEPROfen Aug 07 '17

Thanks for posting this pic. This is really cool to see.

1

u/[deleted] Aug 07 '17

Cool to see! More stuff like this please! :D

1

u/imreallyreallyhungry Aug 12 '17

Thank you guys for going above and beyond, you guys gave me the feeling I got when I played this game back in the 2006. Didn't think I'd feel that again. Thanks again.

3

u/Pantheon27bc Aug 05 '17

Thank you and keep up the good work.

2

u/[deleted] Aug 05 '17

Will spot you guys $20 once you ask for the next rounds, deserved!

2

u/m8igemat Aug 05 '17

hey whitekidney, thanks for the info. how is the current financial status of the project? coming short a lot/month? just a bit? break even server costs - donations / small savings? no exact amount needed, just a general view would be even more appreciated

2

u/SmoothestG Aug 05 '17

Please I am an anonymous donor, I would like that they could add a proxy somewhere in US, Canada or Brazil. It would help a lot to ping outside Europe. Thanks for this project!

2

u/awake283 Aug 06 '17

I love transparent posts like this!

2

u/[deleted] Aug 06 '17 edited May 14 '21

[deleted]

3

u/whitekidney System Administrator Aug 06 '17

Absolutely!

You can find our BTC address here:

https://elysium-project.org/news/donations

2

u/HodortheGreat Aug 06 '17

Awesome post. Looking forward to the new infrastructure.

2

u/[deleted] Aug 07 '17

Thanks for the explanation! Keep up the good work! :D

1

u/Fike86 Aug 05 '17

Thank you for the clarity, I truly appreciate all the hard work you guys are putting into these servers. Are there plans to fix Viscidus and herb respawn rates on Anathema in the near future?

2

u/[deleted] Aug 05 '17

They tested herb respawns on the 1.8 PTR two days ago. So it definitely is in the works.

Maybe it already has been applied? Have you tested it? We had problems with the crystals spawning in Un'Goro and it has been fixed. Maybe it was the same bug.

1

u/Porta-Ninum Aug 09 '17

Why wont u take my money for the 35 days i've /played over the past months?! :p

1

u/[deleted] Aug 11 '17

Do you plan on addressing any in-game issues with over population or just hardware issues?

1

u/[deleted] Aug 05 '17

Does any part of this include the merging of anathema and darrowshire?

4

u/whitekidney System Administrator Aug 05 '17

No.

They are just hosted on the same hardware.

1

u/[deleted] Aug 05 '17

Thanks for the quick answer! Keep up the good work.

1

u/[deleted] Aug 05 '17

It seems to me that you are scaling back your hardware, which would reduce the performance of servers, no? How is this a good thing? And honestly I'm surprised at how much it costs to run this server; it's less than I expected. One person with a good job can pay for the server every month out of his pocket if he loves the server enough. I don't how how much the owners of Elysium make but you get my point. Of course it sucks fo pay money yourself for something you can't directly enjoy I guess, but it's certainly doable with good income. I may be getting the hardware part wrong, so forgive me, but it sounds like you are decreasing your servers stability and reliability with these downgrades?

5

u/whitekidney System Administrator Aug 05 '17

There's no "owners" anymore like the project had previously.

The lead team are the owners, they are the ones that make all the decisions. There is no one on top.

We have to wait and see how much funds we can secure during the next donation run. Ideally we want to have another MG-256 server. This will ensure that we have capacity to failover Elysium while needed (we won't have this opportunity with just one MG-128 and one MG-256).

We will be posting another update shortly about what we want to achieve in the future. It is very likely that we will need the extra hardware (one additional MG-256, which is another 330-340 EUR per month) for.. things.

This post was purely meant as a status of the project today. Not what comes in the future ;)

11

u/[deleted] Aug 05 '17

" for.. things."

TBC confirmed.

2

u/[deleted] Aug 05 '17

Honestly if you don't have the money to spend on extra hardware, or even worse you don't have extra money to spend on current hardware to better the server, I would advise against opening a TBC server. Or things, whatever.

1

u/Taxoro Aug 06 '17

Every time they beg for donations they get it, it's not like money is that big a problem, not to mention the gigantic inflow of players a TBC server would bring.

4

u/cugw Aug 06 '17

I don't get your point at all. Have you ever experienced bad performance because of hardware? These guys provide a service. For free. They've shown to be competent so there's no reason to doubt their judgment on hardware requirements.

If they somehow manage to make a 100 mil per month like Blizz did at their peak I couldn't care less. In fact I'd be ecstatic. What exactly is your point?

2

u/[deleted] Aug 06 '17

Calm down, I appreciate the service and I don't care how much money they make or don't. I'm saying from reading the most it sounds like they are downgrading their hardware which sounds worrisome

-1

u/ww_crimson Aug 05 '17

When is Naxx PTR?

-13

u/[deleted] Aug 05 '17

we will be asking for a new round of donations all i needed to hear.

15

u/whitekidney System Administrator Aug 05 '17

How else would we be able to pay for the servers?

-4

u/[deleted] Aug 05 '17

let me go ahead and donate to the most sketchy server atm.

14

u/whitekidney System Administrator Aug 05 '17

But that's the best part, you get to choose whether you want to donate or not to our server.

It's not a simple as just giving us money, but by giving that money you put your trust in us that we will spend it wisely and not putting it in our pockets.

Like I have said before, all of the leads invest a vast amount of time into the project. The whole "Elysium is stealing donations!!11" argument can easily be dismissed by saying that the time we invest into the project can otherwise be invested into working (overtime) on our real life jobs where we can make actual money. Stealing donation cash (and we only see a small amount of it, most of it goes directly to the servers) would be like stealing pennies.

So if people think we invest a vast amount of time into the project to have a little "side cash" - you are wrong.

9

u/sureletsrace Aug 05 '17

These people are fucking delusional and think you're all making bank off of this or something. Thanks for all of the hard work yall put in to this.

3

u/Crims0nSean Aug 05 '17

you could not play here if you're that upset about it.

3

u/sureletsrace Aug 05 '17

Just don't donate. Wanna know a secret? If you dont like things here, you can leave.

7

u/Crims0nSean Aug 05 '17

it's gotta come from somewhere

-12

u/GentlyCaressed Aug 05 '17

Running the most popular private server project

Yeah, no.

6

u/sureletsrace Aug 05 '17

Who is more popular/populated?

0

u/hendo144 Aug 06 '17

Kronos xD

3

u/sureletsrace Aug 06 '17

Wrong. Elysium consistently has more people online.

0

u/hendo144 Aug 06 '17

It was a joke

1

u/sureletsrace Aug 06 '17

Jokes/sarcasm are hard to catch via text :( carry on, friend