Redstone Wire, Ltd.

Donations, costs, traffic!

For most of 2010 traffic had been stable, we were slowly growing but nothing drastic… then it hit, late July Minecraft really started to gain momentum, first we had the TF2 blog about Minecraft, then the popular Youtubers talking about minecraft and from there it snowballed, Minecraft has now been featured in magazines, newspapers, blogs, TV shows, radio shows, podcasts and more, think back to Minecraft.net crashing for an idea of the sort of troubles we were going through, we’re expotentially more server heavy than minecraft.net!

Early last week started seeing increasing load to the servers, much more than our already high powered setup could handle, we discussed the possible solutions and we decided that the only choice was to upgrade our current servers and bring more online, unfortunately we didn’t have the capital to do so. The idea of donations was suggested by WedTM, we weren’t too optimistic but we decided to try it, our only alternative was to have 90% downtime and that just wouldn’t work. 

To say we were blown away by the generosity is a huge understatement, you guys blew our minds. It’s an honour for us to operate a community of such generous people, we love you guys. 

Now for some statistics…

minecraftforum.net

In the past 30 days minecraftforum.net has had 2,286,011 visits, 16,238,746 page views and 825,505 unique visitors. Each visitor spends an average of 7 minutes and 36 seconds on the site and views 7.16 pages. 

minecraftwiki.net

In the past 30 days minecraftwiki.net has had 2,986,539 visits, 15,350,723 page views and 900,921 unique visitors. Each visitor spends an average of 6 minutes and 40 seconds on the site and views 5.14 pages. 

Hardware

This sort of traffic isn’t easy to handle, at the beginning of the year we were running from a small 512MB Ram VPS and now we’re running from the following machines:

Web Server #1

• 2x Xeon-Nehalem 5520
• 12GB DDR3 Ram

Database Server #1

• 2x Xeon-Westmere 5670
• 12GB DDR3 Ram

Load Balancer #1

• Xeon-Lynnfield 3470
• 2GB DDR3 Ram

Shared between these servers we have 12TB of bandwidth on a 1GBPS line, we have also been using the Softlayer Cloud to bring online extra instances that can support spikes in traffic that we receive during Friday updates. 

Donations

Now for a breakdown on the donations we’ve received over the past 7 days.

Total Donations: 226
Total Sent: $2,800.01
Total Received: $2,635.62 (after Paypal fees)

Average Donation: $12.38
Biggest Donation: $100.00
Smallest Donation: $0.06

Donations in £: 56
Donations in $: 170

Users with gold forum names: 197

How we’ve used donations

Softlayer Hosting costs: $1,527
Softlayer Account Balance: $409
Software costs: $200
Saved “emergency fund”: $327

Money gained from £/$ currency conversions and increase value of £ over $: $34

The future…

Starting in November we’ll be receiving the ad revenue from the websites, as such we’ll no longer need donations to remain online and any donations made will be used to support the community, as of today we’ll be removing the announcement regarding donations. 

Day 1: A Blog Post

I’d like to take the time to pat ourselves on the back. Over the last 4 days both Andrew and I have worked almost around the clock to turn what used to be a simple forum and wiki installation into something that is scaleable, efficent, and redundant.

We’re not all the way there yet, there are still some things that need to be fixed, however those are really only lacking in funding, so that shouldn’t be an issue too much longer.

The second part of this blog post is going out to our donors. I posted a humble plea for help from the forum users, and in just under a week, you’ve donated nearly $2,000 to our backend infrastructure. 

This has purchased the following equipment

• Xeon-Lynnfield 3470 2GB DDR3 - Load Balancer
• 2x Xeon-Nehalem 5520 12GB DDR3 - Frontend Webserver
• 2x Xeon-Westmere 5670 12GB DDR3 - Backend Database Server
  

Now, the awesome part of all of this, is that that is our MAIN infrastructure. However, Andrew and I have configured the setup in such a way that we can pull up any number of nodes to add additional resources during peak times, on a cost per hour pricing scheme. 

Node images have already been configured, so during extra load (Seecret Friday Updates, anyone?) it literally is a click of a button, and an extra line in a configuration file.

Now, your donations have gotten us a long way, and hopefully they will hold us until November when our first round of funding comes in from adSense. Alot of the users who donated have already been given their shiny new titles on the forums. There are still a few that have yet to be setup, but I’ll be getting to those before the end of the weekend.

Tomorrow and Friday will really show us if our hard work has paid off. I’m expecting issues, however, I don’t think we’ll have extremely prolonged downtimes. I’m estimating maximum of 30 minutes as we allocate additional resources to take on the load.

If you haven’t donated yet, but are wanting to, We’ll be keeping the donation jar open for only a bit longer. Then that door, and all the benefits it brings, will close forever.

Citric and I have been dreaming up some fairly cool projects, and we’re planning on allowing donators first access to those. Kind of like a beta preview. 

We don’t have all the details hammered out quite yet, and frankly, I’ll probably get knocked down a few pegs for letting this much info out, but I feel like our users probably want to know what’s in store.

TL;DR: Donors are awesome, new ideas on the way.

All sites are operational!

There isn’t a single system in our configuration that doesn’t have the ability to dynamically increase power now, whether it be adding new frontend webservers, or adding more nodes for the SQL backends.

Unfortunately, that means more costs (doesn’t it always), we’re actively seeking alternative methods of funding….

Forums temporarily disabled… again

We’re once again seeing incredibly high load with the database server, this high load is mainly caused by the forum. We’ve temporarily disabled the forum until we can resolve the issue, this is to prevent both sites from going down, with the forum disabled the wiki can remain fully functional. 

We’ll keep you updated on progress, we’re currently investigating solutions to the ongoing problems we’re facing with load, hopefully we can get through this rough patch soon and resume normal service!

You can follow us on twitter for instant updates on things that happen:@redstonewire

Forums offline temporarily

We’ve taken the forums temporarily offline due to very high loads we’re seeing, they will hopefully be back later today. We’ve taken them down to avoid their high load affecting the wiki, the wiki will remain online during this forum downtime. 

We’re sorry for the problems that are happening right now, the popularity minecraft has received recently through TV, comics and Youtube users has affected us just as much as minecraft.net, we’re seeing millions of page views per day which is more than we can handle right now. 

Follow us on twitter @redstonewire for alerts when we return. 

update: We’re back online, sorry about that, we had a misconfigured load balancer, all is fixed!

Server issues 20th - 21st September

Since earlier this week visits to the forum and wiki have been increasing every hour, we’re now delivering millions of page views every day and because of this we’re constantly trying to stay ahead of the load, each day it gets more and more and it’s becoming hard for our servers to handle.

Last night we found that our new web server wasn’t handling the load properly, PHP was causing huge problems and we decided to bring online even more servers, this seems to have helped somewhat, combined with our new loadbalancer we seem to be just about handling the current load and due to the new configuration of our backend we’re able to (relatively) easily deploy new temporary servers to handle the load.

We’re sorry for all the downtime recently, it hurts us just as much as it hurts you but with our levels of growth and volunteer based team we’re doing as best we can. If you have any questions feel free to use the comments below and we’ll try and get back to you!

We’d also like to say a special thanks to all those who have donated to us over the past few days, without you we wouldn’t be able to move through this period of load troubles, hopefully we can return to normal service soon! We’ll have a proper status page online soon allowing you to see exactly what problems happened where and how we’re resolving these. 

As always you can follow us on Twitter for updates @redstonewire and I am available via email (sam@redstonewire.com) to either help with your queries or redirect you to the correct team member.

Forum temporarily disabled

After switching to the new set of servers this morning we’ve noticed an issue with posting on the forum, we’ve temporarily disabled the forum until this issue can be resolved. 

You can follow us on twitter @redstonewire and we’ll report when everything is back online. Thanks for bearing with us while we work through these problems, we’ll hopefully return to normal service later today!

Minecraft wiki remains online and operational. 

update: We’re back online.

Wiki and Forums are BACK!

Alright, after almost 24 hours of downtime, we’re back. New server is chugging along AMAZINGLY, and we’re pushing almost 250 requests per second.

One little word of advice. Next time your looking at a munin graph and you can’t understand why users are getting 12 second load times when your only using 40Mbps, remember.

HALF DUPLEX IS A BITCH.

A day in the life of a RSW SysAdmin…

Okay, here’s the deal.

Once the previous servers started dying horrible, horrible deaths due to minecraft.net going down, and us basically taking the brunt of it, we ordered a new server. This one is a dual proccessor, Xeon 5520. That’s 16 cores. It’s also got 12 GB of DDR3 RAM.

This is destined to be our frontend webserver, while the previous box, a single core xeon 3460 (8 cores), with 4GB of DDR3 will be migrated to a dedicated database box.

The servers were delivered to us around 2AM PST today. Aera and I both worked feverishly through the morning to get all the configuration files setup, security in place and DNS migrated. Everything was on track, and it looked like we were about ready to go around 11AM PST.

Then the shit hit the fan.

SSH died. Server rebooted, and we logged in. It died again.

I logged in via the private network interface, which has KVM, and watched the startup process. AH! There it is. /dev/sdb throws a kernel panic when mounting.

We called the datacenter, and had them replace the drive. Same thing.

We called the datacenter again, and had them replace the raid controller and the drive again. Same thing.

By this time it was 2PM PST. I’d had enough, so had aera. we did an OS Reload.

The OS reload had some issues, and had to be manually restarted SEVERAL times with a technician at the datacenter.

As soon as the reload was done, we attempted to mount /dev/sdb again, and wouldn’t you know it.

IT FAILED AGAIN.

Technician was onsite this time, and we performed a chassis swap.

This is where we are currently at.

Through all of this, I would like to state that Softlayer (our provider) has been AMAZING. They have always been willing to sit with us on the phone and go through every nuiance and details of the system. 

All sites down

Yesterday, we were looking through the logs, and noticed that we were very quickly hitting our limits on the current hardware.

We made a plea via the forums for help with funding a new server, and it turns out it wasn’t quite soon enough:

That’s the graph for JUST the wiki. As you can see, the bandwidth increase is insane! We’ve got the new server installed, and it’s being configured as I type this. Keep watching this blog for new updates.