Interesting post from WoW general forum.

[Guest]

http://forums.worldofwarcraft.com/thread.a...p=1#post1907702

Carne

This info was given to me by a former employee of a very large internet backbone server group. which shall remain nameless, but I'm sure the people who know who I mean allready are aware of this problem, but don't want to discuss it.

I don't know how long this will stay up before being deleted or locked, but here it goes.

The biggest reason why certain relms and servers keep crashing, and certain changes effect servers almost randomly, is not entirely blizzards fault. They use major backbone companies to host the servers used for this game- not their own proprietary servers they used for battle.net, though they did outsource allot of the b.net servers too, later in b.nets evolution. But I digress.

Blizzard as a company does not have the nationwide and worldwide infrastructure to completely control the internet access and server networks needed to make this game work. They contract other telco companies to handle alot of it for them. ALL mmorpgs do the same, do varying degrees.

The problem with blizzard is that they are in a contract with certain companies they cannot break, which includes hardware and bandwidth limitaions.

From what I was told, they optioned for growth at a certain rate and at a certain cost. They have been renegotiating for more mandwidth and control, but it's costing them dearly. They CANNOT discuss this openly in any way as it could be construed by the parties in question as being slanderous or libleous in certain contexts. Rather than take the chance of someone slipping up and saying a little too much at one time, they addopted a 100% non-disclosure policy on these issues.

Once any side gets lawyers involved, any and everything can and will be used against you. They also don't want to risk 10.000 angry gamers floding the internet companies with hate mail or spam about the issue. Blizzards contracts call for them to be the responsible party for customer contact in this situation.

On a related note, I was also told that the day to day costs or running the servers is immense- to the tune of several million dollars a month. As technology improves, they can expand the servers and increase the numbers of active servers and decrease the lag and technicle issues involved while keeping the costs about the same as allready contacted for, or even less.

One big factor in the cost I was surprised to hear was what my friend refered to as the "defense budget"- the cost of protecting the servers themselves from direct attack via the net. They are assaulted constantly by hackers who attempt to crash or hack in to the servers, either maliciously or to track the info flowing both ways on the networks. Its not a smal feet to protect them, and still allow the game to flow at all, let alone smoothly. The figures discussed put the wow servers at about the same target level as the pentagon, if not more so, in the attempts to hack in to, moslty my rank amatuers, but some serious damage has been done by pros as well.
"
I dont think blizzard, or any other mmorpg company really wants to dicuss the 2nd point, it being the dirty little secreat hiding out in plain sight no one wants to talk about. Again, loose lips sink ships, and if they did give out anything close to detailed info on how or when the servers were being attacked, it would invite other to try the same.

Well, hope everyone takes this for what it is worth.

And before anyone asks, blizzard certainly knows who I am, but under no circumstances will I divulge the source of my info, to them or to anyone else so dont ask."









having played lots of MMORPs I find it really surprising that a successful on like is running so poorly this far it. Normally success lets "throw money at the problem" and gennerally that works. WoW performing like it does at this point leads me to susppect this analysis is largely true. Especillly the part about scalability.

[Malakar]

On a related note, I was also told that the day to day costs or running the servers is immense- to the tune of several million dollars a month.

If they're referring to anything other than the ISP costs, that sounds like utter BS to me. I see no major legitimate costs for running servers after the hardware is bought and the system is fine tuned.

One big factor in the cost I was surprised to hear was what my friend refered to as the "defense budget"- the cost of protecting the servers themselves from direct attack via the net. They are assaulted constantly by hackers who attempt to crash or hack in to the servers, either maliciously or to track the info flowing both ways on the networks. Its not a smal feet to protect them, and still allow the game to flow at all, let alone smoothly. The figures discussed put the wow servers at about the same target level as the pentagon, if not more so, in the attempts to hack in to, moslty my rank amatuers, but some serious damage has been done by pros as well.

Sensationalism... and throwing money at it isn't going to do much.


On the general topic of outsourcing servers, the idea seems absolutely ridiculous to me, especially when contracts are involved. I'm not saying Blizzard didn't do it, I'm saying it's really stupid if they did.

[Kevin]

If they're referring to anything other than the ISP costs, that sounds like utter BS to me. I see no major legitimate costs for running servers after the hardware is bought and the system is fine tuned.
Sensationalism... and throwing money at it isn't going to do much.
On the general topic of outsourcing servers, the idea seems absolutely ridiculous to me, especially when contracts are involved. I'm not saying Blizzard didn't do it, I'm saying it's really stupid if they did.
[right][snapback]70930[/snapback][/right]

Yeah, outsourcing makes no sense. Though the person very well may have meant colocating which is probably what really happened.

It's plausible that server problems are ISP related. Of course Blizzards network engineering and network coding track record isn't the best either. Since each server is a cluster in itself there is a lot of network coding and set-up that has to go into this.

Another possibility really is bad server hardware. I still shudder when I think about the first couple of Dell servers we got in here for our exchange servers. We worked with Dell for months trying to figure out the SAN failures, the network issues, the crashes. Since it was an exchange server there was also the work with MS and the fights between "Its the software! It's the hardware!". We eventually had to scrap them and started running our mailservers on whatever hardware we could throw together. Went from the 2 Dell servers that server the whole campus (though not very well!) to something like 25 homemade boxes. The Dells did work great without any problems at all up to a certain loading level. Then they had hardware issues. Pretty much every piece in the boxes were replaced during the months of testing. We find out later that we had servers number like 5 and 8 off the Dell assembly line for that model number. Other locations soon started having other problems with them as well. Dell did not have another equivalent model server that would really suite our needs at the time and since they are a business and weren't going to give them too us new ones and we couldn't afford to buy anything else anyway (we literally had some old desktop systems running as mail servers when we finally decided to cut the losses). Other sites had similar problems with the same servers and others were running higher loads than we were and had not issues at all. Hardware really can be that squirelly. Those old Dells (since Dell would not take them back) are now running as non critical FTP servers off site and are highly under utilized for what they could do, but won't.

We have some Dell servers again now and are running a lof of VMWare servers on them they are rock solid and doing a lor more than the flakely ones that pretty much were rebuilt from the inside out several times. Hardware really can be that squirelly and it really can take a long time to track it when there are a lot of other factors. And it really can take a lot of money to just kill the old and replace with new. It still doesn't mean my patience isn't running thin, and it still doesn't mean they couldn't have handled things better. I know we could have done better than we did with our situation here, but I do have a tendency to cut them some slack.

Oh well that's enough of a ramble.

[Guest]

I agree that security babble was a non-issue.

Yes its a cost, but its cost that all MMORPs have. It really has no bearing on why WoW servers are worse than other many other MMORPs.

[savaughn]

If they're referring to anything other than the ISP costs, that sounds like utter BS to me. I see no major legitimate costs for running servers after the hardware is bought and the system is fine tuned.[right][snapback]70930[/snapback][/right]

Mal, that's the funniest thing I've heard in the last week! I run these numbers for a living. You're talking about servers that are running 24x7 in a high stress environment under loads that stagger the software and bring it the systems to their knees. Minimum support costs are probably around 1/16th a head (figure $100000 cost per head count) per system, with a substantial internal cost multiplier because of the 24x7 support.

And that's before you factor in any engineering costs. Servers don't support themselves. Ones under the kinds of loads we're seeing on the high pop WoW servers, even more so. Every server roll out I've done in the last four years had far more costs in support/expense than they did in capital.

The hardware cost is the cheap part. Especially when you capitalize it over four years anyway.

[lfd]

You're talking about servers that are running 24x7 in a high stress environment under loads that stagger the software and bring it the systems to their knees.
[right][snapback]70941[/snapback][/right]

Then they're woefully underspecced for the tasks they have to perform.

[Bolty]

Interesting read, and thanks for posting it - but unfortunately, with absolutely no way to verify the post and its info, it's really all moot.

I personally don't buy that it's the backbone - the servers just can't seem to handle the stress of loads of people in one place. The warning signs were there in plenty during beta, when some of the first ever PvP raids would regularly crash the server. I was involved in one of those, maybe even the first one ever - hard to verify - a Horde attack on Stormwind that brought the server down. Twice. It was a useless attack anyway, since all the guards were level 90 back then. :)

The game has a really hard time handling a bunch of players in one place.

-Bolty

[Guest]

As some one on a "bad"server I think there is more than that.

I suspect that even the "good" servers could crash for the reason you mentioned. Every MMORP I ever played except Planetside tended to crash or fail once you put several hundred people fighting all in each others LOS. PlanestSide only avoided this by very minamal graphic variation and have simple combat interactions.

This is for the fairly obvious reason that every actions from a character has to be relayed to every other character that can see it in combat. This makes the server load increase expodentially as plyers bunch together.
Of course this problem can be lreduced with better coding or more powerful servers.


The "bad" servers though have issues spreading accross the whole server. Some of the bad server only have medium populations even but they seem to be sharing resourses whith high population servers. Sometimes its the item server that lags on these clusters while the others appear to work fine.

Im not saying that the quote I linked is correct, I am just pointing out that I disagree with your reasoning.

[Lissa]

Sensationalism... and throwing money at it isn't going to do much.
On the general topic of outsourcing servers, the idea seems absolutely ridiculous to me, especially when contracts are involved. I'm not saying Blizzard didn't do it, I'm saying it's really stupid if they did.
[right][snapback]70930[/snapback][/right]

Wrong, outsourcing servers and services does make sense in a business standpoint if it would cost you more to host the server and services yourself. What Blizzard is involved in is what is known as an ASP, Application Service Provider. (I happen to work at one of the largest Community Healthcare Clinics in the nation for disadvantaged peoples and we host a Medical database for not only ourselves, but several other Community Healthcare Clinics in Arizona and if my Boss has any say, possibly expanding to included other CHCs from New Mexico, Colorado, Utah, Nevada, and California.) The reasons to go with an ASP is that the cost of operating the servers and program would be more costly than to lease the services from someone else. (We spent $250k initially to buy the servers, software, and connectivity for our own remote sites along with out main clinic, and this is for about 500 connected users at one time, if you figure that a MMORPG has 10 times that, the cost could easily climb to several million dollars, and we have spend more money purchasing more servers and connection to adequately host other CHCs in Arizona, we have probably spent an additional $100k to do so, meaning out datacenter is probably on the order of $350k possibly as high as $400k for 500 users connect at one time.) So the reason Blizzard is outsourcing becomes readily apparent when you look at the potential costs that they may have to outlay between hardware (one time capital cost) and continued costs of manpower and maintainence.

[Malakar]

My point was of quality not cost. Blizzard makes plenty of money to cover extreme costs.

I assume Blizzard writes their own server code, if not that's pretty sad. It's better if they set up their own hardware so they can design their systems as one coherent whole.

[Malakar]

Nice numbers, but you didn't say what is causing the costs. Are you saying there's a lot of hardware failures, or are you saying you have to pay lots of techies lots of money to "support" the servers?

Edit: If you could point me in the right direction, I'd like to learn what makes this technology so inherently different from what I'm familiar with. Please don't simply say "high stress", I don't buy that by itself.

[Lissa]

My point was of quality not cost. Blizzard makes plenty of money to cover extreme costs.

I assume Blizzard writes their own server code, if not that's pretty sad. It's better if they set up their own hardware so they can design their systems as one coherent whole.
[right][snapback]71022[/snapback][/right]

And you're still wrong. It has nothing to do about quality and everything to do about cost. Do you know what a Network Administrator gets paid that knows what they're doing? Network Admins that do this kind of high end work typically are Cisco CCNEs, the highest and most difficult certification to get in IT (it usually requires about 5 to 10 years of working with Cisco gear constantly to have even a shot at passing this test on the second try, yes, SECOND try, it has a 85% failure rate for first time takers and these are people that have spent years working with Cisco gear). As such, these people are typically payed on the order of 85k+ minimum a year (more typical salaries are around 120k a year). Now, do you think Blizzard has the ability to pay a team of 5 to 6 people (cause you gotta have atleast 1 on staff at all times) a combined total of 500k to 600k a year? And we're not even talking about the other operating costs that go into a datacenter of this magnitude that Blizzard requires with the number of servers it has. Simply put, this has absolutely nothing to do with quality and everything to do with cost.

[Malakar]

Yes, I think blizzard has plenty of money to do whatever's necessary.

And quality has everything to do with it. Without quality, they'll lose their subscriptions in time. Lower cost isn't exactly that great at the expense of revenue.

As far as the cost thing goes, what exactly makes it cheaper to outsource anyway? Ultimately the bottom-line cost must be paid somewhere along the line, and an additional company has its own overhead.

[Swiss Mercenary]

Lower cost isn't exactly that great at the expense of revenue.

Nit: Since nobody significant is really voting with their dollars, revenue isn't probably taking too big a hit.

[liffuie]

Yes, I think blizzard has plenty of money to do whatever's necessary.

And quality has everything to do with it. Without quality, they'll lose their subscriptions in time. Lower cost isn't exactly that great at the expense of revenue.

As far as the cost thing goes, what exactly makes it cheaper to outsource anyway? Ultimately the bottom-line cost must be paid somewhere along the line, and an additional company has its own overhead.
[right][snapback]71030[/snapback][/right]

Well think of it this way, if they did it themselves they'd have to increase their employee load signifigantly for the people to run the servers, then support people, managers etc, paying there wages, benifits.

thats nothing to say about buying all the hardware and building facitlities, spread over who knows how many seperate physical locations across the us.

How much would it costs them to actualy just do the start up costs of that magnitude? you'd prob need a corporation 4x the size of blizzard(i'm just guessing here) to start that sort of nation wide infrastructre lonewolf

[Tharn]

As far as the cost thing goes, what exactly makes it cheaper to outsource anyway? Ultimately the bottom-line cost must be paid somewhere along the line, and an additional company has its own overhead.

Outsourcing can be cheaper when you're outsourcing somehting what is not core business of your company and therefore there are other companies who can utilize:
- experienced experts on the subject
- economies of scale
However you can in general outsource only something, what's usualy done elsewhere too.

Blizzard is game development company so their core business is making game design and content. They could have bought some graphics engine for WoW, sure. However, I'm afraid this network design is too proprietary for each MMOG company to be able to work with some standard tools or even outsource it.

There's also the issue that when systems are getting under heavier load, unexpected bugs start to surface, which is I guess plays big part in WoW problems.

Btw, Malakar, you must pay maintenance fee for both hardware and software you're using on the servers, if you ever want some support (that's including fixing bugs in software, btw). You can count with 5-20% annualy of purchasing price (depends highly on what purchasing price you negotiate too), where some notoriously known companies are very bastardly.

I however wonder why, if this is caused by some company's product, Blizzard didn't draw their CEO by an ear and made a nice big press conference.

[savaughn]

Nice numbers, but you didn't say what is causing the costs. Are you saying there's a lot of hardware failures, or are you saying you have to pay lots of techies lots of money to "support" the servers?

Edit: If you could point me in the right direction, I'd like to learn what makes this technology so inherently different from what I'm familiar with. Please don't simply say "high stress", I don't buy that by itself.
[right][snapback]71023[/snapback][/right]

I'd have to know what kind of systems you're familiar with to answer, but in general what we're talking about is streamed business logic requests from hundreds of thousands of simultaneous sources, virtually every single one of which needs to be parsed, turned into a back end database check, and the returned. This is going to be on the order of millions of events per second as your base load. This is then made substantially more complex by stream interaction. For example, if you have 40 people in visible range of each other, the stream complexity goes up by the square of the LOS, or 1600 times the front end server processing. Also, the database can be driven from multiple directions by simultaneous events (say, two people trying to mine the same node), meaning that you can richochet through business logic/dbase loops with event interrupts. (Loot lag seems so simple to fix when viewed client side but the server side part of that is down right frightening.)

I'd call that pretty high stress. By comparison systems like Google's character match index queries are snoozing.

[Malakar]

Yes you are quite right. But if this situation were worse, say, nobody could log in at all for 3 months, how many of us would still be paying then?

[Malakar]

Well I guess I was not clear. I didn't mean to dispute that the servers are high stress, I think it's pretty clear that they're under extremely high stress. I meant to say I don't see how that naturally requires high ongoing costs, after you have built and tweaked the system.

What I'm trying to say is that yes, it's a massive and highly complex system. But eventually, assuming no core system changes are needed, they should be able to get it running pretty smoothly with little interaction needed.

I'd like to learn what makes this technology so inherently different from what I'm familiar with.

Let me try to explain what I meant by this.

I've dealt with servers that are nowhere near the same ballpark as the capacity or scale of MMO servers. But they're still computer technology, so they should have some basic similarities. I'd expect that their hardware does not simply die when put under loads that are too heavy, rather you'd get serious lag. I'd expect that their software may experience extreme bottlenecks too, which would also cause lag. I'd expect some crashes. But that's in the beginning. I'd also expect for the system to be able to be smoothened out eventually, to the point that its maintenance can be relatively automated assuming no core changes.

[Malakar]

- economies of scale

OK, that makes sense.