Interesting post from WoW general forum.

[Lissa]

Well I guess I was not clear. I didn't mean to dispute that the servers are high stress, I think it's pretty clear that they're under extremely high stress. I meant to say I don't see how that naturally requires high ongoing costs, after you have built and tweaked the system.

What I'm trying to say is that yes, it's a massive and highly complex system. But eventually, assuming no core system changes are needed, they should be able to get it running pretty smoothly with little interaction needed.
Let me try to explain what I meant by this.

And that's where the fallacy comes into play. You see, anytime you update code on a system, new bugs are introduced because either the bug was lying dormant and required some new code to come out or the new code clashes with some aspect of the running code and brings about the bug. Then we have other issues that crop up due to advances in the workstation hardware that can bring about still other bugs. In essence, the more you add to the system, the more complex it becomes and the more complex it becomes the easier it is to break somewhere which then requires manhours to go back and find what broke and fix and *hope* that your fix doesn't introduce another bug somewhere else.

I've dealt with servers that are nowhere near the same ballpark as the capacity or scale of MMO servers. But they're still computer technology, so they should have some basic similarities. I'd expect that their hardware does not simply die when put under loads that are too heavy, rather you'd get serious lag. I'd expect that their software may experience extreme bottlenecks too, which would also cause lag. I'd expect some crashes. But that's in the beginning. I'd also expect for the system to be able to be smoothened out eventually, to the point that its maintenance can be relatively automated assuming no core changes.
[right][snapback]71090[/snapback][/right]

That's another fallacy. The technology behind clustered servers is very much more complex and it isn't simply similar in anyway. The main reason for clustering is to increase uptime and add some fault tolerance, but even then, something can go awry (take the data center being hit by the tornado during beta). Due to the increased complexity, you have a lot more to worry about and need a lot more expertise in dealing with them than just being a normal server jockey. Calling it simply, the same technology, is not truly understanding the complexity of these beasts.

It gets even more complex in clustering depending on how your cluster is configured too as you could have several servers active at the same time while you have other servers that are passive waiting for one of the active servers to die. I was frankly amazed at how quickly they were able to bring the servers back up during the beta when they were hit by that tornado. Replacing servers in a cluster when one of them dies isn't easy and you hope and pray that while you are repairing the downed machine something doesn't happen to the other machines in the cluster (cause then you may be looking at rebuilding the cluster from scratch, a job that can take several working days). Clusters require a great deal of care to make sure that they continue to run properly.

[Malakar]

... You see, anytime you update code on a system, new bugs are introduced because either the bug was lying dormant and required some new code to come out or the new code clashes with some aspect of the running code and brings about the bug.  Then we have other issues that crop up due to advances in the workstation hardware that can bring about still other bugs.  In essence, the more you add to the system, the more complex it becomes and the more complex it becomes the easier it is to break somewhere which then requires manhours to go back and find what broke and fix and *hope* that your fix doesn't introduce another bug somewhere else.

You go to explain how bugs/problems are introduced when you make changes to a system, in response to a quote which contained "assuming no core system changes are needed."

The technology behind clustered servers is very much more complex ...

Irrelevant. What is being discussed is whether or not these servers can eventually operate smoothly with little need for babysitting. That has absolutely nothing to do with how complex the system is. How complex the system is relates to how long it takes and how hard it is to make it smooth, not to the possibility.

Of course natural disasters like tornadoes change things, but they'll destroy any system anyway, and are kind of out of scope.

By the way, I'm getting the feeling of an implication that normal servers/PCs aren't complex, which is absurd. Perhaps you might want to learn assembly, or maybe even delve into hardware design.

... and it isn't simply similar in anyway.

I was under the impression that these servers ran on silicon chips.

[Lissa]

You go to explain how bugs/problems are introduced when you make changes to a system, in response to a quote which contained "assuming no core system changes are needed."

How much do you understand about programming? From the sounds of it, very little. Just writing some code can cause problems with the core when you are not even trying to effect the core. Bugs occur in ways that don't make sense because a bug is a logic error, and humans are not 100% logical when they code. As such, when trying to add more value to a program, it is quite possible to introduce a new bug that effects the core program when you don't even try to effect the core program, that is the way things are.

Irrelevant. What is being discussed is whether or not these servers can eventually operate smoothly with little need for babysitting. That has absolutely nothing to do with how complex the system is. How complex the system is relates to how long it takes and how hard it is to make it smooth, not to the possibility.

And as I have said, Clusters are far more different beasts than a standard server. If you had ever built one or worked on one you would know this. As something becomes more complex you have to watch it more carefully which does require more expenditures in resources to make sure the system is functioning correctly. Thus you will be able to sit back and just not bother with the system as minor fluctuations in the system can have dire consequences due to the complexity and you can never just walk away from a system like this as those minor fluctuations can have ripple effects that could eventually bring down the system. These types of systems need constant survilance to make sure that they will not go down.

Of course natural disasters like tornadoes change things, but they'll destroy any system anyway, and are kind of out of scope.

By the way, I'm getting the feeling of an implication that normal servers/PCs aren't complex, which is absurd. Perhaps you might want to learn assembly, or maybe even delve into hardware design.
I was under the impression that these servers ran on silicon chips.
[right][snapback]71110[/snapback][/right]

No, you do not understand. PC and server are complex, but clusters, servers working in concert to increase uptime and fault tolerance are like monkey to a human. Both the monkey and human are complex, but the human has even more complexity to it, thus the same thing with clusters to stand alone servers. There is far more going on between clustered servers than there is between a server and clients connecting to it. Again, if you had ever worked with a cluster, you would understand quite well what I am talking about.

[Raven Vale]

Going back to the original post , was that a cut & paste ? There are a lot of basic spelling mistakes there , personally I think its all made up . That is only my opinion , nice post to wind people up .

[savaughn]

You go to explain how bugs/problems are introduced when you make changes to a system, in response to a quote which contained "assuming no core system changes are needed."
Irrelevant. What is being discussed is whether or not these servers can eventually operate smoothly with little need for babysitting. That has absolutely nothing to do with how complex the system is. How complex the system is relates to how long it takes and how hard it is to make it smooth, not to the possibility.

Of course natural disasters like tornadoes change things, but they'll destroy any system anyway, and are kind of out of scope.

By the way, I'm getting the feeling of an implication that normal servers/PCs aren't complex, which is absurd. Perhaps you might want to learn assembly, or maybe even delve into hardware design.
I was under the impression that these servers ran on silicon chips.
[right][snapback]71110[/snapback][/right]

Congratulations! You win the Snarky Trolling Post of the Thread award! Your large pile of victory tar will be sent out shortly.

You are accurate in that eventually the support costs for a system go down over time. A typical timeline for complex systems of this nature would be 2-4 years of development during which your ability to cope with events is negligible because you have a moving target, followed by another 2-3 years of high end production support where you can actually hammer out as much automation of your support costs as possible. After this period, overall support will be much lower. This, of course, assumes that you never upgrade the software or the hardware. Changing your back end SAN, for example, can very likely drive you straight back into development level support costs.

At the current time, we know for a fact that Blizz is in development mode. The company has made a complete overhaul of the hardware platform and has been making substantial back end changes. Considering the economics of game development, it wouldn't surprise me to find out that they were understaffed on top of that which means they probably are working hard to just keep the systems running right now with little free availability to improve their monitoring systems.

Irrelevant. What is being discussed is whether or not these servers can eventually operate smoothly with little need for babysitting.
[right][snapback]71110[/snapback][/right]

Normally I ignore this kind of redirection, but not with this level of spite in your post. This is complete misdirection. We were not talking about some distant far off eventuality. Someone estimated Blizz's CURRENT COST for support and you chimed in on how that was ludicrous because hardware runs itself and needs no care or maintenance.

I'm pulling my support experience from knowing the support infrastructure of systems that support a company of 80,000 employees and has more than 25 24x7 factories. Where are you pulling your system support cost model from?

[Guest]

Claps - nice.

I love it when people are held accountable for diversions and rhetorical garbage in threads.

[Sabra]

There are a lot of basic spelling mistakes there , personally I think its all made up . That is only my opinion , nice post to wind people up .
[right][snapback]71131[/snapback][/right]

Second that. This is just the kind of specious gossip that is grist for the mill and resolves nothing. Interesting, sure, but without verification, a total waste of time.

I've been thinking about this whole server issue and I've finally just gone back to the advice I received from my mother many years ago: Don't complain without solution.

[Malakar]

You are accurate in that eventually the support costs for a system go down over time.  A typical timeline for complex systems of this nature would be 2-4 years of development during which your ability to cope with events is negligible because you have a moving target, followed by another 2-3 years of high end production support where you can actually hammer out as much automation of your support costs as possible.  After this period, overall support will be much lower.  This, of course, assumes that you never upgrade the software or the hardware.

And that's the only thing I was trying to get to be admitted, bad wording and timelines aside.

This is complete misdirection.  We were not talking about some distant far off eventuality.  Someone estimated Blizz's CURRENT COST for support and you chimed in on how that was ludicrous because hardware runs itself and needs no care or maintenance.

After reading this I went back to read my original post, which did not come off like I had remembered it. It now makes sense how the thread progressed, and why the argument I was trying to make was being ignored at times. Sorry for the miscommunication.

That being said, FWIW I did make some clarification on what I was trying to say in the post Lissa responded to, and I was irritated at the time (largely my own fault!), which is why I responded like that.

[Malakar]

... Just writing some code can cause problems with the core when you are not even trying to effect the core.  Bugs occur in ways that don't make sense because a bug is a logic error, and humans are not 100% logical when they code.  As such, when trying to add more value to a program, it is quite possible to introduce a new bug that effects the core program when you don't even try to effect the core program, that is the way things are.

You're quite right about that. However with good coding techniques you can keep this to a minimum, and after some debugging time, most of these vulnerabilities between separate components can be dealt with. Perhaps this problem could be exacerbated a lot depending on how the team of programmers work together though. And yes I realize that this problem is naturally exacerbated to some extent by a team being required in the first place.

Also, I'd expect that at some point, content updates in WOW will be mostly data and little code. New instances, monsters, spells, etc may require code sometimes, but probably not as often.

[liffuie]

Also, I'd expect that at some point, content updates in WOW will be mostly data and little code. New instances, monsters, spells, etc may require code sometimes, but probably not as often.
[right][snapback]71184[/snapback][/right]

not realy, in EQ, over the course of its timeline, they've overhauled the graphic engine 3 seperate times. they've redone the way items are stored and logged twice. Changed mob ai that i know of twice (mostly just how they pathe, and reacte to differnt ability in relation to agro generation) also changed how how spells interact with npcs.

if your saying any of these are just little code changes, is a vast over simplification. add to the fact despite EQ's success by the 5th year mark, every programer from the orginal team to kunark xpac had left the company. Trying to do changes while not break old legacy code that may be a core function and that no one in the current team 100% understands is not exactly an exact science.

[Guest]

Claps - nice.

I love it when people are held accountable for diversions and rhetorical garbage in threads.
[right][snapback]71145[/snapback][/right]

EDIT: Ive been informed that I was misunderstood. Im refereing to rhetorical diversions - not a thread diversion.

[Malakar]

Yeah, that requires a fairly ideal situation.

[Guest]

Yest it was a cut and paste. Believe it or not other people in the world besides me make spelling errors.

[Guest]

Its very reasonable to not believe the original quote. It has flaws and isnt verified. But...

"Don't complain without solution."

That reasoning is ludacris. No body has a solution - because Blizz wont explain the problem. If someone dropped a TV on your head from a 20 story building would you say "I dont mind because I dont know who did it."?

At least make sense please.

[Ruvanal]

Yest it was a cut and paste. Believe it or not other people in the world besides me make spelling errors.
[right][snapback]71194[/snapback][/right]

It would be best if when you do a cut and paste, that you put the pasted portion inside of quote tags to show it off better. That lets most readers then know when your input starts and stops compared to someone elses writing.

[Lissa]

Yeah, that requires a fairly ideal situation.
[right][snapback]71193[/snapback][/right]

And a fairly ideal situatin will never happen with MMORPG.

[Occhidiangela]

http://forums.worldofwarcraft.com/thread.a...p=1#post1907702

Carne

[right][snapback]70922[/snapback][/right]

Thanks for the link, Ghostiger. I had to stop after a few pages, I got the gist of the discussion and a few of Crane's follow on comments. Not sure how credible the source is.

The rest of the discussion here is too close to what is going on at work for me, though I do appreciate Lissa and a few other's insights. The system we are trying to make work, content provided by a contractor for our operations, is over budget and late on schedule, but we, like Blizz, really have no viable plan B. Being in that position suxors ass, since our downstream customers come to us for resolution of "it don't bloody well work right." I have a good feeling for what Blizz faces each day with the user community being less than pleased.

Savaughn's timeline on getting a big system to settle in sure rings a bell with me, and our system is nowhere near as complex or bandwidth intensive as the dynamic environment of MMORPG.

As to cost, an awful lot of things in life cost more than one thinks. Until one gets into the belly of the beast, for example, one really does not understand the infamous $700.00 hammer. :whistling:

Occhi

[Sabra]

Its very reasonable to not believe the original quote. It has flaws and isnt verified.  But...

"Don't complain without solution."

That reasoning is ludacris. No body has a solution - because Blizz wont explain the problem.

It's spelled "ludicrous," and if you're looking to Blizz to create your solutions, you're in a really sad situation.

[Occhidiangela]

EDIT: Ive been informed that I was misunderstood. Im refereing to rhetorical diversions - not a thread diversion.
[right][snapback]71191[/snapback][/right]

*Snigger*

Thread hijacking is a well established Lounge tradition. I have hijacked many a thread, not infrequently in Net54 days to avoid starting another thread. Remember the speed scrolling days of the Lounge? :rolleyes:

I also remember when we all used to use conventions such as "OT" or "NS" or "VLT" or "VLS" or "NT" is post titles. With this forum set up, each reply does not get a title, so it is a bit trickier to telegraph a digression.

For anyone on this board to complain about thread hijacking and digression is rather droll. I replied to you, Ghost, but that last comment goes for any number of folks who have complained about digressional behaviour.

Occhi
"Varsity Letter Winner in Thread Hijacking"

[Darian]

You're quite right about that. However with good coding techniques you can keep this to a minimum[right][snapback]71184[/snapback][/right]

You'd be amazed (well, maybe not) how many developers who have great ideas and write important code don't practice good coding techniques.

Also, I'd expect that at some point, content updates in WOW will be mostly data and little code. New instances, monsters, spells, etc may require code sometimes, but probably not as often.

Actually... until I recently changed departments to work on something else after five torturous years, I admined the servers and processes which drive my company's business infrastructure (read: heavily dependent on frequently-changing business rules). Even with core code in place which allows those business rules to be changed without any actual code changes (reloads from DB), mere data changes have on occasion caused the servers to melt down, because the rules were written badly. I would go so far as to say it's not just possible, but extremely likely, that something as innocuous as an item/content update can and will wreck a WoW server at some point.

GIGO, after all.