In a recent article, Matthew Norwood Programming Bad Performance talks about identifying possible causes for the problem of a badly performing web service. This is his list :
Possible causes to consider:
- Remote end of the connection
- Internet connectivity
- Firewall
- Intrusion prevention sensor/Content filter/Other security hardware
- Router/Switch/Load balancer problem on the internal network hosting the site
- Server hosting the site
- Web server software on server hosting the site(ie IIS,Apache)
- Web site code (ie HTML,ASP,JScript,CSS,XML)
It’s a good article, and he is making good points. But I want to take it a step further and talk about why this approach can make it look like a networking problem, when it was much more likely to be a server problem.
Perception is Reality
One of the hardest lessons I’ve ever learned is that PERCEPTION IS REALITY. That is, whatever a person sees of a problem, and the way that THEY see it, is the only REALITY that they know. Sounds simple.
The problem is that many people
- do not think
- do not understand
- are not equipped to comprehend
- do not ask
- will not ask
Lets assume that you can’t fix these problems ( sadly, not a bad assumption ) in real life. Therefore, as an engineer, you need to understand how to hack other peoples brain such that they†perceive correctly.
So back to the list. In this list, five out of eight problems are networking problems. Therefore, to someone who has no comprehension of technology, this list suggests that 60% of the problem is networking related. Therefore, the network is a big problem.
No joke, that’s what people think.
So the problem with this list, is that he focusses on the complexity of the network and not on the complexity of the server systems. And while the programmer claims to have investigated everything,……..
My List
Here is the list that I would have drawn up:
- Physical Server performance hosting the site
- Operating System performance of the servers
- Database performance issues 7 Middleware performance
- Web server software on server hosting the site(ie IIS,Apache)
- code problem or bug
- coding performance issue
- HTML content structure (ie HTML,ASP,JScript,CSS,XML)
- Invalid testing showing a false error
- Router/Switch/Load balancer problem on the internal network hosting the site
- Firewall / Intrusion prevention sensor/Content filter/Other security hardware
- Internet connectivity
- Remote end of the connection
In this list, the perception that goes out is that the network can only impact a very small part of the overall system. That, in fact, is the correct perception.
Occam’s Razor
Occam’s Razor is the scientific principle that the simplest answer is always the most correct answer.
ìAll things being equal, the simplest solution tends to be the best one.î
Applying Occam’s Razor to this problem: If you compare complexity of the network infrastructure to the complexity of the server infrastructure, then the problem is most likely to be in the server because that is the simplest answer.
The location of the most complex element in a system, is where the problem is most likely to be. That’s the simplest answer.
You must correctly comprehend that the server platform is made up of hundreds of separate pieces of software, and the application platform, the programming language, the programmer’s implementation are all equally formed of dozens of elements. The applied interaction of the all these elements is far more complicated than the interaction of the network equipment.
Where this Matters
Presenting the first list to management or PMs is correct and focusses on the elements that you can control. But it fails to recognise that the complexity and the problem is most likely to not be the network. As a network engineer, you tend to think in terms of what you know, what you perceive ? That’s your reality.
But you have failed to comprehend that presenting your reality to others, makes that reality true in their minds. So the Server guy sits there going “we’ve checked everything” and you are saying “well, it could be these things”. That’s how networks get blamed for everything. You got to careful about designing your response to these issues.
And that’s how your salary review gets hammered.
The EtherealMind View
What I’m describing here is fundamental hacking of your presentation process. You need to carefully design your response in these situations to ensure that you create a balanced perspective of the problem. You need to be careful not to present your understanding of the problem because your vision leans into your comprehension – that’s your perception of reality. Think about the bigger picture, develop a comprehension of the overall eco-system in which you operate. .
The Network is inherently more reliable, better performing and less complex than ANY server platform. Make sure that you present that reality when you are meeting to resolve problems and make that the perception for other people’s reality.
You’ll spend less time solving someone else’s problems that way and get to go home on time, we all want that.
Greg,
I don’t disagree with everything you have written. Yes, the list I provided looks at it from a network perspective. However, I can assure you that if this were a bigger issue involving multiple groups, the list would have been just as long as yours. Although I wrote the post as if it were a structured dialogue that occurred over time, it was really about 5 of us standing around a white board talking it through start to end. Remember that I write things from the point of view of a corporate IT person. I will work with all of these people for a considerable time, so I won’t necessarily have the liberty to be as blunt as I would if the relationships were more temporary. I can use that to my advantage though, as the more I work with them, the more they will come to believe me when I tell them the problem is not the network. We all trust “known” entities over “unknown”. Unless of course you are a consultant. Everyone believes them. Don’t ask me why. I have no idea.
Within a few minutes of talking with the application engineer, I knew this wasn’t a network problem. Every ounce of geek within me was screaming “systems issue”. There’s two ways I can go with this. First, I could flat out state that this was not a network problem. Once I was told about the MTU issue, I knew it wasn’t any of the gear I manage. When the problem occurs before it hits the fiber or copper, I could care less about it. That’s not within my realm of responsibility. Second, even though I know it isn’t a network problem, I want to help the app engineer reason out where the problem resides. I want him to see the logic in tracking down the source of the problem and let him realize that the issue lies within the code or the box the code resides on.
The problem with taking the first approach is that you better be right 100% of the time. We have all been in situations where we try and explain something like MTU to non-network people and they just aren’t getting it. I try and remember that I live in this networking world 24×7 and they do not. While I wish I could just emphatically state that the problem resides elsewhere, that doesn’t always work. If I happen to be wrong, there’s going to be all kinds of problems. It isn’t my pride that I am worried about wounding. It is the reputation of my group.
If I take the second approach and let others come to the same conclusion, it takes up more of my time, but in the long run, the network team will fare better. Over time, as you work with more and more people using the second method, they start to look more and more toward things other than the network. That’s not always the case, but I have seen quite a few people change their mindset from “it’s always the network” to “it’s not the network”.
I don’t think it is too out of line for me to suggest that network people usually have to have a basic understanding of all other IT disciplines(servers, storage, database, security, applications). The inverse is not always true. Is that fair? Not at all, but every single company I have worked for seems to have that unwritten standard. That’s not to say that other groups don’ t have people that understand networking. They do in some cases. I have found that by taking a little extra time to educate people about how the network functions, it pays off in the end. If I build a reputation over time of working with other groups and it never ends up being a network issue, they stop looking at the network as the first thing to blame when something goes wrong. There are always exceptions to the rule.
My company rolled out a fairly big application last year. Initially, when there were problems, the network was the first piece to be blamed. My group would respond and we would provide our findings to management. It was almost NEVER the network. As we “proved” the network over time, less blame was thrown our way. In the event someone tried to lay blame on the network, my manager would simply ask “How many times in the last 10 incidents or so has it been the network?”. This usually caused the other group(s) to look in another direction(systems, application,etc).
I guess I’m trying to “draw an long bow” from your article. I don’t doubt that you personally handle the situation correctly. What I’m attempting to illustrate is that you need to be conscious of how people perceive you and your work. I was really using your article as basis of how some people get it wrong by only talking about what “they know” rather than considering the entire problem.
PS: It’s worth remembering that Consultants don’t work on problems, we “attend” them to provide advice. That’s why we are always right.
I tend to line up more with Greg on this one. I manage both complex edge infrastructures for Internet facing applications as well as Internal applications on the WAN. It doesn’t matter whether you are talking about a router, a switch, a firewall, a load balancer, a server, an application, a database, or whatever. These days everything is complicated. Here’s the difference. Networking gear (including FWs and LBs operating at L3/4) tend to show their complexity during implementation. Once an application is deployed, usually the L3/4 infrastructure stays relatively static unless changed to support enhancements in the application. Yes, we may upgrade a router but our change cycle is much less frequent than what goes on at the server level. On the server, even after deployment, you constantly have M$ Tuesdays (you also patch Unix. Don’t lie!) as well as application patches, driver updates, features, enhancements etc. That’s why app/dev is a never ending process. And let’s not forget AntiVirus or other non application related items on the server (CSA anyone?) that can be a factor. These items are just as fluid post implementation as they are prior to. So when I dial into a bridge and everyone says the sky is falling and “nothing has changed” I tend to take it with a smile and start by interrogating the problem as opposed to blaming any particular group. Most times, it’s not a network issue. Notice, I didn’t say it’s NEVER a network layer issue. Just mostly.
We also use application layer firewalls for the obvious additional security. The side perk is that these are great products for keeping your programmers in check. Many a bridge call have ended very quickly after “nothing changed” but there are new paths being blocked at the application firewall. Hmm…. Looks like the vendor wasn’t exactly forthcoming with the details of the last build that you casually threw into PROD without a change control doc eh? Sucks being you.
Lastly, the greatest tool out there is still a good old fashioned wireshark capture or just a headers capture from tcpdump. Sniffer captures do not lie. They can be misinterpreted or taken out of context but the do not lie. Nothing is easier for me than taking a snippet of a wireshark capture and pasting it into an email thread and telling the vendor his application is doing XYZ and to call me back when it’s fixed. It usually kills the thread. They may not like me for it. But they’ll respect me.
It seems that the onus, or burden of proof, is often on the network engineer to figure out just what the problem is, since, as you say, the network is this mysterious entity that most people don’t know anything about. I have found two types of server admins in my experience:
The ones that assume it’s their problem and never think about the network, but then it proves to be a network problem (often when attempting to establish communication between a service network and an internal network).
The ones that assume (along with all the users, usually) that it is a network problem and very quickly start pointing fingers
I have found that over the years I have learned a lot more about efficient database programming, analyzing server cpu, memory, and disk utilization, etc, than I ever wanted to, as a means to simply find what the real problem is. Greg points out that proving it’s not the network, or at least casting plausible reason to believe it’s not, will get you home on time. I agree with that, but in a world of people saying, “It’s not my problem” and washing their hands of it, I want to put forth a little more effort to get to the bottom of things.
Something else that I have learned is that humility is really important. If you start acting cocky and the problem turns out to be yours, you really look like, and are, a jerk. Taking more of a “Let’s figure this out together” attitude is much more likely to lead to success and team unity in the end. The real art is being able to foster and maintain that attitude when other people seem to be out for blood.
My 2c