In many organisations there are processes in place to manage change, depending on the size of the organisation the change control process can be very in depth and frustrating or can be non-existent. I have experienced many people talking about change management process as if it was the root of all evil. What I would say is whatever organisation you work for you should embrace change management.
So what do I mean when I say personal change control; irrespective of the organisations change control processes you should have your own change control process to follow. Your process should cover the following areas at the very minimum:
- What is being changed
- In case of emergency
- Is it working (Pre-change)
- Detailed script for change
- Is it working (Post-change)
- Details script for back out
What is being changed?
Okay if you are just changing one device, then maybe it is not so important, if you are making changes to a number of items then it will become more important.
- to tick each item off once it has been updated.
- if there are network issues after the change then having a visual list may assist in quickly homing in on the issues, even if it is to eliminate your change.
- if you are not around when an issue occurs, then a†colleague†can easily see what has been updated and can rule in or out whether is need†further†investigation.
It is very important that you have some recognition of what impact your change may have on other things that it is connected to. If you know your change is going to reboot a network switch then it is a good idea to let the users know who are connected to the switch that is going to be rebooted.
Or perhaps you’re making a QOS change in the remote site and if something had to go wrong you may expect some voice quality issues, then it is good to know this upfront.
The reason for considering this upfront is two fold:
- To make users aware of any issues ahead of time.
- For you to have an idea of areas that may be impacted, so when a trouble ticket arrives the source of the problem can be identified quickly.
In case of emergency
This is extremely important when you do not hold the keys to all the systems, you might have to call out another engineer to reboot a server or you need to gain access to a computer room, so you need to number for the server team ,security etc . What you don’t want to be doing is trying to look up the telephone numbers, or call someone who might have the number you’re looking for. Have a list of important information to hand. e.g contact numbers, vendor support numbers, service contract numbers …
Is it working (Pre-change)
Before you start any change you need to make sure that the systems are operating as you would expect, this is particularly relevant if after the change you are expecting same results.
One thing that drives me insane is when a person who is ìassistingî the change process by providing testing, they will come up to you near the end of testing and tell you that application X is not working, you will try and resolve the issue only to be scratching your head, then you go back and ask whether or not it was working before the change and they respond ìI don’t knowî.
Detailed script for change
When it comes to performing changes I don’t want to have to invest any time thinking about the changes that are being made, I would prefer to save my brainpower for troubleshooting any issues that may arise as a result of the change that I have not foreseen. Now I can appreciate that we do not all have test labs to create detailed scripts for changes, but even in the absence of test labs you should always make an effort to write down each configuration line or each process step to the best of your ability, by physically going through the process it forces you to think through the change in advance and therefore alleviating the need to use excessive brainpower during implementation.
Of course if you have a test lab then that’s fantastic because you can simply use cut and paste at the time of change.
Is it working (Post change)
Now depending what type of change your making will depend on your post change checks, some post change checks just mean you retest the pre-change checks, some changes may introduce new functionality, then your post change checks should do their best to demonstrate that that functionality has been implemented, also any changes to a live environment should also verify that the systems are running as before.
Detailed script for a Backout
Now at the beginning of almost all changes I do a network equipment I will back up the configuration to a TFTP server, at this time after working with networking equipment for over 10 years I cannot recall restoring a configuration in an operational environment from a TFTP server when the change has went wrong, this is truly the last gasp step. Most changes will need to backed out by un-doing previous commands, again to remove brainpower required to undo the change, then have the back out scripted.
There are volumes of information written on change management processes and different standards from ITIL, and Sigma 6.this post is not meant to replace any of these existing processes, †the purpose of this post is to identify, in my opinion the process every network engineer should be following, irrespective of the change management processes in place in the organisation where the change is being made.