A couple of days ago I was reading “The Ultimate Book of Heroic Failures” by Stephen Pile while waiting for a flight; it’s a very amusing book and I would recommend it to anyone, but not digressing it led me to consider and examine some extremely common examples of worst practices I’ve seen in my years of solution engineering.
Although the following are all hypothetical examples, they DO represent problems which can and do occur in any IT environment. Note: these are generic examples to protect the innocent and all come from simple human error which happens to us all.
Worst Practice 1: Hard Coded Configuration
This scenario exemplifies what can happen when IT is forced to cut corners and rush in order to make deadlines. Say you are a developer implementing some changes to database connection methods. You go into the config and you set up a change in a customer facing application. Because you’re pressed for time, you just put the UAT database details into the file and make a mental note to go back later and do it properly, utilizing whatever method you have for tracking different configurations across environments. After some time passes it goes into production and there are performance issues. Hours can be spent examining all aspects of the production database before anyone realizes the configuration simply isn’t being used and there is a small UAT database in a dark corner of the data center sweating in it’s new-found production role.Solution:
Using an ARA solution like XL Deploy makes it very simple to use placeholders to populate environment specific values. In the scenario above it is likely that the connection details for the environments would (or should) be already known to your ARA, making using an environment independent placeholder just as easy as using a hard coded value.
Worst Practice 2: Load Balancer
In this environment you have one load balancer talking to two data centers, A and B, each with two web and app servers. One of the benefits of this hot-hot configuration is at application update time, you can update one data center at a time during low load, just by taking that one out of the load balancer pool. This enables live upgrading for you site and full uptime.
Mistakes can be made however. During an upgrade in the early hours of a Sunday morning you take out site A from the pool, upgrade goes great so you add it back in and take site B out of the pool. Then you upgrade site B successfully. All good right? Not so fast, while the upgrade of the application components went great, one important last step was skipped, putting site B back into the load balanced pool. This is easily forgotten with a manual deployment, especially as this error doesn’t become apparent until the site hits full load during the day when you really need all that horsepower a hot-hot configuration gives you.Solution:
Deployment automation provides a key benefit here. Through automation we avoid mistakes like missing steps, especially when the impact of missing a step isn’t immediately obvious. If the above deployment was automated with XL Deploy, site B would be automatically added back to the pool once it has passed its automatic stiff tests. Avoiding embarrassing performance issues.
Worst Practice 3: Application Roll Out
Your team is preparing for a new application to go live on Monday morning in an environment which has four application servers. During the app roll out on Sunday night there is a deployment hiccup. The application comes up across 3 of the 4 nodes, but not on the 3rd. This now leads to a length debugging process where operations are forced to start to compare the problematic node to the working ones, figure out what is different and then get this resolved. This leads to an extension of the change window and pressure to get the app running for business hours. Eventually it’s discovered there are some missing shared libraries, this is resolved and the app eventually gets up and running.
Analysis of this rollout later in the week shows that there were some manual preparation steps to get the environment ready, and just so happens that this particular library was not added to the node.Solution:
This common issue goes to show that innocent mistakes can be made in any environment. The solution to this is not only automating deployments, but also ensuring that your deployment packages are complete, so they don’t make assumptions about what is in a particular environment. XL Deploy works with complete deployment packages, and can analyze the current environment state to figure out what has changed and what needs to be deployed. If an automated deployment solution had been used here the issue would not have occurred. To take this to the next level it is also important to be able to track if changes have been made manually, through a side channel if you will. XL Deploy 4.5 provides configuration comparison features to help detect this and let you know what has changed.
These three examples can really happen to any IT team if they are rushed into making changes to their environment. More than ever these days, IT is overtasked and underappreciated. Complex, heterogenous environments are compounding this problem and making day-to-day life for IT Operations difficult. This is where deployment automation plays a critical roll in removing that risk of error, by enabling operations and development to reliably deploy every time .
Otherwise, your example may serve as a cautionary tale of worst practices that IT should avoid in the next edition!