Easing The Pain of Being On-Call

Every time the phone rang in the middle of the night I pretty much knew that some soulless voice would be on the line. The monotonous robotic voice would state something like “JOB J-S-1-5-3 ABENDED AT LINE NUMBER 12 WITH ERROR CODE 1234.”

For the thousands of IT staff that are “on-call,” this is a familiar, and very much unwelcome, experience. Over the years, systems support staff have seen their sleep interrupted by beepers, Blackberry vibrations, and cell phone calls. I guess this beats having the home phone ring and wake up the spouse and kids. It is hard enough getting yourself out of dreamland and back to the cold “back-to-work” mentality in the middle of the night.

I remember the first time I received a beeper to be on-call in the early 90’s. I felt so important! Whenever I was at a party I made sure to wear my beeper proudly on my belt, never tucked away in my pocket. And when it went off, it was with great pride I would proclaim just loud enough for all to hear, “Excuse me but I need to call into the computer center.”

Smart IT Columns

Understanding Your 'Idiot' Manager Is Everyone Replaceable? Managing Vendor Demonstrations Blogging Your Way Up the Career Ladder FREE Tech Newsletters Instant Messaging Planet HTML CIO Update CodeGuru Update Enterprise Networking Planet Practically Networked HTML Enterprise Storage Forum Text Enterprise Storage Forum HTML Optically Networked HTML Intranet Journal Update Datamation IT Management Careers Datamation IT Management Update Developer.com Update Gamelan Java Update Goodies to Go Javascript Weekly HTML JARS Java Update OpenSource Update SysOpt Tech Notes Grid Computing Planet HTML E-Security Planet Text E-Security Planet HTML Virtual Dr. Text

That got old very, very fast.

When the beeper started going off in the middle of dinner, during a crucial point in my favorite TV show (this was pre DVR), and repeatedly in the middle of the night, I started to think being on-call wasn’t a very glamorous responsibility. As a matter of fact, it was the pits. And sometimes after working into the wee hours of the morning to solve a problem, I had to be back at work very early writing more code while feeling groggy and irritable. I began to wonder how the heck we could do more to reduce the number of calls or at least make the life of on-call staff a bit less stressful.

It should be quite obvious that better coding standards and techniques will reduce the number of bugs. And you don’t have to work in a CMM or ISO9001 environment to have processes in place. However, in many IT shops the staffing levels still haven’t recovered from the post-bubble staff reductions and it is human nature to look for time saving short cuts when we are overworked. It is also true in small to mid-size businesses that there isn’t a dedicated support staff and that every team member must take a turn at support or be in the escalation hierarchy.

So I get that everyone is stretched. However, to reduce your late night calls, you must not only put standards and processes into place, but put in a mechanism to enforce them. I’m a big fan of peer-to-peer, standards-based, code and unit test reviews – without a manager in the room. Teams can do a very good job of policing themselves, and with more vocal participation. Once code passes this review, then passes quality assurance and integration testing, it can be blessed for a promotion to your production environment.

A developer should never be able to promote their code into production without following this process – except while on call. If they have to fix something to get the system running again, then consider it a temporary patch that must then go through the same review and testing process, preferably the next day.

At least today remote access is fast and easy. Back in the Stone Age, dialing into the mainframe on a 2400-baud line was very time consuming. Make sure your team has the tools in place at home to remotely diagnose a problem, such as Citrix Presentation Server and a high speed Internet connection. Don’t ask them to rely on their home computers for support. You don’t want their teenager to download a virus on a computer they need for support. Either assign them a laptop or have one that is shared by the support team.

Next page: More Solutions for On-Call Help