How Schwab Transformed its On-Call Support

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

My last article about managing teams with on-call responsibilities generated a lot of interest from readers who prompted me to find a real world example where a team had implemented a successful model. I wanted to find an instance where the team members, management, and the end-users all came out winners.

I was lucky enough to track down Mary Lavine, Managing Director, IT, Schwab Development. Even though most of her teams are focused on application development, some need to provide support for those applications in production environments.

One team in her department that was shouldering a heavy support burden was the Content Management System. She brought in an experienced senior manager, Mei Wan, who was familiar with the area and knew she was walking into a difficult situation. Her team was responsible for development, tools, systems administration, technical and business support. They barely had time to breathe, getting calls at all hours.

Smart IT Columns

How to Sell Product Purchases Internally

Is Everyone Replaceable?

Managing Vendor Demonstrations

Understanding Your ‘Idiot’ Manager

I asked if Wan had a particular example of a team member coming to her with complaints. “There were so many times I couldn’t keep track!,” said Wan. “No one was happy and no one was being successful.”

Multiple team members came to her with the same message: “We are doing way too many jobs, working around the clock nights and weekends, to keep going at this pace. There are things I have to offer the company way beyond what I’m doing today.” The not so subtle message was that management had to make changes or start losing team members.

“After hearing this, I told Mary that I needed six months to make significant progress on this problem” said Wan. They both agreed these developers were hanging on by a thread and they would start to lose them if the environment didn’t improve. “I agreed to give her team air cover,” said Lavine. “If the business side complained we weren’t delivering enough for them, I would ensure we were making progress and point to incremental improvements to buy us time.”

And they both knew that each developer owned a large chunk of systems knowledge that would be almost impossible to replace. “We felt if we lost any one of them we effectively lost the system because they each owned a key core competency,” said Lavine. “It was an easy decision to support these changes because we could qualify what the business would NOT be able to do for multiple months while we tried to hire, then train a replacement. Not to mention that during this time the others on the team would be even more stretched to cover the loss of expertise.“

Wan started to collect feedback on improving the support situation, spending time listening to each of her developers. “These were smart people that were just being told what to do. I simply leveraged their ideas to start building a better support process,” said Wan. By listening to their ideas and finding small wins, she built up trust with them and that bought her more time to fix the problems.

Next page: Offloading Duties

After analyzing the support calls, it turned out that the majority of calls did not require a developer to fix the issue. The first thing Wan did was offload these systems administration support duties to more appropriate groups, such as having operations staff reboot servers. More pivotal was the decision to offshore the after-hours first level support to India.

Smart IT Columns

How to Sell Product Purchases Internally

Is Everyone Replaceable?

Managing Vendor Demonstrations

Understanding Your ‘Idiot’ Manager

But offshoring can have many challenges, so Wan made a special effort to fully involve the team. “The developers wanted to be hands-on in picking the offshore team, because they knew higher quality candidates would translate into less support work for them in the long run,” says Wan. “The developers conducted the interviews and then trained the new team, even making one trip to India. We then brought the entire offshore team to the States for more training and team assimilation.”

The developers entered a weekly rotation of providing second level support. They were already being compensated an additional 10% of salary for every week they were officially assigned on-call duties. Wan made some more subtle changes as well. “Frankly, I wanted to treat them with more humanity,” says Wan. “We arranged for telecommuting to give them more work/life balance. This way if they were working on a problem through the night, they didn’t have to worry about getting up early to commute to the office.”

But what if someone gets caught in a bad cycle of support calls and puts in significant extra time? “You can reward them with comp time or in special cases the reward can be financial,” said Wan.

Lavine chimed in with a good point. “You have to be careful not to create a downward spiral by creating incentives for heroics. If you see these bad support cycles continuing month over month, then there is a process problem that needs to be addressed and you really want to reward people for fixing the process, not the individual problems. You ideally want to encourage sound testing, deployment and support processes that reduce the calls and their severity.”

The team also spent a significant amount of time to build a support process around a Service Level Agreement, so that the business side was confident that support issues would be resolved quickly. This helps prevent circumvention of the process, because if a business person doesn’t trust the process, they pick up the phone and call the subject matter expert who may or may not be on call.
Next page: Being Kind to Developers

The last change was to bring in more project work that appealed to the developers. “We currently have a pilot project using AJAX, which is a challenging very visible project to take advantage of the team’s true talents,” says Lavine.

Smart IT Columns

How to Sell Product Purchases Internally

Is Everyone Replaceable?

Managing Vendor Demonstrations

Understanding Your ‘Idiot’ Manager

So they have reduced the support load, allowed more workplace flexibility, established a good process, and provided challenging projects that will boost employees’ careers. Sounds like a winning formula. I asked Lavine where other managers who are stuck in support hell should start if they want to make these kinds of changes.

Her first suggestion was building a statement of vision for upper management. “Be clear on the problem you are trying to solve and the impact to the business,” says Lavine. “Talk about the potentially positive results, such as better SLA, less turnover, and cost reduction from improved productivity. It never hurts to outline what’s in it for senior management, and then delve into solutions. In our case we stabilized the CMS platform and dealt with a morale problem that would have otherwise ended up on the doorstep of upper management.”

Second, she suggests a high level case plan that specifically describes how to obtain a more stable environment, reduce turnover, and deliver high degree of quality applications. “Try to keep all this to one page, just touching the high level points, and be clear on the business implications of what will happen if these changes are not made,” says Lavine.

Finally, she said that before you deliver your vision and plan, spend some time with your business partners. “If you go to IT upper management with a plan that the business side has already bought into, you will have a better chance at securing their approval,” says Lavine.

There you have it folks – some real world guidance on how to create your own escape from on-call hell.