Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Siebel Systems’ assertions that it enjoys an unprecedented 90-plus percent customer satisfaction rating came under fire recently in a report from Nucleus Research that questioned not just Siebel’s customer satisfaction rating but its return on investment claims.
Within a week, Siebel released its most recent customer satisfaction survey (conducted by Satmetrix, a research firm in which Siebel is a minority owner and strategic partner) that reiterated Siebel’s past claims of near universal satisfaction.
Then the real fun began. First there was the verbal sparring. Siebel claimed the Nucleus survey was a “random” survey that wasn’t representative of its 3,500 customers. Nucleus stood by its research, with one of its researchers stating that “If you have three best friends and two say you’re a jerk, what does that tell you?” Meanwhile, the question of Siebel’s invincibility or lack thereof became the talk of the enterprise applications industry.
Unfortunately for Siebel, a look at the methodology and the questions used to come up with Siebel’s customer satisfaction ratings make it much easier to believe Nucleus than Satmetrix.
Siebel, to its credit, sent me copies of the Satmetrix surveys, and I have to admit on paper they look pretty straightforward. By all appearances, Satmetrix is accurately reporting how customers fill out their surveys.
But that doesn’t mean Siebel can claim that these results are unimpeachable. On the contrary, there are three good reasons why the Satmetrix surveys are skewed in Siebel’s favor:
The lack of anonymity of the survey. The Siebel customers are sent a URL that links them to a survey that matches the status of their implementation. The survey is filled out at a web site, which means that Satmetrix knows exactly who is answering what question in what way. Anything less than total anonymity brings into question whether the respondent is being completely candid. (This, by the way, is how Siebel’s employee surveys are done. Tom Siebel also claims near 100 percent employee satisfaction, despite much anecdotal evidence to the contrary. Without anonymity, would you tell your employer what you really think?)
The survey question which leads Siebel to claim 100 percent loyalty has a tricky internal logic. Customers are asked, “Based on your experience with Siebel and its products, are you most likely to continue purchasing or using their products?” The response can only be a simple “yes” or “no.” The problem is the “or” in the question. If you plan to never buy software from Siebel again but will continue to use the product you’ve installed, you have to answer “yes.” Unless you’re tearing the system out, you have to answer “yes” and join the rest of the “loyal” customers. Siebel’s customer loyalty figure really measures who is planning to throw out an existing implementation and who isn’t.
Satmetrix’s results don’t match the realities of the software market. If Satmetrix is to be believed, then out of the 3,500 Siebel customers, 140 are dissatisfied. If Nucleus is to be believed, 14 out of 23 marquee customers, gleaned from Siebel’s web site, have no positive ROI and are, most likely, unhappy as well.
The problem with the Siebel numbers is that they are simply too high: you’re asked to believe, despite serious questions about the objectivity of the survey, that Siebel’s customer satisfaction is not only through the roof, but it’s better than any enterprise software company in the history of the industry. No software company has ever enjoyed the level of customer satisfaction that Siebel claims, and based on a close look at their survey, those claims are simply unwarranted.
In the end, I think Nucleus has the better handle on Siebel’s customer satisfaction. Nucleus did in-depth interviews in person, is able to promise anonymity, conducted the survey without any input or funding from any vendor, and came up with a result that is in line with what we know about the entire software industry: Users of all enterprise software products, including Siebel and its rivals, are just never that satisfied.
