A few weeks ago, when I wrote that, “forced to choose, the average FOSS-based business is going to choose business interests over FOSS [free and open source software] every time,” many people, including Mathew Aslett and Matt Assay, politely accused me of being too cynical. Unhappily, you only have to look at the relations between Red Hat and Fedora, the distribution Red Hat sponsors, during the recent security crisis for evidence that I might be all too accurate.
That this evidence should come from Red Hat and Fedora is particularly dismaying. Until last month, most observers would have described the Red Hat-Fedora relationship as a model of how corporate and community interests could work together for mutual benefit.
Although Fedora was initially dismissed as Red Hat’s beta release when it was first founded in 2003, in the last few years, it had developed laudatory open processes and become increasingly independent of Red Hat. As Max Spevack, the former chair of the Fedora Board, said in 2006, the Red Hat-Fedora relationship seemed a “good example of how to have a project that serves the interests of a company that also is valuable and gives value to community members.”
Yet it seems that, faced with a problem, Red Hat moved to protect its corporate interests at the expense of Fedora’s interests and expectations as a community — and that Fedora leaders were as surprised by the response as the general community.
Outline of a crisis
What happened last month is still unclear. My request a couple of weeks ago to discuss events with Paul W. Frields, the current Fedora Chair, was answered by a Red Hat publicist, who told me that the official statements on the crisis were all that any one at Red Hat or Fedora was prepared to say in public — a response so stereotypically corporate in its caution that it only emphasizes the conflict of interests.
However, the Fedora announcements mailing list gave the essentials. On August 14, Frields sent out a notice that Fedora was “currently investigating an issue in the infrastructure systems.” He warned that the entire Fedora site might become temporarily unavailable and warned that users should “not download or update any additional packages on your Fedora systems.” As might be expected, the cryptic nature of this corporate-sounding announcement caused considerable curiosity, both within and without Fedora, with most people wanting to know more.
A day later, Frield’s name was on another notice, saying that the situation was continuing, and pleading for Fedora users to be patient. A third notice followed on August 19, announcing that some Fedora services were now available, and providing the first real clue to what was happening when a new SSH fingerprint was released.
It was only on August 22 that Frields was permitted to announce that, “Last week we discovered that some Fedora servers were illegally accessed. The intrusion into the servers was quickly discovered, and the servers were taken offline . . . .One of the compromised Fedora servers was a system used for signing Fedora packages. However, based on our efforts, we have high confidence that the intruder was not able to capture the passphrase used to secure the Fedora package signing key.”
Since then, plans for changing security keys have been announced. However, as of September 8, the crisis continues, with Fedora users still unable to get security updates or bug-fixes. Three weeks without these services might seem trivial to Windows users, but for Fedora users, like those of other GNU/Linux distribution, many of whom are used to daily updates to their system, the crisis amounts to a major disruption of service.
A conflict of cultures
From a corporate viewpoint, Red Hat’s close-lipped reaction to the crisis is understandable. Like any company based on free and open source software, Red Hat derives its income from delivering services to customers, and obviously its ability to deliver services is handicapped (if not completely curtailed) when its servers are compromised. Under these circumstances, the company’s wish to proceed cautiously and with as little publicity as possible is perfectly natural.
The problem is that, in moving to defend its own credibility, Red Hat has neglected Fedora’s. While secrecy about the crisis may be second nature to Red Hat’s legal counsel, the FOSS community expects openness.
In this respect, Red Hat’s handling of the crisis could not contrast more strongly with the reaction of the community-based Debian distribution when a major security flaw was discovered in its openssl package last May. In keeping with Debian’s policy of openness, the first public announcement followed hard on the discovery, and included an explanation of the scope, what users could do, and the sites where users could find tools and instructions for protecting themselves.
Debian took an undue amount of criticism for the flaw — not least because Debian has always claimed to be one of the more secure distributions. But, in fact, its response was not only in the forthright tradition of FOSS, but also in accord with best security practices, which reject the idea of security through obscurity — the idea that keeping a problem secret is the best way to ensure that it won’t be exploited.
By contrast, the Fedora-Red Hat announcements not only concealed information, but gave users no way to investigate their own system for problems, nor any means of protection beyond the negative one of not installing or updating. Faced with a security problem, Red Hat reacted far less like Debian and much more like Microsoft, which is notorious for denying security problems until a patch is ready. No doubt it tried to protect its corporate interests, but it did next to nothing for users. When trouble came, FOSS interests and standards were apparently jettisoned in favor of immediate business concerns.
The damage to Fedora’s credibility is potentially immense. In a matter of days, Red Hat has quashed Fedora’s claim to independence. It has also threatened the credibility of the Red Hat employees who manage Fedora — people whose devotion to FOSS has always been clear in their actions and dedication. Frields especially is hard hit, having apparently signed his name to announcements written in a style so different from his normal one that he was likely just signing statements written by Red Hat executives and lawyers.
Nor am I alone in this perception. The Fedora Board itself seems perfectly aware of its embarrassing position, if the minutes of its last meeting are any indication. As might be expected, much of the meeting was devoted to “discussion about the incident handling,” and the summary expresses concerns that officially neither Red Hat nor Fedora admitted.
“Could other groups have been brought into knowledge of the incident earlier?” the minutes ask. “Could the Fedora Board have been notified or kept in the loop better?” The summary goes on to note that events were “complicated by co-announcement made by Red Hat,” and notes an “ongoing tension between Fedora being able to act independently and Red Hat being liable for Fedora’s actions.” The summary also says that the board doesn’t “want to get into a situation where every Fedora decision or announcement has to be vetted through Red Hat executive levels” before discussing possible plans to avoid similar situations in the future.
What is interesting about this summary is that it seems to confirm the worst possible misgivings. Not only was Fedora not allowed to act independently in the crisis, but even its board was apparently not adequately informed. Having witnessed the dedication and energy of a few members of the board, as well as several Red Hat employees who work full-time on Fedora, I can only sympathize with them as they learn to live with the fact that, at the first crisis, their idealism was swept aside by immediate corporate concerns.
An inevitable divide?
Both Fedora and Red Hat will undoubtedly weather this crisis. It may even be that, as the Fedora board minutes optimistically assume, that the experience can be used to improve how Fedora and Red Hat interact when other problems arise.
Still, looking at how the crisis was handled, you might be forgiven for being pessimistic. If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies — especially publicly-traded ones — will act any better?
I’d like to think that Red Hat panicked or took bad advice. Perhaps it will show more respect to FOSS in the next crisis. But perhaps this example shows that FOSS attitudes and standards can only exist with accepted business practice when times are good — and that, when a problem arises, FOSS becomes the first casualty.