Similar shortcomings occur when projects rely on bug-reports or documentation. Almost by definition, anyone who is comfortable enough to file a bug or contribute to documentation is not a typical user. Although perhaps they are closer to average users than developers, bug-reporters and documentation writers are most reliable when they can project themselves into the experience of less-skilled users. Otherwise, their contribution can be inconsistent, particularly in projects where non-developers are held in low regard and ignored.
Lacking formal usability testing by projects, many developers resort to their own small scale studies. For instance, Allan Day, a GNOME designer, talking about GNOME 3, tells me that "Personally speaking, I have done two rounds of usability testing. The first was conducted around the time of the GNOME 3.0 release, and was done with friends and family. More recently, I conducted a small study on the lock screen last development cycle." He adds that "I'm not the only one who has done this type of self-initiated testing." Almost certainly, the majority of free software projects include people who could make a similar report.
What Day calls "ad hoc usability testing" is considerably better than nothing. However, much of the time, it is done on a developer's own time, rather than as something they are expected to do. It is, perhaps, a reflection of their frustration at the lack of information with which they have to work.
The problem with such small-scale testing means that is its uncertain validity. The small pool of testers can be a problem, especially if you want the opinion of an inexperienced user. If asked too often, friends and family may soon cease to be new users, and their feedback can become less valid or useful over time.
Moreover, to make matters worse, the results of such testing often receive limited circulation. Unlike large-scale usability testing, they may receive only limited attention because only a few developers ever learn about them. Instead of promoting a climate in which usability testing is the norm, their long-term result is to make the little that is done invisible, and to encourage similar informal solutions to continue.
This situation is no one's fault. It is simply the way that free software development has evolved. The result is that, while most contributors to free software projects would agree in theory that usability testing should be implemented, in practice a lack of resources and examples discourages much action. Throughout free software, any usability testing that occurs remains consistently small scale.
Studies like Aakanksha Gaur's current project within GNOME for the Outreach Program for Women, are decidedly the exception. With Day as her mentor, Gaur is attempting to answer basic questions that most developers never ask—questions like, "What are the usability issues encountered by new and existing users of GNOME 3?" and "How is GNOME 3 perceived by new and existing users?"
The increasing number of free software users make the need for answers to such large scale questions more urgent than ever.
Whether usability testing could have prevented the decisions that created the user revolts in KDE, GNOME and Unity is open to debate. Although usability testing is useful, it can only be as good as the questions it sets out to answer. If, for instance, testers never set out to discover whether users prefer an old or a new design, then they are unlikely to ever answer that question.
Just as importantly, the later that usability is done in the development process, the more likely developers to will be to resist major change. At that point they have too much invested to consider scrapping the code or even making major revisions.
Other factors may also intervene. For example, KDE 4.0 was framed as a developer's release. It was never intended for general use, but distributions, eager to have the latest releases, failed to make its status clear.
All the same, begun early enough and repeated throughout development, usability testing could at least help developers to make better decisions—especially if the testing is done by people who did not write the original code and who have some authority to return the code for improvement before it is released.
Today, usability testing is only occasionally integrated into free software projects. Yet, as more casual users are attracted to free software, the need to understand their habits is growing—and the current methods of feedback contribute imperfectly to that need. Properly carried out, usability testing could reconnect developers and users in ways that are badly needed. However, whether it will ever take hold in free software development is another matter altogether.