The Future of Operating Systems Security

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More .

By Ronald L. Mendell for SecurityPortal

Often computer security takes us down strange paths; for example, what is the connection between the Navajo language and the future of operating systems? These subjects seem odd bedfellows to be sure; yet, we shall learn that obscurity, contrary to the general maxim, sometimes does create a degree of security.

The current trends in OS development dwell on the mainstream players: Linux, Unix, and Windows NT/2000 and their offshoots Trinux, Minix, and Windows CE. Linux, for example, will probably continue with a 25% percent annual growth rate for the next couple of years. Factors driving the immense popularity of these OS families include economics, learning inertia, and the low desire for the “overengineering” of security features.

In the wake of vast Y2K expenditures, most businesses do not want to buy new applications or pay for new code to run on operating systems outside of the mainstream. The vast documentation and training apparatus surrounding mainstream OSs encourages inertia among many IT professionals from venturing onto new ground. And, despite the criticisms of maverick OS designers, the off-the-shelf security features of mainstream OSs engender contentment in many business people.

Wide markets for popular OS products create economies of scale that make the IT world possible. They provide standardization, which reduces costs. This intellectual “common market,” however, has a drawback. It also creates a knowledge base for hackers, crackers, software pirates, and other computer criminals. They can learn the lingua franca quickly and benefit from the economies of scale too.

During the “Iron Age” of mainframe dominance, when memory still existed as ferromagnetic cores, computing was industrial, and very much a closed shop. Operating systems and programming languages were arcane. A natural divide existed between computing and “the world.”

The microcomputer revolution empowered script kiddies and other, more inquisitive, barbarians to begin an onslaught against IT. With the advent of wireless computing and distributed operating systems, the dangers continue to evolve and to multiply.

With the desire to capture new markets, scalability in operating systems becomes an overriding concern. Unfortunately, scalability in some cases may conflict with trusted OS design. The desire for the former is a dynamic impulse, while trusted OS design seeks stasis.

Trusted operating systems protect objects such as memory, hard disks, printers, tape drives, programs, and shared data such as databases. They effect protection through access control lists and matrices. They invoke security principles such as least privilege, separation of privilege, permission-based access, and least common mechanism.

To what extent these traditional security models will hinder scalability and the new wireless wave remains subject to speculation. However, the emergence of two security camps is foreseeable.

In the first camp (mainstreamers) ease of access will dominate and invoke only minimal protections. The second camp will place a high premium on going back to the drawing board on OS security issues. This new drawing board solicits three different approaches: “Navajo Speakers,” “Pioneers,” and “Small Kingdoms.”

In the Second World War, the one Native American language not studied by German or Japanese scholars, prior to the conflict, was Navajo. A complex language of limited distribution, Navajo provided a means of secure communication on battlefields. Two native speakers could develop a code using Navajo words to communicate over radio or landlines. It was an example of “security through obscurity” that worked.

One path of OS evolution for increased security could be using programming languages or existing OS models well outside of the mainstream. “Iron Age” languages such as PL/1, APL, or SNOBOL have low frequency with today’s programmers. No one’s rushing around to document them. (Multics, a precursor to Unix, was written using PL/1.) Nonstandard dialects of DOS such as PTS-DOS and FreeDOS are other possibilities.

Obviously issues of speed and integration become serious considerations. Yet, considering the “old” and the unusual in software may offer OS security alternatives for those desiring to work outside of the limelight. The wider the distribution of a language or a secure OS, the more potential attackers have to work with when breaking it.

“Pioneers” seek new designs for operating systems, often thinking beyond kernel architecture. TUNES and Aegis (from MIT) eliminate the kernel to increase system performance and security. SPACE (from the University of Santa Barbara) develops protection as multiple layers rather than relying upon the kernel. And SPIN (from the University of Washington) allows code migration in kernel space, blurring the distinction between applications and the kernel. EROS (from the University of Pennsylvania) uses constructor and confinement mechanisms that go beyond what’s available with POSIX, NT, or Java’s sandbox.

“Small Kingdoms” are either operating systems with very small followings (of users) or leave very small footprints. Geex!, JeniOS, and Proolix (from the former USSR) are all along this line. In this family are also embedded operating systems such as Chorus, coniX, VxWorks, and DR-DOS, which serve as alternatives to Windows CE, Palm OS, and QNX. As computers become more of the handheld variety, these micro-OS systems will undergo continual security scrutiny. In agriculture, having many strains of corn or wheat insures survival through diversity; the same goes for embedded OSs. Monoculture, one OS for all portables, may not be the best way to go.

What we’ve tried to do is look outside the box, to think beyond the current arena dominated by POSIX and NT/2000 OS models. And, while many of the alternatives may fall discarded on the roadside, the creativity they engender gives us the edge in developing more secure and tamper-resistant OS products. The following suggestions or predictions seem reasonable:

Most purchasers will not pay for industrial-strength security features in operating systems designed for mass markets.
Certain segments of the market with high-level security concerns may use OS products outside of the mainstream.
In developing industrial-strength security, OS designers need to consider the rich heritage from programming’s past, to do thinking beyond the kernel and root, and to seek a small footprint in the IT world as to publicity. This does not mean that designers should go to secret, proprietary OS products. But customizing OS packages to specific security needs, rather than the mass production of general countermeasures, will improve and raise the wall of protection in critical areas.
The proposition that one solution fits all environments seems outdated as a computer security philosophy. Mainstream OS products cannot be stretched or patched safely to fit every security requirement.
And, as Linus Torvalds stated to CNET in October 1999, “What will drive the software industry is specialized software for specialized needs.” Cookie-cutter solutions will become things of the past.