A hardened kernel is (or is capable of being) stripped down to the bare essentials needed to run the applications it's intended for, and one in which the facilities needed are audited, limited, and protected.
My goal is to create a system with standard (POSIX, 4.4BSD) programming semantics, based on a well-understood, well-documented kernel, that lends itself well to secure application programming, and is also capable of securely running legacy applications, in situations where the source code to those applications may or may not be available.
By and large, most of what I've done working towards this goal has been a reaction to the security problems the security community has already witnessed. Part of what I'm attempting to do is to look at underlying program->OS interactions that enable security problems to occur ("enable" being an important word here, in contrast to "cause").
I'm still working on a coherent design, or some concept that will unite my scatterbrained ideas. I don't have it yet. What I do have is a collection of small snippets of kernel and userland code, which, applied, greatly limits what can be done by a user with access to a BSD system, whether that access comes through a terminal session or the manipulation of a mail daemon.
Contact me via Email if you're interested in details, source code, or development help.
Unix grew up with the security concept of "SUID". "SUID" attempts to solve a security design problem, which is that users sometimes need to do specific things that they should not be able to do in general. A good example of such a problem is editing the password database - users need to change their passwords, but they obviously shouldn't be able to change everyone's password.
Unix works around this problem with "SUID" programs. SUID programs are, ostensibly, a very restricted gateway to enhanced privilege. The concept is simple: an SUID program runs as the user who owns it. A root-owned SUID program runs as root. If you have a root-owned SUID password-changing program, any user can change their password.
Over the years, hundreds of security problems have been caused by SUID-root programs that had holes in them. Part of the problem is due to the quality of the code, and ignorance of secure programming issues. Part of the problem is due to the scope afforded to SUID programs. Finally, some of the problem is that most SUID programs are SUID to "root"; rather than thinking of how to grant the least possible amount of privilege to a user, the SUID program claims all possible privilege to do what it needs to do.
This problem is exaggerated by the fact that the kernel doesn't care about any user's identity unless the user is root. There are no checks in the 4.4BSD kernel for "user bin" or "user daemon". If something in the kernel requires privilege, it requires root credentials. This issue is most obvious in things like privileged ports; the kernel's requirement that privileged ports be bound as root leads to large application programs (like rlogin) running as "root".
One of the first things I did with 4.4BSD was to locate all the points in the kernel that checked for "root" credentials, and replaced it with something configurable (through sysctl). With a sysctl and a few chmod's, a hole in rlogin gets you the ability to bind a privileged port - and that's it, because rlogin can now run as user "network" instead of "root".
This is actually really easy to do, once you understand the "sysctl" interface. A slightly more involved task is to run down all the privileged userland programs, and figure out what unique privileges they require, and then ensuring that they claim only those privileges. Once the kernel is modified to forget about "root", this becomes primarily a filesystem security thing.
A much harder problem is consideration of how different privileges interact; if you give it a few moments thought, you realize that in many situations, the privilege of being able to bind privileged ports is equivalent to root (it allows you to spoof rlogin to localhost, or rebind the NFS server port). Since privileges that required root access obviously didn't need to consider this too much, there are quite a few places where the new privilege system requires indepth examination.
A real approach to addressing this problem is to program the kernel not to allow users to execute arbitrary programs. Rather, the kernel allows programs to be executed only if they're in one of several "trusted paths" - places where only "root" can add new files, and where the admins are confident only legitimate programs are located.
This is really of limited benefit at a point where the security community is primarily concerned with security problems that allow attackers to take over entire running privileged processes; trusted path execution does nothing to solve the problem of a daemon with a stack overrun. However, it does make mounting an attack against the system from the local host harder, and can actually completely prevent the exploitation of some vulnerabilities.
The problem with this is that gaining total access to a Unix system doesn't necessarily require access to the filesystem. For instance, process debugging facilities will allow "root" to take over a non-protected legitimate program running on the system, which can be used to evade the filesytem restrictions. The chrooted program can also work in concert with unprivileged processes on the system to elevate their privilege.
Fortunately, from within the kernel, it's really easy to determine if a process is chrooted. Using that information, it's possible to restrict a large portion of the kernel's facilities from chrooted programs; process debugging, device creation, privileged ports, and raw sockets can all be restricted from any process with a modified root directory.
In fact, most Unix security problems involve tricking privileged programs into doing things that they were never intended to do - for instance, tricking "traceroute" into executing a shell as root, or tricking "cron" into making the system shell SUID root.
It'd be nice if there was a way to tell the kernel what any given privileged program is allowed to do, and then have the kernel only allow it to do that. "traceroute" would be killed instantly if it tried to execute "/bin/sh", and "cron" would die as soon as it called "chmod".
This approach allows the security of privileged programs to be enhanced without necessarily even touching the source code; the admins or developers come up with a profile that details what each program can do, and loads it into the kernel. The program runs, oblivious to this, until someone successfully exploits a security vulnerability in it, and causes the program to violate it's specification.
Some of the things I've done to work towards this turned out to be surprisingly easy. In OpenBSD, less than 100 lines of code are required to rig up a kernel where a program can permanently "turn off" an arbitrary system call, preventing it from ever executing that system call again (so, for example, after "lpd" gets to a certain point in the code, it can never again call "execve") - what's more, 4 lines of code makes this restriction inherited to the process' children.
More flexible facilities are a bit harder, but rudimentary support isn't rocket science. The issues come up when you start dealing with protecting system calls like "open" - you can't just turn off "open()" - it's called all over the place in application code. Instead, you start thinking about how to provide restrictions to open() itself... "this program can open files in '/tmp', but never, ever in '/etc'".
The problem then becomes coming up with some sort of coherent interface that covers everything you'd want to restrict about every complicated systen call ("can open sockets, but not raw sockets unless they're ICMP and not header-included"). I have restriction hooks for most of the problematic system calls, but I haven't come up with a good way to present an interface to their configuration.