Most likely rooter will run a variant/flavor of debian linux, with significant
improvements and new software pushed upstream.

### Linux Firmware Distributions

These Linux-based distributions are all designed to run on very light-weight
hardware. They use special filesystems optimized for mostly reads from a static
archive with a few small writes to persist configuration information.

* [OpenWrt](http://openwrt.org): Most popular and best maintained community
  project. Has an entire packaging system for add-ons. Very wide hardware
  support.
* [DD-WRT](http://www.dd-wrt.com/site/index): non-libre, not recommended
* [Tomato](https://en.wikipedia.org/wiki/Tomato_(firmware)): elegant, well
  selected feature set, pleasant UI. Vanilla version has had little recent
  development or porting, many forked variants. 

### BSD Firewall Distributions

There are a series of "network appliance" firewall distributions oriented at
business and enterprise networks. They usually run on dedicated hardware more
powerful than commodity home routers, scaling up to multi-unit high performance
rack mounted machines:

* [pfSense](http://www.pfsense.org/): FreeBSD, forked from m0n0wall
* [m0n0wall](http://m0n0.ch/wall/): FreeBSD, php interface
* [Smoothwall](): commercial with "free" version
* [zrouter](http://zrouter.org) freebsd router

It's also worth noting that Juniper Network's junos router operating system is
based on FreeBSD. Juniper equipment isn't as widespread as Cisco's iOS-based
(not to be confused with the Apple operating system) routers, but is used for
many of the most crucial 40Gbps+ routers at submarine fiber shore stations.

### Debian/OpenWrt Hybrids

A lot of people have thought about mixing the kernel, drivers, and
configuration system from OpenWrt with the mainstream packages from Debian.

* [DebianWRT](http://wiki.debian.org/DebianWRT)
* [debwrt](http://www.debwrt.net/) is a stale project to bring debian packages
  to the OpenWRT kernel/init system.

### Genode

Interesting nested resource-deterministic operating system "framework"; sort of
like a generalization of a hypervisor. Strong ideas for minimizing the
potential for security bugs and non-recoverable crashes. Might be appropriate
for ensuring that "essential" network services keep running even if higher
level apps crash or are compromised.

[More thoughts
here](http://mailman.rooter.is/pipermail/talk/2012-June/000023.html).

### Other

* [LibreWRT](http://librewrt.org/index.php?title=Main_Page)
* [FreeWRT](https://www.freewrt.org/trac/)
* [Embedded Debian](http://wiki.debian.org/Embedded_Debian),
  [embedian](http://www.emdebian.org/)
* [Zeroshell](http://www.zeroshell.net/eng/)