Most likely rooter will run a variant/flavor of debian linux, with significant
improvements and new software pushed upstream.

### Linux Firmware Distributions

These Linux-based distributions are all designed to run on very light-weight
hardware. They use special filesystems optimized for mostly reads from a static
archive with a few small writes to persist configuration information.

* OpenWrt: Most popular and best maintained community project. Has an entire
  packaging system for add-ons. Very wide hardware support.
* DD-WRT (non-libre, not recommended)
* [https://en.wikipedia.org/wiki/Tomato_(firmware)](Tomato): elegant, well
  selected feature set, pleasant UI. Vanilla version has had little recent
  development or porting, many forked variants. 

### BSD Firewall Distributions

There are a series of "network appliance" firewall distributions oriented at
business and enterprise networks. They usually run on dedicated hardware more
powerful than commodity home routers, scaling up to multi-unit high performance
rack mounted machines:

* pfSense
* m0n0wall
* Smoothwall

It's also worth noting that Juniper Network's junos router operating system is
based on FreeBSD. Juniper equipment isn't as widespread as Cisco's iOS-based
(not to be confused with the Apple operating system) routers, but is used for
many of the most crucial 40Gbps+ routers at submarine fiber shore stations.

### Debian/OpenWrt Hybrids

A lot of people have thought about mixing the kernel, drivers, and
configuration system from OpenWrt with the mainstream packages from Debian.

Eg, [http://wiki.debian.org/DebianWRT](DebianWRT).

### Genode

Interesting nested resource-deterministic operating system "framework"; sort of
like a generalization of a hypervisor. Strong ideas for minimizing the
potential for security bugs and non-recoverable crashes. Might be appropriate
for ensuring that "essential" network services keep running even if higher
level apps crash or are compromised.

[http://mailman.rooter.is/pipermail/talk/2012-June/000023.html](More thoughts
here).

### Other

* http://librewrt.org/index.php?title=Main_Page
* https://www.freewrt.org/trac/
* http://wiki.debian.org/Embedded_Debian