backup WIP

17 files changed, 462 insertions, 28 deletions

diff --git a/NOTES b/NOTES
new file mode 100644
index 0000000..19ce4b8
--- /dev/null
+++ b/NOTES
@@ -0,0 +1,38 @@
+system config files should live here in ./config
+system config scripts should live here in ./scripts
+    torouter-takeover
+        add torproject keys (prompt?)
+        setup sources.list
+        apt-get update
+        install dependancies like tor and torouterui
+        run torouter_preboot.sh
+    torouter-preboot
+        check that dependancies are installed (tor, torouterui, etc) or fail
+        copy/install all configuration files
+        install 
+        create users and groups
+        remove/disable unwanted packages
+        setup firstboot
+    torouter-firstboot (just generates ssh keys; could be done elsewhere?)
+    torouter-copy2internal (TODO)
+    torouter-setupuboot (TODO)
+
+torouter-prep contains all the above files
+
+torouter_easy_setup.sh installs torouter-prep, executes torouter-takeover,
+then reboots. it should be included in torouter-prep (for signing?)
+
+install.sh script executes withing qemu:
+    configures all installed packages
+    run torouter_preboot.sh
+    install .deb src packages
+    kernel re-packing
+    (re)set root password
+    (re)set user password
+    unconfigure sshd (so it will get re-configured on 1st boot)
+
+differences between takeover and .img build process:
+- don't need new ssh keys with takeover
+- don't need kernel re-pack with takeover
+- don't need fstab re-write with takeover
+
diff --git a/README b/README
index 8ba5f55..c830ded 100644
--- a/README
+++ b/README
@@ -1,33 +1,130 @@
-This is the place where we put all things relating to the Torouter.
+ _                        _            
+| |_ ___  _ __ ___  _   _| |_ ___ _ __ 
+| __/ _ \| '__/ _ \| | | | __/ _ \ '__|
+| || (_) | | | (_) | |_| | ||  __/ |   
+ \__\___/|_|  \___/ \__,_|\__\___|_|   
 
-The project home page:
-https://trac.torproject.org/projects/tor/wiki/doc/Torouter
+Project wiki page: https://trac.torproject.org/projects/tor/wiki/doc/Torouter
 
-The project road map:
-https://trac.torproject.org/projects/tor/wiki/doc/Torouter/Roadmap
+= Repository Contents =
 
-Your DreamPlug has arrived in a disabled state - you will need to hack it:
+  README            this file
+  ./doc             (WIP) text file user and developer documentation
+  ./freedom-maker   firmware build scripts
+  ./config          torouter configuration files
+  ./packages        source for torouter-specific software packages
 
-0) re-flash from inside the OS it shipped with:
-  https://trac.torproject.org/projects/tor/wiki/doc/DebianDreamPlug#Step9:IfyoucannotsuccessfullystarttheUSBdevices
+= Installation and Build Instructions =
 
-1) Create a bootable USB disk, attach the JTAG, boot/re-flash, ..., anonymity!
+There are several ways to get the debian-based torouter system running on a
+DreamPlug device:
 
-2) A new thing that is currently undocumented
+ - Flash a pre-built image onto an external USB stick
 
-To create a Torouter from this repository you'll want to clone it:
+ - Flash a pre-built image onto an internal microSD card
 
-  git clone git://git.torproject.org/torouter.git
+ - Build and flash a new image from source
 
-Please see the freedom-maker/README if you wish to create a bootable USB disk.
-You will want a USB disk and the DreamPlug JTAG for reflashing, reinstalling,
-and development related activities.
+ - Install the FreedomBox distribution or a "vanilla" debian, then run a
+   torouter takeover/upgrade script
 
------------------------------------------------------------------------------
+For the later, see the "Building" section, then follow either the USB stick or
+microSD card directions.
 
-If you wish to turn a DreamPlug into a Torouter simply follow these steps:
+Regardless of the installation method and target, if the device is in a
+"factory fresh" state some bootloader preparation is required. See the
+"DreamPlug Preparation" section below; you will need serial console access to
+the DreamPlug, for example with the sometimes included GuruPlug JTAG device.
 
-scp torouter_easy_setup.sh DreamPlug:/tmp/
-ssh DreamPlug torouter_easy_setup.sh
+=== Flash Pre-Built Image to External USB stick ===
+
+TODO:
+
+    $ dd if=$IMGFILE of=$USBBASEDEV
+
+=== Flash Pre-Built Image to Internal microSD card ===
+
+TODO:
+
+    $ dd if=$IMGFILE of=$USBBASEDEV
+
+Mount the device, make changes to /etc/fstab. Also need to make changes to the
+initrd image?
+
+Make changes to u-boot configuration.
+
+=== Torouter Takeover Script ===
+
+Simply copy over the torouter_easy_setup.sh and execute it (with sudo
+permissions) on the device. Substituting $DPHOST for an actual IP/hostname):
+
+    scp packages/torouter_easy_setup.sh $DPHOST:/tmp/
+    ssh $DPHOST /tmp/torouter_easy_setup.sh
+
+= Build a Torouter Image From Scratch =
+
+NOTE: it may be (?) that the deb.torproject.org apt repo needs to exist in the
+*host* operating system. See https://www.torproject.org/docs/debian.html.en
+
+These instructions assume a debian wheezy host system.
+
+In brief, the build process proceeds to:
+
+1. use the ``multistrap`` tool to fetch debian packages and build a complete
+   root filesystem
+2. copies in some torouter-specific configuration files
+3. optionally copies in locally-built torouter-specific .deb files
+3. emulates booting into the system using qemu to run some more configuration
+   steps
+
+By default, the multistrap process will try to fetch pre-build torouter
+software and configuration packages (such as the web user interface) from
+torouter.torproject.org. If you've made local changes to those packages, you
+can bundle in your own .deb files by building them into the
+./packages/packages-for-upload directory, though the local packages may get
+overwriten by apt updates if the version number is not high enough. See
+./packages/README for build instructions.
+
+To proceed with building the torouter image, first fetch build-system
+dependancies (this probably isn't an exhaustive list):
+
+    $ apt-get install rsync multistrap qemu-user-static u-boot-tools git mercurial debootstrap extlinux
+
+Then enter the freedom-maker directory and run a build:
+
+    $ cd ./freedom-maker
+    $ sudo ./buildrootfs.sh
+
+If that was ultimately successful (you can ignore some apt warnings and errors
+that may scroll by), you now have a root filesystem ready to be copied to a USB
+stick. Insert a USB storage device and partition it to have a small (~80MB)
+boot partition followed by an ext3 root partition (1GB or so minimum). Make
+note of the boot and root partition nodes (eg, $BOOTNODE=/dev/sdc1 and
+$ROOTNODE=/dev/sdc2), then copy over the built distribution:
+
+    $ sudo ./copy2card.sh $BOOTNODE $ROOTNODE
+
+If that was successful, contratulations, you now have a usable torouter thumb
+stic! You could skip ahead now to DreamPlug Perparation.
+
+If you want to dump your fresh un-booted image to a static .img file (which
+could be distributed to others or flashed to another stick quickly), run the
+copy2img.sh script with the "base" device of the thumbstick as an argument (eg,
+$STICKNODE=/dev/sdc):
+
+    $ sudo ./copy2img.sh $STICKNODE
+
+=== DreamPlug Preparation ===
+
+Prepping a vanilla DreamPlug device is somewhat involved and will require a
+access to the hardware serial console (eg, using a GuruPlug-branded JTAG device
+functions as a USB serial console adapter). 
+
+First, upgrade u-boot to a recent version (see directions in freedom-maker
+directory), then set the appropriate bootloader environment variables:
+
+    TODO: XXX:
+
+FYI, there are at least two variants of the DreamPlug hardware, with slightly
+differing chipsets.
 
-The DreamPlug should now be configured as a Torouter.
diff --git a/TODO b/TODO
index 1a13573..21dc926 100644
--- a/TODO
+++ b/TODO
@@ -1 +1,3 @@
 - fix signing key for torrouter.torproject.org (should have a keyring .deb)
+- re-enable source builds in install.sh
+- random apt-get segfault issue
diff --git a/freedom-maker/README b/freedom-maker/README.freedom-maker
index 6039861..6039861 100644
--- a/freedom-maker/README
+++ b/freedom-maker/README.freedom-maker
diff --git a/freedom-maker/TODO.freedom-maker b/freedom-maker/TODO.freedom-maker
new file mode 100644
index 0000000..184aca5
--- /dev/null
+++ b/freedom-maker/TODO.freedom-maker
@@ -0,0 +1,25 @@
+- [ ] initial boot configuration improvements, as there are a few
+  things we really do want to defer to first boot, or at least make it
+  trivial to fix up per-system
+
+	- [X] ssh host keys
+	- [ ] root password
+- [ ] add Sheeva as a build target
+- [X] add x86 virtualized as a build target (use vmdebootstrap?)
+- [ ] investigate and integrate software components that looks interesting
+	- channel-server, the buddycloud channels service for XMPP
+	- babeld, loop-free distance-vector routing protocol
+	- tahoe-lafs, secure distributed filesystem
+- [ ] virtualbox should build with dreamplug configuration
+
+*** stuff to do for first feature release ***
+
+  - [ ] pre-configured user interface
+	- [X] DHCP and HTTP servers on eth0 and/or eth1
+	- [ ] plinth-based initial config interface
+
+  - [ ] freedom buddy installed
+  - [ ] openvpn server setup
+  - [ ] the privoxy fork configured as a transparent proxy
+  - [ ] plinth configured to at display these config details and to
+   control whatever is feasible
diff --git a/freedom-maker/bin/finalize b/freedom-maker/bin/finalize
index 9995b77..a6cb7da 100755
--- a/freedom-maker/bin/finalize
+++ b/freedom-maker/bin/finalize
@@ -49,8 +49,8 @@ echo $hostname > $target/etc/hostname
 chown root:root $target/sbin/copy2dream
 chmod 744 $target/sbin/copy2dream
 
-# prepare chroot project finalization
-cp bin/projects-chroot $target/
+# prepare chroot package finalization
+cp bin/packages-chroot $target/
 
 echo "Using qemu-user-static to perform first-boot configuration now."
 chmod 755 $target/install.sh
@@ -58,8 +58,8 @@ cp /usr/bin/qemu-arm-static $target/usr/bin
 echo "Running install script from source/install.sh"
 chroot $target /install.sh
 echo "Running install script from bin/projects-chroot"
-chroot $target /projects-chroot
+chroot $target /packages-chroot
 
 # clean up.
-rm "${target}/projects-chroot"
+rm "${target}/packages-chroot"
 rm "${target}/usr/bin/qemu-arm-static"
diff --git a/freedom-maker/bin/packages b/freedom-maker/bin/packages
new file mode 100755
index 0000000..15d17a4
--- /dev/null
+++ b/freedom-maker/bin/packages
@@ -0,0 +1,35 @@
+#! /bin/bash
+# 
+# Copyright 2012 by Nick Daly <nick.m.daly@gmail.com>
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# 
+
+# projects
+#
+# Installs external projects to the system.
+
+
+# We don't tolerate errors.
+set -e
+
+echo "Adding extra torouter stuff..."
+mkdir -p $homedir/uap_hack
+cp -v $basedir/../../uap_hack/* $homedir/uap_hack
+mkdir -p $homedir/src
+git clone $basedir/../.. $homedir/src/torouter
+mkdir -p $target/tmp/dpkg
+cp -v $basedir/../../packages/packages-for-upload/*.deb $target/tmp/dpkg
+
+chown -R 1000:1000 $homedir
diff --git a/freedom-maker/bin/packages-chroot b/freedom-maker/bin/packages-chroot
new file mode 100755
index 0000000..42020c4
--- /dev/null
+++ b/freedom-maker/bin/packages-chroot
@@ -0,0 +1,4 @@
+#! /bin/sh
+
+dpkg -i /tmp/dpkg/*.deb
+
diff --git a/freedom-maker/buildrootfs.sh b/freedom-maker/buildrootfs.sh
new file mode 100755
index 0000000..856f49f
--- /dev/null
+++ b/freedom-maker/buildrootfs.sh
@@ -0,0 +1,122 @@
+#!/bin/bash
+# 
+# Copyright 2011 by Bdale Garbee <bdale@gag.com>
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# 
+
+# based on work by <ivan@sanchezortega.es>, who released his script under
+# the following license terms:
+#  ----------------------------------------------------------------------------
+#  "THE BEER-WARE LICENSE" (Revision 42):
+#  As long as you retain this notice you can do whatever you want with 
+#  this stuff. If we meet some day, and you think this stuff is worth it, 
+#  you can buy me a beer in return.
+#  ----------------------------------------------------------------------------
+
+# mk_dreamplug_rootfs
+#
+# Runs multistrap and readies the resulting root filesystem to silently 
+# complete package configuration on the first boot-up.
+#
+# Accepts the multistrap config file name as an argument.
+
+# We don't tolerate errors.
+set -e
+
+architecture=armel
+if [ -n "$1" ]
+then
+    architecture=$1
+fi
+
+config=multistrap-configs/torouter-$architecture.conf
+if [ -n "$2" ]
+then
+    config=$2
+fi
+
+# users
+hostname='torouter'
+rootpassword='freedom'
+user='torouter'
+userpassword='freedom'
+export hostname
+export rootpassword
+export user
+export userpassword
+
+# where to build images, etc
+basedir=`pwd`/build
+source=`pwd`/source
+target=$basedir/$architecture
+tmpdir=$basedir/tmp
+pkgcache=$tmpdir/aptcache
+homedir=$target/home/$user
+export basedir
+export source
+export target
+export tmpdir
+export pkgcache
+export homedir
+
+# clear any old cruft
+if (mount | grep $target/var/cache/apt) 
+then
+    umount $target/var/cache/apt/
+fi
+
+# make the directories we'll need.
+mkdir -p $target
+rm -rf $target/*
+mkdir -p $tmpdir
+mkdir -p $pkgcache
+mkdir -p $target/var/cache/apt/ && mount -o bind $pkgcache $target/var/cache/apt/
+mkdir -p $target/var/cache/apt/archives
+mkdir -p $target/usr/bin
+mkdir -p $homedir
+
+echo "Multistrapping..."
+# XXX: DEATH: work around torrouter.torproject.org GPG key issue
+# XXX: see also auth=false in multistrap config files
+# multistrap -f $config -d $target
+multistrap --no-auth -f $config -d $target
+rm -f $target/etc/apt/sources.list.d/multistrap-debian.list
+
+# un-do the bind mount so we don't trip over it later
+umount $target/var/cache/apt/
+
+# copy!
+echo "Copying the source directory to the torouter rootfs..."
+rsync -av $source/ $target
+
+# add extra packages to the image
+bin/packages
+
+# torouter!
+echo "Explicitly copying some torouter files..."
+# TODO: these should be redundant
+# Override the above stuff - we know better
+cp ../packages/torouter-prep/configs/interfaces $target/etc/network/interfaces
+# Stop the libertas module from loading
+cp ../packages/torouter-prep/configs/modprobe.d-blacklist.conf $target/etc/modprobe.d/blacklist.conf
+
+# cleanup and finalize the image so it boots correctly.
+echo "Finalizing..."
+bin/finalize
+
+# finish!
+echo "Syncing..."
+sync
+echo "Finished. You may now copy the rootfs to the plug."
diff --git a/freedom-maker/copy2card.sh b/freedom-maker/copy2card.sh
new file mode 100755
index 0000000..0a5208d
--- /dev/null
+++ b/freedom-maker/copy2card.sh
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+set -e
+
+if [ -n "`mount | grep 'mnt'`" ]
+then
+    echo "/mnt mount point is in use?"
+    exit 1
+fi
+
+if ! [ -b "$1" ]
+then
+    echo "need to specify a target boot partition (/dev/sdb1)"
+    exit 1
+else
+    BOOTDEV=$1
+fi
+
+if ! [ -b "$2" ]
+then
+    echo "need to specify a target root partition (/dev/sdb2)"
+    exit 1
+else
+    ROOTDEV=$2
+fi
+
+echo "mounting rootdev..."
+mount $ROOTDEV /mnt
+echo "rsyncing over rootfs"
+rsync -atv --progress --delete --exclude=boot build/armel/ /mnt/
+echo "sync filesystems..."
+sync
+echo "un-mounting rootdev..."
+umount /mnt
+
+echo "mounting bootdev..."
+mount $BOOTDEV /mnt
+echo "copying boot files..."
+cp -v build/armel/boot/* /mnt
+echo "sync filesystems..."
+sync
+echo "un-mounting bootdev..."
+umount /mnt
diff --git a/freedom-maker/copy2img.sh b/freedom-maker/copy2img.sh
new file mode 100755
index 0000000..c11b0a6
--- /dev/null
+++ b/freedom-maker/copy2img.sh
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+set -e
+
+if [ -b "$1" ]
+then
+    DEV=$1
+else
+    echo "need to specify a card device (eg, /dev/sdb)"
+    exit 1
+fi
+
+NAME="torouter-dev"
+if [ -n "$2" ]
+then
+    NAME=$2
+fi
+
+echo "dd-ing $DEV to $NAME.img"
+dd if=$DEV of=$NAME.img bs=1M
+echo "gzip compressing image"
+gzip $NAME.img
+
diff --git a/freedom-maker/mk_dreamplug_rootfs b/freedom-maker/mk_dreamplug_rootfs
index bc75855..0c452e4 100755
--- a/freedom-maker/mk_dreamplug_rootfs
+++ b/freedom-maker/mk_dreamplug_rootfs
@@ -85,6 +85,7 @@ mkdir -p $pkgcache
 mkdir -p $target/var/cache/apt/ && mount -o bind $pkgcache $target/var/cache/apt/
 mkdir -p $target/var/cache/apt/archives
 mkdir -p $target/usr/bin
+mkdir -p $homedir
 
 # multistrap
 echo "Multistrapping..."
diff --git a/freedom-maker/multistrap-configs/torouter-armel.conf b/freedom-maker/multistrap-configs/torouter-armel.conf
new file mode 100644
index 0000000..9931e1e
--- /dev/null
+++ b/freedom-maker/multistrap-configs/torouter-armel.conf
@@ -0,0 +1,14 @@
+[General]
+include=torouter-base.conf
+arch=armel
+aptsources=Debian torproject torouter armel 
+debootstrap=Debian torproject torouter armel
+
+[armel]
+packages=linux-image-3.2.0-3-kirkwood flash-kernel u-boot-tools u-boot wireless-tools
+source=http://http.debian.net/debian/
+keyring=debian-archive-keyring
+suite=wheezy
+components=main
+omitdebsrc=false
+
diff --git a/freedom-maker/multistrap-configs/torouter-base.conf b/freedom-maker/multistrap-configs/torouter-base.conf
new file mode 100644
index 0000000..acfa6ce
--- /dev/null
+++ b/freedom-maker/multistrap-configs/torouter-base.conf
@@ -0,0 +1,29 @@
+[General]
+cleanup=false
+noauth=false
+unpack=true
+debootstrap=Debian torproject torouter
+aptsources=Debian torproject torouter
+
+[Debian]
+packages=base-files netbase openssh-server zile wget iproute net-tools hostname module-init-tools wget resolvconf udev isc-dhcp-client ifupdown devio initramfs-tools uboot-mkimage parted dosfstools apt iputils-ping haveged python pandoc python-simplejson make isc-dhcp-server dialog locales-all uaputl uapevent dnsmasq iptables vim-nox less screen lsof tcptraceroute traceroute mtr-tiny openntpd dnsmasq unbound python-augeas python-flask python-daemon
+source=http://http.debian.net/debian/
+keyring=debian-archive-keyring
+suite=wheezy
+components=main
+omitdebsrc=false
+
+[torproject]
+packages=tor tor-geoipdb libnatpmp-dev libnatpmp1 libminiupnpc-dev libminiupnpc5 tor-arm
+source=http://deb.torproject.org/torproject.org/
+keyring=deb.torproject.org-keyring
+suite=wheezy
+components=main
+omitdebsrc=false
+
+[torouter]
+packages=ttdnsd torouter-tui torouter-prep
+source=http://torrouter.torproject.org/torrouter/
+suite=torrouter
+components=main
+omitdebsrc=false
diff --git a/freedom-maker/source/etc/init.d/first-run b/freedom-maker/source/etc/init.d/first-run
index 8df2c2d..c811ae0 100755
--- a/freedom-maker/source/etc/init.d/first-run
+++ b/freedom-maker/source/etc/init.d/first-run
@@ -64,8 +64,8 @@ create-keys
 # wifi-ap-setup
 
 # the last things we do before quitting.
-tweak-kernel
+#tweak-kernel
 remove-self
 
-echo "Kernel flashed.  Rebooting."
-reboot
+#echo "Kernel flashed.  Rebooting."
+#reboot
diff --git a/freedom-maker/source/install.sh b/freedom-maker/source/install.sh
index 76547ba..fc13d2b 100644
--- a/freedom-maker/source/install.sh
+++ b/freedom-maker/source/install.sh
@@ -19,7 +19,8 @@ dpkg --get-selections > /tmp/selections
 mkdir -p /sourcecode
 cd sourcecode
 cut -f 1 < /tmp/selections | cut -d ':' -f 1 > /tmp/packages
-apt-get source --download-only `cat /tmp/packages`
+# TODO: torouter
+#apt-get source --download-only `cat /tmp/packages`
 
 # sshd may be left running by the postinst, clean that up
 /etc/init.d/ssh stop
diff --git a/freedom-maker/source/tmp/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.apt-key b/freedom-maker/source/tmp/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.apt-key
new file mode 100644
index 0000000..5b6a4d3
--- /dev/null
+++ b/freedom-maker/source/tmp/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.apt-key