stage2: Noobs export

* build.sh: Support comments in package files

This patch allows the use of hash comments inside patch files.  It's a
little ugly, but it strips comments and collapses all whitespace down to
single space characters between package names.  It handles comments
anywhere in a line, as well.

Was unsure if \ continuation of the long sed line or the inclusion of a
couple of lines of comments explaining what the sed expressions are
doing would be appreciated, so didn't include them in this patch.

* build.sh: whitespace fix

* build.sh: Use sed script for packages files

Broke the sed expressions out of build.sh and put them their own
documented sed script.  This greatly improves readability and avoids
build.sh getting messier.

Broke the substitution command into two separate subs.  The first just
deletes comments, and the second collapses all whitespace into a single
space.  This too is easier to read, and catches a couple of edge cases
that would result it not all whitespace being collapsed.  The result may
still have (one) leading and/or trailing space, which is acceptable.

License.md

@@ -0,0 +1,13 @@
+Copyright (c) 2015 Raspberry Pi (Trading) Ltd.
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1,10 +1,78 @@
 #TODO
 
-1. Image export
-1. NOOBS export
 1. Simplify running a single stage
 1. Documentation
 
 #Dependencies
 
-`quilt kpartx realpath qemu-user-static debootstrap zerofree`
+`quilt kpartx realpath qemu-user-static debootstrap zerofree pxz zip`
+
+#Config
+
+Upon execution, `build.sh` will source the file `config` in the current
+working directory.  This bash shell fragment is intended to set needed
+environment variables.
+
+The following environment variables are supported:
+
+ * `IMG_NAME`, the name of the distribution to build (required)
+ * `APT_PROXY`, proxy/cache URL to be included in the build
+
+A simple example for building Raspbian:
+
+```bash
+IMG_NAME='Raspbian'
+```
+
+#Stage Anatomy
+
+
+
+#Raspbian Stage Overview
+
+The build of Raspbian is divided up into several stages for logical clarity
+and modularity.  This causes some initial complexity, but it simplifies
+maintenance and allows for more easy customization.
+
+ - Stage 0, bootstrap.  The primary purpose of this stage is to create a
+   usable filesystem.  This is accomplished largely through the use of
+   `debootstrap`, which creates a minimal filesystem suitable for use as a
+   base.tgz on Debian systems.  This stage also configures apt settings and
+   installs `raspberrypi-bootloader` which is missed by debootstrap.  The
+   minimal core is installed but not configured, and the system will not quite
+   boot yet.
+
+ - Stage 1, truly minimal system.  This stage makes the system bootable by
+   installing system files like `/etc/fstab`, configures the bootloader, makes
+   the network operable, and installs packages like raspi-config.  At this
+   stage the system should boot to a local console from which you have the
+   means to perform basic tasks needed to configure and install the system.
+   This is as minimal as a system can possibly get, and its arguably not
+   really usable yet in a traditional sense yet.  Still, if you want minimal,
+   this is minimal and the rest you could reasonably do yourself as sysadmin.
+
+ - State 2, lite system.  This stage produces the Raspbian-Lite image.  It
+   installs some optimized memory functions, sets timezone and charmap
+   defaults, installs fake-hwclock and ntp, wifi and bluetooth support,
+   dphys-swapfile, and other basics for managing the hardware.  It also
+   creates necessary groups and gives the pi user access to sudo and the
+   standard console hardware permission groups.
+
+   There are a few tools that may not make a whole lot of sense here for
+   development purposes on a minimal system such as basic python and lua
+   packages as well as the `build-essential` package.  They are lumped right
+   in with more essential packages presently, though they need not be with
+   pi-gen.  These are understandable for Raspbian's target audience, but if
+   you were looking for something between truly minimal and Raspbian-lite,
+   here's where you start trimming.
+
+ - Stage 3, desktop system.  Here's where you get the full desktop system
+   with X11 and LXDE, web browsers, git for development, Raspbian custom UI
+   enhancements, etc.  This is a base desktop system, with some development
+   tools installed.
+
+ - Stage 4, complete Raspbian system.  More development tools, an email
+   client, learning tools like Scratch, specialized packages like sonic-pi and
+   wolfram-engine, system documentation, office productivity, etc.  This is
+   the stage that installs all of the things that make Raspbian friendly to
+   new users.

build.sh

@@ -16,7 +16,7 @@ EOF
 		fi
 		if [ -f ${i}-packages-nr ]; then
 			log "Begin ${SUB_STAGE_DIR}/${i}-packages-nr"
-			PACKAGES=`cat $i-packages-nr | tr '\n' ' '`
+			PACKAGES="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < ${i}-packages-nr)"
 			if [ -n "$PACKAGES" ]; then
 				on_chroot sh -e - << EOF
 apt-get install --no-install-recommends -y $PACKAGES
@@ -26,7 +26,7 @@ EOF
 		fi
 		if [ -f ${i}-packages ]; then
 			log "Begin ${SUB_STAGE_DIR}/${i}-packages"
-			PACKAGES=`cat $i-packages | tr '\n' ' '`
+			PACKAGES="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < ${i}-packages)"
 			if [ -n "$PACKAGES" ]; then
 				on_chroot sh -e - << EOF
 apt-get install -y $PACKAGES
@@ -42,8 +42,9 @@ EOF
 				rm -rf *-pc
 			fi
 			QUILT_PATCHES=${SUB_STAGE_DIR}/${i}-patches
-			mkdir -p ${i}-pc
-			ln -sf ${i}-pc .pc
+			SUB_STAGE_QUILT_PATCH_DIR="$(basename $SUB_STAGE_DIR)-pc"
+			mkdir -p $SUB_STAGE_QUILT_PATCH_DIR
+			ln -snf $SUB_STAGE_QUILT_PATCH_DIR .pc
 			if [ -e ${SUB_STAGE_DIR}/${i}-patches/EDIT ]; then
 				echo "Dropping into bash to edit patches..."
 				bash
@@ -76,6 +77,7 @@ EOF
 	log "End ${SUB_STAGE_DIR}"
 }
 
+
 run_stage(){
 	log "Begin ${STAGE_DIR}"
 	STAGE=$(basename ${STAGE_DIR})
@@ -155,6 +157,10 @@ export QUILT_NO_DIFF_TIMESTAMPS=1
 export QUILT_REFRESH_ARGS="-p ab"
 
 source ${SCRIPT_DIR}/common
+source ${SCRIPT_DIR}/dependencies_check
+
+
+dependencies_check ${BASE_DIR}/depends
 
 mkdir -p ${WORK_DIR}
 log "Begin ${BASE_DIR}"

depends

@@ -0,0 +1,5 @@
+quilt
+qemu-arm-static:qemu-user-static
+debootstrap
+kpartx zerofree
+pxz zip

export-noobs/prerun.sh

@@ -4,6 +4,7 @@ IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
 NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"
 unmount_image ${IMG_FILE}
 
+mkdir -p ${STAGE_WORK_DIR}
 cp ${WORK_DIR}/export-image/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img ${STAGE_WORK_DIR}/
 
 rm -rf ${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}

scripts/dependencies_check

@@ -0,0 +1,30 @@
+# dependencies_check
+# $@	Dependnecy files to check
+#
+# Each dependency is in the form of a tool to test for, optionally followed by
+# a : and the name of a package if the package on a Debian-ish system is not
+# named for the tool (i.e., qemu-user-static).
+dependencies_check()
+{
+	local depfile deps missing
+
+	for depfile in "$@"; do
+		if [[ -e "$depfile" ]]; then
+			deps="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < ${BASE_DIR}/depends)"
+
+		fi
+		for dep in $deps; do
+			if ! hash ${dep%:*} 2>/dev/null; then
+				missing="${missing:+$missing }${dep#*:}"
+			fi
+		done
+	done
+
+	if [[ "$missing" ]]; then
+		echo "Reqired dependencies not installed"
+		echo
+		echo "This can be resolved on Debian/Raspbian systems by installing:"
+		echo "$missing"
+		false
+	fi
+}

scripts/remove-comments.sed

@@ -0,0 +1,11 @@
+# Deletes comments and collapses whitespace in ##-packages files
+
+# Append (N)ext line to buffer
+# if (!)not ($)buffer is EOF, (b)ranch to (:)label loop
+:loop
+N
+$ !b loop
+
+# Buffer is "line1\nline2\n...lineN", del comments and collapse whitespace
+s/#[^\n]*//g
+s/[[:space:]]\{1,\}/ /g

stage2/01-sys-tweaks/01-run.sh

@@ -8,7 +8,6 @@ install -d						${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d
 install -m 644 files/ttyoutput.conf			${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d/
 
 install -m 644 files/50raspi				${ROOTFS_DIR}/etc/apt/apt.conf.d/
-install -m 644 files/98-rpi.conf			${ROOTFS_DIR}/etc/sysctl.d/
 
 
 on_chroot sh -e - <<EOF

stage2/01-sys-tweaks/files/98-rpi.conf

@@ -1,3 +0,0 @@
-kernel.printk = 3 4 1 3
-vm.swappiness=1
-vm.min_free_kbytes = 8192

stage3/00-install-packages/00-packages

@@ -6,3 +6,5 @@ omxplayer
 raspberrypi-artwork
 policykit-1
 udisks
+rfkill
+chromium-browser rpi-chromium-mods

stage3/00-install-packages/01-run.sh

@@ -0,0 +1,9 @@
+#!/bin/bash -e
+
+on_chroot sh -e - <<EOF
+update-alternatives --install /usr/bin/x-www-browser \
+  x-www-browser /usr/bin/chromium-browser 86
+update-alternatives --install /usr/bin/gnome-www-browser \
+  gnome-www-browser /usr/bin/chromium-browser 86
+EOF
+

stage4/00-install-packages/00-packages

@@ -27,3 +27,10 @@ piclone
 pulseaudio-module-bluetooth
 wiringpi pigpio python-pigpio python3-pigpio raspi-gpio python-gpiozero python3-gpiozero python3-rpi.gpio
 python-spidev python3-spidev
+realvnc-vnc-server realvnc-vnc-viewer
+python-twython python3-twython
+python-smbus python3-smbus
+python-flask python3-flask
+python-picraft python3-picraft
+python3-codebug-tether python3-codebug-i2c-tether
+python-sense-emu python3-sense-emu sense-emu-tools python-sense-emu-doc

stage4/00-install-packages/01-run.sh

@@ -0,0 +1,3 @@
+#!/bin/bash -e
+
+rm -f "${ROOTFS_DIR}/root/.vnc/private.key"