Update release notes

Move python-games to Raspbian Full and install as a Debian package.

.dockerignore

@@ -1,3 +1,5 @@
 output/
 work/
 deploy/
+apt-cacher-ng/
+.git/objects/*

.gitignore

@@ -1,6 +1,9 @@
 deploy/*
 work/*
 config
+postrun.sh
 SKIP
+SKIP_IMAGES
 .pc
 *-pc
+apt-cacher-ng/

Dockerfile

@@ -1,10 +1,12 @@
-FROM debian:jessie
+FROM debian:stretch
+
+ENV DEBIAN_FRONTEND noninteractive
 
 RUN apt-get -y update && \
     apt-get -y install \
         git vim parted \
         quilt realpath qemu-user-static debootstrap zerofree pxz zip dosfstools \
-        bsdtar libcap2-bin rsync grep \
+        bsdtar libcap2-bin rsync grep udev xz-utils curl xxd file \
     && rm -rf /var/lib/apt/lists/*
 
 COPY . /pi-gen/

README.md

@@ -1,12 +1,24 @@
 # pi-gen
+
 _Tool used to create the raspberrypi.org Raspbian images_
 
-### TODO
-1. Documentation
 
 ## Dependencies
 
-`quilt parted realpath qemu-user-static debootstrap zerofree pxz zip dosfstools bsdtar libcap2-bin grep rsync`
+pi-gen runs on Debian based operating systems. Currently it is only supported on
+either Debian Stretch or Ubuntu Xenial and is known to have issues building on
+earlier releases of these systems.
+
+To install the required dependencies for pi-gen you should run:
+
+```bash
+apt-get install quilt parted realpath qemu-user-static debootstrap zerofree pxz zip \
+dosfstools bsdtar libcap2-bin grep rsync xz-utils file git curl
+```
+
+The file `depends` contains a list of tools needed.  The format of this
+package is `<tool>[:<debian-package>]`.
+
 
 ## Config
 
@@ -29,29 +41,121 @@ The following environment variables are supported:
    will not be included in the image, making it safe to use an `apt-cacher` or
    similar package for development.
 
+   If you have Docker installed, you can set up a local apt caching proxy to
+   like speed up subsequent builds like this:
+
+       docker-compose up -d
+       echo 'APT_PROXY=http://172.17.0.1:3142' >> config
+
+ * `BASE_DIR`  (Default: location of `build.sh`)
+
+   **CAUTION**: Currently, changing this value will probably break build.sh
+
+   Top-level directory for `pi-gen`.  Contains stage directories, build
+   scripts, and by default both work and deployment directories.
+
+ * `WORK_DIR`  (Default: `"$BASE_DIR/work"`)
+
+   Directory in which `pi-gen` builds the target system.  This value can be
+   changed if you have a suitably large, fast storage location for stages to
+   be built and cached.  Note, `WORK_DIR` stores a complete copy of the target
+   system for each build stage, amounting to tens of gigabytes in the case of
+   Raspbian.
+   
+   **CAUTION**: If your working directory is on an NTFS partition you probably won't be able to build. Make sure this is a proper Linux filesystem.
+
+ * `DEPLOY_DIR`  (Default: `"$BASE_DIR/deploy"`)
+
+   Output directory for target system images and NOOBS bundles.
+
+ * `USE_QEMU` (Default: `"0"`)
+
+   Setting to '1' enables the QEMU mode - creating an image that can be mounted via QEMU for an emulated
+   environment. These images include "-qemu" in the image file name.
+
+
 A simple example for building Raspbian:
 
 ```bash
 IMG_NAME='Raspbian'
 ```
 
+
+## How the build process works
+
+The following process is followed to build images:
+
+ * Loop through all of the stage directories in alphanumeric order
+
+ * Move on to the next directory if this stage directory contains a file called
+   "SKIP"
+
+ * Run the script ```prerun.sh``` which is generally just used to copy the build
+   directory between stages.
+
+ * In each stage directory loop through each subdirectory and then run each of the
+   install scripts it contains, again in alphanumeric order. These need to be named
+   with a two digit padded number at the beginning.
+   There are a number of different files and directories which can be used to
+   control different parts of the build process:
+
+     - **00-run.sh** - A unix shell script. Needs to be made executable for it to run.
+
+     - **00-run-chroot.sh** - A unix shell script which will be run in the chroot
+       of the image build directory. Needs to be made executable for it to run.
+
+     - **00-debconf** - Contents of this file are passed to debconf-set-selections
+       to configure things like locale, etc.
+
+     - **00-packages** - A list of packages to install. Can have more than one, space
+       separated, per line.
+
+     - **00-packages-nr** - As 00-packages, except these will be installed using
+       the ```--no-install-recommends -y``` parameters to apt-get.
+
+     - **00-patches** - A directory containing patch files to be applied, using quilt.
+       If a file named 'EDIT' is present in the directory, the build process will
+       be interrupted with a bash session, allowing an opportunity to create/revise
+       the patches.
+
+  * If the stage directory contains files called "EXPORT_NOOBS" or "EXPORT_IMAGE" then
+    add this stage to a list of images to generate
+
+  * Generate the images for any stages that have specified them
+
+It is recommended to examine build.sh for finer details.
+
+
 ## Docker Build
 
 ```bash
 vi config         # Edit your config file. See above.
 ./build-docker.sh
 ```
+
 If everything goes well, your finished image will be in the `deploy/` folder.
-You can then remove the build container with `docker rm pigen_work`
+You can then remove the build container with `docker rm -v pigen_work`
 
 If something breaks along the line, you can edit the corresponding scripts, and
 continue:
 
-```
+```bash
 CONTINUE=1 ./build-docker.sh
 ```
 
-There is a possibility that even when running from a docker container, the installation of `qemu-user-static` will silently fail when building the image because `binfmt-support` _must be enabled on the underlying kernel_. An easy fix is to ensure `binfmt-support` is installed on the host machine before starting the `./build-docker.sh` script (or using your own docker build solution).
+After successful build, the build container is by default removed. This may be undesired when making incremental changes to a customized build. To prevent the build script from remove the container add
+
+```bash
+PRESERVE_CONTAINER=1 ./build-docker.sh
+```
+
+There is a possibility that even when running from a docker container, the
+installation of `qemu-user-static` will silently fail when building the image
+because `binfmt-support` _must be enabled on the underlying kernel_. An easy
+fix is to ensure `binfmt-support` is installed on the host machine before
+starting the `./build-docker.sh` script (or using your own docker build
+solution).
+
 
 ## Stage Anatomy
 
@@ -86,11 +190,11 @@ maintenance and allows for more easy customization.
    standard console hardware permission groups.
 
    There are a few tools that may not make a whole lot of sense here for
-   development purposes on a minimal system such as basic python and lua
+   development purposes on a minimal system such as basic Python and Lua
    packages as well as the `build-essential` package.  They are lumped right
    in with more essential packages presently, though they need not be with
    pi-gen.  These are understandable for Raspbian's target audience, but if
-   you were looking for something between truly minimal and Raspbian-lite,
+   you were looking for something between truly minimal and Raspbian-Lite,
    here's where you start trimming.
 
  - **Stage 3** - desktop system.  Here's where you get the full desktop system
@@ -98,20 +202,73 @@ maintenance and allows for more easy customization.
    enhancements, etc.  This is a base desktop system, with some development
    tools installed.
 
- - **Stage 4** - complete Raspbian system.  More development tools, an email
-   client, learning tools like Scratch, specialized packages like sonic-pi and
-   wolfram-engine, system documentation, office productivity, etc.  This is
-   the stage that installs all of the things that make Raspbian friendly to
-   new users.
+ - **Stage 4** - Raspbian system meant to fit on a 4GB card.  More development
+   tools, an email client, learning tools like Scratch, specialized packages
+   like sonic-pi, system documentation, office productivity, etc.  This is the
+   stage that installs all of the things that make Raspbian friendly to new
+   users.
+
+ - **Stage 5** - The official Raspbian Desktop image. Right now only adds
+   Mathematica.
 
 ### Stage specification
-If you wish to build up to a specified stage (such as building up to stage 2 for a lite system), place an empty file named `SKIP` in each of the `./stage` directories you wish not to include.
 
-Then remove the `EXPORT*` files from `./stage4` (if building up to stage 2) or from `./stage2` (if building a minimal system).
+If you wish to build up to a specified stage (such as building up to stage 2
+for a lite system), place an empty file named `SKIP` in each of the `./stage`
+directories you wish not to include.
 
-```
+Then add an empty file named `SKIP_IMAGES` to `./stage4` (if building up to stage 2) or
+to `./stage2` (if building a minimal system).
+
+```bash
 # Example for building a lite system
-$ touch ./stage3/SKIP ./stage4/SKIP
-$ rm stage4/EXPORT*
+echo "IMG_NAME='Raspbian'" > config
+touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP
+touch ./stage4/SKIP_IMAGES ./stage5/SKIP_IMAGES
+sudo ./build.sh  # or ./build-docker.sh
 ```
-If you wish to build further configurations upon (for example) the lite system, you can also delete the contents of `./stage3` and `./stage4` and replace with your own contents in the same format.
+
+If you wish to build further configurations upon (for example) the lite
+system, you can also delete the contents of `./stage3` and `./stage4` and
+replace with your own contents in the same format.
+
+
+## Skipping stages to speed up development
+
+If you're working on a specific stage the recommended development process is as
+follows:
+
+ * Add a file called SKIP_IMAGES into the directories containing EXPORT_* files
+   (currently stage2, stage4 and stage5)
+ * Add SKIP files to the stages you don't want to build. For example, if you're
+   basing your image on the lite image you would add these to stages 3, 4 and 5.
+ * Run build.sh to build all stages
+ * Add SKIP files to the earlier successfully built stages
+ * Modify the last stage
+ * Rebuild just the last stage using ```sudo CLEAN=1 ./build.sh```
+ * Once you're happy with the image you can remove the SKIP_IMAGES files and
+   export your image to test
+
+# Troubleshooting
+
+## `binfmt_misc`
+
+Linux is able execute binaries from other architectures, meaning that it should be
+possible to make use of `pi-gen` on an x86_64 system, even though it will be running
+ARM binaries. This requires support from the [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc)
+kernel module.
+
+You may see the following error:
+
+```
+update-binfmts: warning: Couldn't load the binfmt_misc module.
+```
+
+To resolve this, ensure that the following files are available (install them if necessary):
+
+```
+/lib/modules/$(uname -r)/kernel/fs/binfmt_misc.ko
+/usr/bin/qemu-arm-static
+```
+
+You may also need to load the module by hand - run `modprobe binfmt_misc`.

build-docker.sh

@@ -1,4 +1,5 @@
 #!/bin/bash -e
+
 DOCKER="docker"
 set +e
 $DOCKER ps >/dev/null 2>&1
@@ -12,18 +13,19 @@ if ! $DOCKER ps >/dev/null; then
 fi
 set -e
 
-config_mount=
+config_file=()
 if [ -f config ]; then
-	config_mount="-v $(pwd)/config:/pi-gen/config:ro"
+	config_file=("--env-file" "$(pwd)/config")
 	source config
 fi
 
 CONTAINER_NAME=${CONTAINER_NAME:-pigen_work}
 CONTINUE=${CONTINUE:-0}
+PRESERVE_CONTAINER=${PRESERVE_CONTAINER:-0}
 
 if [ "$*" != "" ] || [ -z "${IMG_NAME}" ]; then
 	if [ -z "${IMG_NAME}" ]; then
-		echo "IMG_NAME not set in 'build'" 1>&2
+		echo "IMG_NAME not set in 'config'" 1>&2
 		echo 1>&2
 	fi
 	cat >&2 <<EOF
@@ -31,7 +33,8 @@ Usage:
     build-docker.sh [options]
 Optional environment arguments: ( =<default> )
     CONTAINER_NAME=pigen_work  set a name for the build container
-    CONTINUE=0                  continue from a previously started container
+    CONTINUE=1                 continue from a previously started container
+    PRESERVE_CONTAINER=1       keep build container even on successful build
 EOF
 	exit 1
 fi
@@ -45,29 +48,39 @@ fi
 if [ "$CONTAINER_EXISTS" != "" ] && [ "$CONTINUE" != "1" ]; then
 	echo "Container $CONTAINER_NAME already exists and you did not specify CONTINUE=1. Aborting."
 	echo "You can delete the existing container like this:"
-	echo "  docker rm $CONTAINER_NAME"
+	echo "  $DOCKER rm -v $CONTAINER_NAME"
 	exit 1
 fi
 
 $DOCKER build -t pi-gen .
 if [ "$CONTAINER_EXISTS" != "" ]; then
-	trap "echo 'got CTRL+C... please wait 5s';docker stop -t 5 ${CONTAINER_NAME}_cont" SIGINT SIGTERM
+	trap "echo 'got CTRL+C... please wait 5s'; $DOCKER stop -t 5 ${CONTAINER_NAME}_cont" SIGINT SIGTERM
 	time $DOCKER run --rm --privileged \
 		--volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \
+		-e IMG_NAME="${IMG_NAME}"\
 		pi-gen \
 		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
 	cd /pi-gen; ./build.sh;
 	rsync -av work/*/build.log deploy/" &
-	wait
+	wait "$!"
 else
-	trap "echo 'got CTRL+C... please wait 5s'; docker stop -t 5 ${CONTAINER_NAME}" SIGINT SIGTERM
-	$DOCKER run --name "${CONTAINER_NAME}" --privileged \
-		-v $(pwd)/deploy:/pi-gen/deploy \
-		${config_mount} \
+	trap "echo 'got CTRL+C... please wait 5s'; $DOCKER stop -t 5 ${CONTAINER_NAME}" SIGINT SIGTERM
+	time $DOCKER run --name "${CONTAINER_NAME}" --privileged \
+		-e IMG_NAME="${IMG_NAME}"\
+		"${config_file[@]}" \
 		pi-gen \
 		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
 	cd /pi-gen; ./build.sh &&
 	rsync -av work/*/build.log deploy/" &
-	wait
+	wait "$!"
 fi
+echo "copying results from deploy/"
+$DOCKER cp "${CONTAINER_NAME}":/pi-gen/deploy .
+ls -lah deploy
+
+# cleanup
+if [ "$PRESERVE_CONTAINER" != "1" ]; then
+	$DOCKER rm -v $CONTAINER_NAME
+fi
+
 echo "Done! Your image(s) should be in deploy/"

build.sh

@@ -1,22 +1,23 @@
 #!/bin/bash -e
-
+# shellcheck disable=SC2119,SC1091
 run_sub_stage()
 {
 	log "Begin ${SUB_STAGE_DIR}"
-	pushd ${SUB_STAGE_DIR} > /dev/null
+	pushd "${SUB_STAGE_DIR}" > /dev/null
 	for i in {00..99}; do
-		if [ -f ${i}-debconf ]; then
+		if [ -f "${i}-debconf" ]; then
 			log "Begin ${SUB_STAGE_DIR}/${i}-debconf"
 			on_chroot << EOF
 debconf-set-selections <<SELEOF
-`cat ${i}-debconf`
+$(cat "${i}-debconf")
 SELEOF
 EOF
+
 		log "End ${SUB_STAGE_DIR}/${i}-debconf"
 		fi
-		if [ -f ${i}-packages-nr ]; then
+		if [ -f "${i}-packages-nr" ]; then
 			log "Begin ${SUB_STAGE_DIR}/${i}-packages-nr"
-			PACKAGES="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < ${i}-packages-nr)"
+			PACKAGES="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < "${i}-packages-nr")"
 			if [ -n "$PACKAGES" ]; then
 				on_chroot << EOF
 apt-get install --no-install-recommends -y $PACKAGES
@@ -24,9 +25,9 @@ EOF
 			fi
 			log "End ${SUB_STAGE_DIR}/${i}-packages-nr"
 		fi
-		if [ -f ${i}-packages ]; then
+		if [ -f "${i}-packages" ]; then
 			log "Begin ${SUB_STAGE_DIR}/${i}-packages"
-			PACKAGES="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < ${i}-packages)"
+			PACKAGES="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < "${i}-packages")"
 			if [ -n "$PACKAGES" ]; then
 				on_chroot << EOF
 apt-get install -y $PACKAGES
@@ -34,18 +35,18 @@ EOF
 			fi
 			log "End ${SUB_STAGE_DIR}/${i}-packages"
 		fi
-		if [ -d ${i}-patches ]; then
+		if [ -d "${i}-patches" ]; then
 			log "Begin ${SUB_STAGE_DIR}/${i}-patches"
-			pushd ${STAGE_WORK_DIR} > /dev/null
+			pushd "${STAGE_WORK_DIR}" > /dev/null
 			if [ "${CLEAN}" = "1" ]; then
 				rm -rf .pc
-				rm -rf *-pc
+				rm -rf ./*-pc
 			fi
-			QUILT_PATCHES=${SUB_STAGE_DIR}/${i}-patches
-			SUB_STAGE_QUILT_PATCH_DIR="$(basename $SUB_STAGE_DIR)-pc"
-			mkdir -p $SUB_STAGE_QUILT_PATCH_DIR
-			ln -snf $SUB_STAGE_QUILT_PATCH_DIR .pc
-			if [ -e ${SUB_STAGE_DIR}/${i}-patches/EDIT ]; then
+			QUILT_PATCHES="${SUB_STAGE_DIR}/${i}-patches"
+			SUB_STAGE_QUILT_PATCH_DIR="$(basename "$SUB_STAGE_DIR")-pc"
+			mkdir -p "$SUB_STAGE_QUILT_PATCH_DIR"
+			ln -snf "$SUB_STAGE_QUILT_PATCH_DIR" .pc
+			if [ -e "${SUB_STAGE_DIR}/${i}-patches/EDIT" ]; then
 				echo "Dropping into bash to edit patches..."
 				bash
 			fi
@@ -67,10 +68,10 @@ EOF
 			./${i}-run.sh
 			log "End ${SUB_STAGE_DIR}/${i}-run.sh"
 		fi
-		if [ -f ${i}-run-chroot ]; then
-			log "Begin ${SUB_STAGE_DIR}/${i}-run-chroot"
-			on_chroot < ${i}-run-chroot
-			log "End ${SUB_STAGE_DIR}/${i}-run-chroot"
+		if [ -f ${i}-run-chroot.sh ]; then
+			log "Begin ${SUB_STAGE_DIR}/${i}-run-chroot.sh"
+			on_chroot < ${i}-run-chroot.sh
+			log "End ${SUB_STAGE_DIR}/${i}-run-chroot.sh"
 		fi
 	done
 	popd > /dev/null
@@ -80,18 +81,20 @@ EOF
 
 run_stage(){
 	log "Begin ${STAGE_DIR}"
-	STAGE=$(basename ${STAGE_DIR})
-	pushd ${STAGE_DIR} > /dev/null
-	unmount ${WORK_DIR}/${STAGE}
-	STAGE_WORK_DIR=${WORK_DIR}/${STAGE}
-	ROOTFS_DIR=${STAGE_WORK_DIR}/rootfs
-	if [ -f ${STAGE_DIR}/EXPORT_IMAGE ]; then
+	STAGE="$(basename "${STAGE_DIR}")"
+	pushd "${STAGE_DIR}" > /dev/null
+	unmount "${WORK_DIR}/${STAGE}"
+	STAGE_WORK_DIR="${WORK_DIR}/${STAGE}"
+	ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs
+	if [ ! -f SKIP_IMAGES ]; then
+		if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then
 			EXPORT_DIRS="${EXPORT_DIRS} ${STAGE_DIR}"
 		fi
+	fi
 	if [ ! -f SKIP ]; then
 		if [ "${CLEAN}" = "1" ]; then
-			if [ -d ${ROOTFS_DIR} ]; then
-				rm -rf ${ROOTFS_DIR}
+			if [ -d "${ROOTFS_DIR}" ]; then
+				rm -rf "${ROOTFS_DIR}"
 			fi
 		fi
 		if [ -x prerun.sh ]; then
@@ -100,16 +103,16 @@ run_stage(){
 			log "End ${STAGE_DIR}/prerun.sh"
 		fi
 		for SUB_STAGE_DIR in ${STAGE_DIR}/*; do
-			if [ -d ${SUB_STAGE_DIR} ] &&
-			   [ ! -f ${SUB_STAGE_DIR}/SKIP ]; then
+			if [ -d "${SUB_STAGE_DIR}" ] &&
+			   [ ! -f "${SUB_STAGE_DIR}/SKIP" ]; then
 				run_sub_stage
 			fi
 		done
 	fi
-	unmount ${WORK_DIR}/${STAGE}
-	PREV_STAGE=${STAGE}
-	PREV_STAGE_DIR=${STAGE_DIR}
-	PREV_ROOTFS_DIR=${ROOTFS_DIR}
+	unmount "${WORK_DIR}/${STAGE}"
+	PREV_STAGE="${STAGE}"
+	PREV_STAGE_DIR="${STAGE_DIR}"
+	PREV_ROOTFS_DIR="${ROOTFS_DIR}"
 	popd > /dev/null
 	log "End ${STAGE_DIR}"
 }
@@ -119,6 +122,7 @@ if [ "$(id -u)" != "0" ]; then
 	exit 1
 fi
 
+
 if [ -f config ]; then
 	source config
 fi
@@ -128,14 +132,17 @@ if [ -z "${IMG_NAME}" ]; then
 	exit 1
 fi
 
-export IMG_DATE=${IMG_DATE:-"$(date -u +%Y-%m-%d)"}
+export USE_QEMU="${USE_QEMU:-0}"
+export IMG_DATE="${IMG_DATE:-"$(date +%Y-%m-%d)"}"
 
-export BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 export SCRIPT_DIR="${BASE_DIR}/scripts"
-export WORK_DIR="${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"
-export DEPLOY_DIR="${BASE_DIR}/deploy"
+export WORK_DIR="${WORK_DIR:-"${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"}"
+export DEPLOY_DIR=${DEPLOY_DIR:-"${BASE_DIR}/deploy"}
 export LOG_FILE="${WORK_DIR}/build.log"
 
+export BASE_DIR
+
 export CLEAN
 export IMG_NAME
 export APT_PROXY
@@ -148,6 +155,8 @@ export PREV_STAGE_DIR
 export ROOTFS_DIR
 export PREV_ROOTFS_DIR
 export IMG_SUFFIX
+export NOOBS_NAME
+export NOOBS_DESCRIPTION
 export EXPORT_DIR
 export EXPORT_ROOTFS_DIR
 
@@ -156,29 +165,43 @@ export QUILT_NO_DIFF_INDEX=1
 export QUILT_NO_DIFF_TIMESTAMPS=1
 export QUILT_REFRESH_ARGS="-p ab"
 
-source ${SCRIPT_DIR}/common
-source ${SCRIPT_DIR}/dependencies_check
+# shellcheck source=scripts/common
+source "${SCRIPT_DIR}/common"
+# shellcheck source=scripts/dependencies_check
+source "${SCRIPT_DIR}/dependencies_check"
 
 
-dependencies_check ${BASE_DIR}/depends
+dependencies_check "${BASE_DIR}/depends"
 
-mkdir -p ${WORK_DIR}
+mkdir -p "${WORK_DIR}"
 log "Begin ${BASE_DIR}"
 
-for STAGE_DIR in ${BASE_DIR}/stage*; do
+for STAGE_DIR in "${BASE_DIR}/stage"*; do
 	run_stage
 done
 
 CLEAN=1
 for EXPORT_DIR in ${EXPORT_DIRS}; do
 	STAGE_DIR=${BASE_DIR}/export-image
-	IMG_SUFFIX=$(cat ${EXPORT_DIR}/EXPORT_IMAGE)
-	EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename ${EXPORT_DIR})/rootfs
+	# shellcheck source=/dev/null
+	source "${EXPORT_DIR}/EXPORT_IMAGE"
+	EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs
 	run_stage
-	if [ -e ${EXPORT_DIR}/EXPORT_NOOBS ]; then
-		STAGE_DIR=${BASE_DIR}/export-noobs
+	if [ "${USE_QEMU}" != "1" ]; then
+		if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then
+			# shellcheck source=/dev/null
+			source "${EXPORT_DIR}/EXPORT_NOOBS"
+			STAGE_DIR="${BASE_DIR}/export-noobs"
 			run_stage
 		fi
+	fi
 done
 
+if [ -x postrun.sh ]; then
+	log "Begin postrun.sh"
+	cd "${BASE_DIR}"
+	./postrun.sh
+	log "End postrun.sh"
+fi
+
 log "End ${BASE_DIR}"

depends

@@ -10,3 +10,8 @@ capsh:libcap2-bin
 bsdtar
 grep
 rsync
+xz:xz-utils
+curl
+xxd
+file
+git

docker-compose.yml

@@ -0,0 +1,10 @@
+version: '2'
+
+services:
+  apt-cacher-ng:
+    restart: unless-stopped
+    image: sameersbn/apt-cacher-ng:latest
+    ports:
+    - "3142:3142"
+    volumes:
+    - ./apt-cacher-ng:/var/cache/apt-cacher-ng

export-image/00-allow-rerun/00-run.sh

@@ -1,13 +1,5 @@
 #!/bin/bash -e
 
-if [ -e ${ROOTFS_DIR}/etc/ld.so.preload ]; then
-	mv ${ROOTFS_DIR}/etc/ld.so.preload ${ROOTFS_DIR}/etc/ld.so.preload.disabled
-fi
-
-if [ ! -e ${ROOTFS_DIR}/usr/sbin/policy-rc.d ]; then
-	install -m 744 files/policy-rc.d ${ROOTFS_DIR}/usr/sbin/
-fi
-
-if [ ! -x ${ROOTFS_DIR}/usr/bin/qemu-arm-static ]; then
-	cp /usr/bin/qemu-arm-static ${ROOTFS_DIR}/usr/bin/
+if [ ! -x "${ROOTFS_DIR}/usr/bin/qemu-arm-static" ]; then
+	cp /usr/bin/qemu-arm-static "${ROOTFS_DIR}/usr/bin/"
 fi

export-image/00-allow-rerun/files/policy-rc.d

@@ -1,2 +0,0 @@
-#!/bin/sh
-exit 101

export-image/01-set-sources/00-patches/0-sources.diff

@@ -3,7 +3,7 @@ Index: export-jessie/rootfs/etc/apt/sources.list.d/raspi.list
 --- export-jessie.orig/rootfs/etc/apt/sources.list.d/raspi.list
 +++ export-jessie/rootfs/etc/apt/sources.list.d/raspi.list
 @@ -1,3 +1,3 @@
--deb http://archive.raspberrypi.org/debian/ jessie main ui staging
-+deb http://archive.raspberrypi.org/debian/ jessie main ui
+-deb http://archive.raspberrypi.org/debian/ stretch main ui staging
++deb http://archive.raspberrypi.org/debian/ stretch main ui
  # Uncomment line below then 'apt-get update' to enable 'apt-get source'
- #deb-src http://archive.raspberrypi.org/debian/ jessie main ui
+ #deb-src http://archive.raspberrypi.org/debian/ stretch main ui

export-image/02-network/01-run.sh

@@ -1,3 +1,3 @@
-#/bin/bash -e
+#!/bin/bash -e
 
-install -m 644 files/resolv.conf ${ROOTFS_DIR}/etc/
+install -m 644 files/resolv.conf "${ROOTFS_DIR}/etc/"

export-image/03-finalise/01-run.sh

@@ -1,39 +0,0 @@
-#!/bin/bash -e
-
-IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
-
-on_chroot << EOF
-/etc/init.d/fake-hwclock stop
-hardlink -t /usr/share/doc
-EOF
-
-if [ -d ${ROOTFS_DIR}/home/pi/.config ]; then
-	chmod 700 ${ROOTFS_DIR}/home/pi/.config
-fi
-
-rm -f ${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache
-rm -f ${ROOTFS_DIR}/usr/sbin/policy-rc.d
-rm -f ${ROOTFS_DIR}/usr/bin/qemu-arm-static
-if [ -e ${ROOTFS_DIR}/etc/ld.so.preload.disabled ]; then
-        mv ${ROOTFS_DIR}/etc/ld.so.preload.disabled ${ROOTFS_DIR}/etc/ld.so.preload
-fi
-
-update_issue $(basename ${EXPORT_DIR})
-install -m 644 ${ROOTFS_DIR}/etc/rpi-issue ${ROOTFS_DIR}/boot/issue.txt
-install files/LICENSE.oracle ${ROOTFS_DIR}/boot/
-
-ROOT_DEV=$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')
-
-unmount ${ROOTFS_DIR}
-zerofree -v ${ROOT_DEV}
-
-unmount_image ${IMG_FILE}
-
-mkdir -p ${DEPLOY_DIR}
-
-rm -f ${DEPLOY_DIR}/image_${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.zip
-
-echo zip ${DEPLOY_DIR}/image_${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.zip ${IMG_FILE}
-pushd ${STAGE_WORK_DIR} > /dev/null
-zip ${DEPLOY_DIR}/image_${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.zip $(basename ${IMG_FILE})
-popd > /dev/null

export-image/03-set-partuuid/00-run.sh

@@ -0,0 +1,13 @@
+#!/bin/bash -e
+
+IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
+
+IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')"
+
+BOOT_PARTUUID="${IMGID}-01"
+ROOT_PARTUUID="${IMGID}-02"
+
+sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab"
+sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab"
+
+sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/cmdline.txt"

export-image/04-finalise/01-run.sh

@@ -0,0 +1,84 @@
+#!/bin/bash -e
+
+IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
+INFO_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.info"
+
+on_chroot << EOF
+/etc/init.d/fake-hwclock stop
+hardlink -t /usr/share/doc
+EOF
+
+if [ -d "${ROOTFS_DIR}/home/pi/.config" ]; then
+	chmod 700 "${ROOTFS_DIR}/home/pi/.config"
+fi
+
+rm -f "${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache"
+rm -f "${ROOTFS_DIR}/usr/bin/qemu-arm-static"
+
+rm -f "${ROOTFS_DIR}/etc/apt/sources.list~"
+rm -f "${ROOTFS_DIR}/etc/apt/trusted.gpg~"
+
+rm -f "${ROOTFS_DIR}/etc/passwd-"
+rm -f "${ROOTFS_DIR}/etc/group-"
+rm -f "${ROOTFS_DIR}/etc/shadow-"
+rm -f "${ROOTFS_DIR}/etc/gshadow-"
+rm -f "${ROOTFS_DIR}/etc/subuid-"
+rm -f "${ROOTFS_DIR}/etc/subgid-"
+
+rm -f "${ROOTFS_DIR}"/var/cache/debconf/*-old
+rm -f "${ROOTFS_DIR}"/var/lib/dpkg/*-old
+
+rm -f "${ROOTFS_DIR}"/usr/share/icons/*/icon-theme.cache
+
+rm -f "${ROOTFS_DIR}/var/lib/dbus/machine-id"
+
+true > "${ROOTFS_DIR}/etc/machine-id"
+
+ln -nsf /proc/mounts "${ROOTFS_DIR}/etc/mtab"
+
+find "${ROOTFS_DIR}/var/log/" -type f -exec cp /dev/null {} \;
+
+rm -f "${ROOTFS_DIR}/root/.vnc/private.key"
+rm -f "${ROOTFS_DIR}/etc/vnc/updateid"
+
+update_issue "$(basename "${EXPORT_DIR}")"
+install -m 644 "${ROOTFS_DIR}/etc/rpi-issue" "${ROOTFS_DIR}/boot/issue.txt"
+install files/LICENSE.oracle "${ROOTFS_DIR}/boot/"
+
+
+cp "$ROOTFS_DIR/etc/rpi-issue" "$INFO_FILE"
+
+
+{
+	firmware=$(zgrep "firmware as of" \
+		"$ROOTFS_DIR/usr/share/doc/raspberrypi-kernel/changelog.Debian.gz" | \
+		head -n1 | sed  -n 's|.* \([^ ]*\)$|\1|p')
+	printf "\nFirmware: https://github.com/raspberrypi/firmware/tree/%s\n" "$firmware"
+
+	kernel="$(curl -s -L "https://github.com/raspberrypi/firmware/raw/$firmware/extra/git_hash")"
+	printf "Kernel: https://github.com/raspberrypi/linux/tree/%s\n" "$kernel"
+
+	uname="$(curl -s -L "https://github.com/raspberrypi/firmware/raw/$firmware/extra/uname_string7")"
+
+	printf "Uname string: %s\n" "$uname"
+	printf "\nPackages:\n"
+	dpkg -l --root "$ROOTFS_DIR"
+} >> "$INFO_FILE"
+
+ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')"
+
+unmount "${ROOTFS_DIR}"
+zerofree -v "${ROOT_DEV}"
+
+unmount_image "${IMG_FILE}"
+
+mkdir -p "${DEPLOY_DIR}"
+
+rm -f "${DEPLOY_DIR}/image_${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.zip"
+
+pushd "${STAGE_WORK_DIR}" > /dev/null
+zip "${DEPLOY_DIR}/image_${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.zip" \
+	"$(basename "${IMG_FILE}")"
+popd > /dev/null
+
+cp "$INFO_FILE" "$DEPLOY_DIR"

export-image/prerun.sh

@@ -1,40 +1,43 @@
 #!/bin/bash -e
+
 IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
 
-unmount_image ${IMG_FILE}
+unmount_image "${IMG_FILE}"
 
-rm -f ${IMG_FILE}
+rm -f "${IMG_FILE}"
 
-rm -rf ${ROOTFS_DIR}
-mkdir -p ${ROOTFS_DIR}
+rm -rf "${ROOTFS_DIR}"
+mkdir -p "${ROOTFS_DIR}"
 
-BOOT_SIZE=$(du -sh ${EXPORT_ROOTFS_DIR}/boot -B M | cut -f 1 | tr -d M)
-TOTAL_SIZE=$(du -sh ${EXPORT_ROOTFS_DIR} -B M | cut -f 1 | tr -d M)
+BOOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}/boot" --block-size=1 | cut -f 1)
+TOTAL_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --block-size=1 | cut -f 1)
 
-IMG_SIZE=$(expr $BOOT_SIZE \* 2 \+ $TOTAL_SIZE \+ 512)M
+ROUND_SIZE="$((4 * 1024 * 1024))"
+ROUNDED_ROOT_SECTOR=$(((2 * BOOT_SIZE + ROUND_SIZE) / ROUND_SIZE * ROUND_SIZE / 512 + 8192))
+IMG_SIZE=$(((BOOT_SIZE + TOTAL_SIZE + (800 * 1024 * 1024) + ROUND_SIZE - 1) / ROUND_SIZE * ROUND_SIZE))
 
-fallocate -l ${IMG_SIZE} ${IMG_FILE}
-fdisk ${IMG_FILE} > /dev/null 2>&1 <<EOF
+truncate -s "${IMG_SIZE}" "${IMG_FILE}"
+fdisk -H 255 -S 63 "${IMG_FILE}" <<EOF
 o
 n
 
 
 8192
-+`expr $BOOT_SIZE \* 3`M
++$((BOOT_SIZE * 2 /512))
 p
 t
 c
 n
 
 
-8192
+${ROUNDED_ROOT_SECTOR}
 
 
 p
 w
 EOF
 
-PARTED_OUT=$(parted -s ${IMG_FILE} unit b print)
+PARTED_OUT=$(parted -s "${IMG_FILE}" unit b print)
 BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
 | cut -d" " -f 2 | tr -d B)
 BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
@@ -45,16 +48,22 @@ ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
 ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
 | cut -d" " -f 4 | tr -d B)
 
-BOOT_DEV=$(losetup --show -f -o ${BOOT_OFFSET} --sizelimit ${BOOT_LENGTH} ${IMG_FILE})
-ROOT_DEV=$(losetup --show -f -o ${ROOT_OFFSET} --sizelimit ${ROOT_LENGTH} ${IMG_FILE})
+BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGTH}" "${IMG_FILE}")
+ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGTH}" "${IMG_FILE}")
 echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH"
 echo "/:     offset $ROOT_OFFSET, length $ROOT_LENGTH"
 
-mkdosfs -n boot -F 32 -v $BOOT_DEV > /dev/null
-mkfs.ext4 -O ^huge_file $ROOT_DEV > /dev/null
+ROOT_FEATURES="^huge_file"
+for FEATURE in metadata_csum 64bit; do
+	if grep -q "$FEATURE" /etc/mke2fs.conf; then
+	    ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES"
+	fi
+done
+mkdosfs -n boot -F 32 -v "$BOOT_DEV" > /dev/null
+mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null
 
-mount -v $ROOT_DEV ${ROOTFS_DIR} -t ext4
-mkdir -p ${ROOTFS_DIR}/boot
-mount -v $BOOT_DEV ${ROOTFS_DIR}/boot -t vfat
+mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4
+mkdir -p "${ROOTFS_DIR}/boot"
+mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot" -t vfat
 
-rsync -aHAXx ${EXPORT_ROOTFS_DIR}/ ${ROOTFS_DIR}/
+rsync -aHAXx --exclude var/cache/apt/archives "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/"

export-noobs/00-release/00-run.sh

@@ -2,30 +2,35 @@
 
 NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"
 
-install -v -m 744	files/partition_setup.sh	${NOOBS_DIR}/
-install -v		files/partitions.json		${NOOBS_DIR}/
-install -v		files/os.json			${NOOBS_DIR}/
-install -v		files/Raspbian.png		${NOOBS_DIR}/
-install -v		files/release_notes.txt		${NOOBS_DIR}/
+install -v -m 744	files/partition_setup.sh	"${NOOBS_DIR}/"
+install -v		files/partitions.json		"${NOOBS_DIR}/"
+install -v		files/os.json			"${NOOBS_DIR}/"
+install -v		files/OS.png			"${NOOBS_DIR}/"
+install -v		files/release_notes.txt		"${NOOBS_DIR}/"
 
-tar -v -c -C		files/marketing			-f ${NOOBS_DIR}/marketing.tar .
+tar -v -c -C		files/marketing			-f "${NOOBS_DIR}/marketing.tar" .
 
-BOOT_SIZE=$(xz --robot -l ${NOOBS_DIR}/boot.tar.xz  | grep totals | cut -f 5)
-ROOT_SIZE=$(xz --robot -l ${NOOBS_DIR}/root.tar.xz  | grep totals | cut -f 5)
+BOOT_SIZE="$(xz --robot -l "${NOOBS_DIR}/boot.tar.xz"  | grep totals | cut -f 5)"
+ROOT_SIZE="$(xz --robot -l "${NOOBS_DIR}/root.tar.xz"  | grep totals | cut -f 5)"
 
-BOOT_SIZE=$(expr ${BOOT_SIZE} / 1000000 \+ 1)
-ROOT_SIZE=$(expr ${ROOT_SIZE} / 1000000 \+ 1)
+BOOT_SIZE="$(( BOOT_SIZE / 1024 / 1024 + 1))"
+ROOT_SIZE="$(( ROOT_SIZE / 1024 / 1024 + 1))"
 
-BOOT_NOM=$(expr ${BOOT_SIZE} \* 3)
-ROOT_NOM=$(expr ${ROOT_SIZE} \+ 400)
+BOOT_NOM="$(( BOOT_SIZE * 3 ))"
+ROOT_NOM="$(( ROOT_SIZE + 400 ))"
 
-sed ${NOOBS_DIR}/partitions.json -i -e "s|BOOT_SIZE|${BOOT_SIZE}|"
-sed ${NOOBS_DIR}/partitions.json -i -e "s|ROOT_SIZE|${ROOT_SIZE}|"
+mv "${NOOBS_DIR}/OS.png" "${NOOBS_DIR}/${NOOBS_NAME// /_}.png"
 
-sed ${NOOBS_DIR}/partitions.json -i -e "s|BOOT_NOM|${BOOT_NOM}|"
-sed ${NOOBS_DIR}/partitions.json -i -e "s|ROOT_NOM|${ROOT_NOM}|"
+sed "${NOOBS_DIR}/partitions.json" -i -e "s|BOOT_SIZE|${BOOT_SIZE}|"
+sed "${NOOBS_DIR}/partitions.json" -i -e "s|ROOT_SIZE|${ROOT_SIZE}|"
 
-sed ${NOOBS_DIR}/release_notes.txt -i -e "s|UNRELEASED|${IMG_DATE}|"
-sed ${NOOBS_DIR}/os.json -i -e "s|UNRELEASED|${IMG_DATE}|"
+sed "${NOOBS_DIR}/partitions.json" -i -e "s|BOOT_NOM|${BOOT_NOM}|"
+sed "${NOOBS_DIR}/partitions.json" -i -e "s|ROOT_NOM|${ROOT_NOM}|"
 
-cp -a ${NOOBS_DIR} ${DEPLOY_DIR}/
+sed "${NOOBS_DIR}/os.json" -i -e "s|UNRELEASED|${IMG_DATE}|"
+sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_NAME|${NOOBS_NAME}|"
+sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_DESCRIPTION|${NOOBS_DESCRIPTION}|"
+
+sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|"
+
+cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/"

export-noobs/00-release/files/os.json

@@ -1,8 +1,8 @@
 {
-    "description": "A community-created port of Debian jessie for the Raspberry Pi",
+    "description": "NOOBS_DESCRIPTION",
     "feature_level": 35120124,
-    "kernel": "4.4",
-    "name": "Raspbian",
+    "kernel": "4.14",
+    "name": "NOOBS_NAME",
     "password": "raspberry",
     "release_date": "UNRELEASED",
     "supported_hex_revisions": "2,3,4,5,6,7,8,9,d,e,f,10,11,12,14,19,1040,1041,0092,0093,2082",
@@ -10,9 +10,10 @@
         "Pi Model",
         "Pi 2",
         "Pi Zero",
-        "Pi 3"
+        "Pi 3",
+        "Pi Compute Module 3"
     ],
     "url": "http://www.raspbian.org/",
     "username": "pi",
-    "version": "jessie"
+    "version": "stretch"
 }

export-noobs/00-release/files/partition_setup.sh

@@ -12,20 +12,24 @@ mkdir -p /tmp/1 /tmp/2
 mount "$part1" /tmp/1
 mount "$part2" /tmp/2
 
-sed /tmp/1/cmdline.txt -i -e "s|root=/dev/[^ ]*|root=${part2}|"
+sed /tmp/1/cmdline.txt -i -e "s|root=[^ ]*|root=${part2}|"
 sed /tmp/2/etc/fstab -i -e "s|^.* / |${part2}  / |"
 sed /tmp/2/etc/fstab -i -e "s|^.* /boot |${part1}  /boot |"
 
-if [ -e /mnt/ssh ]; then
+if [ -f /mnt/ssh ]; then
   cp /mnt/ssh /tmp/1/
 fi
 
-if [ -e /mnt/ssh.txt ]; then
+if [ -f /mnt/ssh.txt ]; then
   cp /mnt/ssh.txt /tmp/1/
 fi
 
+if [ -f /settings/wpa_supplicant.conf ]; then
+  cp /settings/wpa_supplicant.conf /tmp/1/
+fi
+
 if ! grep -q resize /proc/cmdline; then
-  sed -i 's| quiet init=/usr/lib/raspi-config/init_resize.sh||' /tmp/1/cmdline.txt
+  sed -i 's| init=/usr/lib/raspi-config/init_resize.sh||;s| quiet||2g' /tmp/1/cmdline.txt
 fi
 
 umount /tmp/1

export-noobs/00-release/files/release_notes.txt

@@ -1,5 +1,168 @@
 UNRELEASED:
+  *
+2018-11-13:
+  * Two versions of image created - "base" image has no optional software packages included; "full" image has all optional packages
+    - Removed from "base" image - LibreOffice, Thonny, Scratch, Scratch 2, Sonic Pi, Minecraft, Python Games, SmartSim, SenseHAT Emulator
+    - Added to "full" image - Mathematica, BlueJ, Greenfoot, Node-RED, Claws Mail, VNC Viewer
+  * Python Games and SmartSim added to Recommended Software
+  * VLC media player with VideoCore hardware acceleration included in image
+  * Version 3.0.5 of Thonny included
+  * Modifications to LXDE components to enable local configuration to override global configuration correctly
+  * Modifications to Appearance Settings to support above configuration changes
+  * Modifications to various initial config defaults and relevant package to support above configuration changes
+  * Selecting default option in Appearance Settings now deletes relevant local configuration files
+  * PiX theme modified so that all changes made in Appearance Settings are in override files rather than in theme files
+  * Design of scrollbar buttons changed
+  * Image Viewer moved into Graphics category on main menu
+  * Recommended Software now installs LibreOffice language support files if needed, and suggests reboot if needed
+  * Latest version of Pepper Flash plugin included
+  * Chromium h264ify plugin permissions set correctly by default
+  * Corrections to various MIME types so that files open in sensible default applications
+  * Set default timezone to 'Europe/London'
+  * Linux kernel 4.14.79
+  * Raspberry Pi firmware 12e0bf86e08d6067372bc0a45d7e8a10d3113210
+2018-10-09:
+  * Raspberry Pi 3A+ support
+  * In startup wizard, assign keyboard to country as per Debian installer recommendations
+  * In startup wizard, add option to use US keyboard in preference to country-specific option
+  * In startup wizard, show IP address on first page
+  * In startup wizard, check for existing wifi network connection and show it if there is one
+  * In startup wizard, install language support packages for LibreOffice and other applications
+  * In startup wizard, improve operation with keyboard only and no mouse
+  * Password change in Raspberry Pi Configuration and startup wizard now works properly if passwords contain shell characters
+  * Battery indicator plugin modified to cope with Pi-top hardware monitor crashing
+  * Networking plugin hides wifi password characters by default
+  * In Scratch 2 GPIO plugin, set pin from dropdown list rather than free text
+  * In Scratch 2 SenseHAT plugin, swap x and y axis values for LED array
+  * Include latest Adobe Flash player (31.0.0.108)
+  * Include latest RealVNC Server (6.3.1)
+  * Include libav-tools
+  * Include ssh-import-id
+  * Removed Mathematica
+  * Merge in latest third-party code for Bluetooth ALSA interface
+  * Add ability to prevent software update changing configuration files, by creating ~/.config/.lock file
+  * Various other small bug fixes, tweaks and changes to text
+  * Make dhcpcd work with 3G devices
+  * Add hw acceleration to ffmpeg
+  * Improved WiFi-BT coexistence parameters
+  * Run fake-hwclock before systemd-fsck-root
+  * Raspberry Pi PoE HAT support
+  * Linux kernel 4.14.71
+  * Raspberry Pi firmware 5b49caa17e91d0e64024380119ad739bb201c674
+2018-06-27:
+  * New first-boot configuration wizard added
+  * Recommended Software installer added
+  * Bluej, Greenfoot, NodeRED, Claws Mail, VNC Viewer removed from image - can now be installed from Recommended Applications
+  * Qpdfview PDF viewer installed instead of Xpdf
+  * Version 65.0 of Chromium browser included, with latest Flash player
+  * Volume up / down keys now change by 5% increments and affect currently-selected output device rather than internal device only
+  * Network plugin now remembers previously-entered WiFi network passwords when prompting for reconnection
+  * Serial port and serial console can now be switched separately in Raspberry Pi Configuration
+  * Lxkeymap keyboard language setting application removed - replaced with dialog within Raspberry Pi Configuration
+  * Wifi country and keyboard language setting dialogs in Raspberry Pi Configuration now callable from other applications
+  * New version of Piboto font included to render with correct weight under some rogue applications
+  * Reconnection to Bluetooth audio devices on reboot improved
+  * Disable click-to-rename behaviour in file manager if single-click selection enabled
+  * Appearance Settings dialog makes config changes to some Qt files to match selected theme
+  * MIME file type associations improved
+  * Multiple desktop management options removed from mouse middle-click menu
+  * Menu shortcuts to Raspberry Pi website amended
+  * Python 2 IDLE menu link removed
+  * Sample Magpi PDF installed in /home/pi/MagPi
+  * Various minor tweaks, bug fixes and appearance changes
+  * Bluetooth updates
+    - Firmware with Bluetooth 4.2 features
+    - SCO profile suppot added via bthelper.service
+  * Linux kernel 4.14.50+
+  * Raspberry Pi firmware 748fb17992426bb29d99224b93cb962fefbdc833
+2018-04-18:
+  * Fixed race between wifi-country.service and raspberrypi-net-mods.service
+  * Linux kernel 4.14.34+
+  * Raspberry Pi firmware 5db8e4e1c63178e200d6fbea23ed4a9bf4656658
+2018-03-13:
+  * Raspberry Pi 3 B+ support
+  * WiFi is disabled until wireless regulatory domain is set (Pi 3 B+ only)
+    - The domain can be done through 'Raspberry Pi Configuration' (rc_gui),
+      'raspi-config' or by setting 'country=' to an appropriate ISO 3166
+      alpha2 country code in /etc/wpa_supplicant/wpa_supplicant.conf.
+  * Default wireless regulatory domain is now unset
+  * Added support to desktop for different screen sizes and resolutions,
+    including multiple preset options in Appearance Settings and pixel doubling
+    option in Raspberry Pi Configuration
+  * Version 2.1.16 of Thonny included
+  * Version 29.0.0.113 of Adobe PepperFlash player included
+  * Version 1.2.post1 of Pygame Zero included
+  * Bluetooth plugin now supports connection to Bluetooth LE HID devices
+  * Network plugin now indicates 5G-compatible APs
+  * Latest changes to Bluez ALSA service merged
+    - service now started on CLI boot as well as GUI boot
+  * Latest changes to dhcpcd networking plugin merged
+  * Improved support for running on pi-top devices
+  * Small design changes to PiX theme and icons
+  * Bug fix - hide spurious window resize handles
+  * Bug fix - Scratch 2 remote GPIO state block now works correctly
+  * Updated WiFi Firmware
+    - brcmfmac43455-sdio 7.45.154
+    - brcmfmac43430-sdio 7.45.98.38
+  * New packages:
+    - policykit-1
+    - obconf
+    - python-buttonshim python3-buttonshim
+    - python-unicornhathd  python3-unicornhathd
+    - python-pantilthat python3-pantilthat
+  * Linux kernel 4.9.80+
+  * Raspberry Pi firmware 3347884c7df574bbabeff6dca63caf686e629699
+2017-11-29:
+  * Added battery monitor plugin for taskbar - works on x86 images or first-generation Pi-Top
+  * Added cutdown mode to PCManFM file manager to reduce complexity
+  * Added ability to rename files in PCManFM by clicking name when selected
+  * Bug fix in Bluetooth ALSA module to reduce truncation of audio at end of playback
+  * Various small tweaks, bug fixes and theme modifications
+  * New kernel and firmware
+2017-09-07:
+  * Disable predictable network interface names for Ethernet devices
+  * Bug fix for keyboard settings dialog in Raspberry Pi Configuration
+  * Bug fix for crash on some videos and animations in Chromium
+  * Bug fix for taskbar crash when running RealVNC server
+  * Bug fix for reloading projects with extensions in Scratch 2
+  * Bug fix for MAC address problem in Bluetooth
+  * Simple mode and new icons in Thonny
+  * New Japanese translations in Raspberry Pi Configuration
+  * Install fonts-droid-fallback for international fonts
+2017-08-16:
+  * Based on Raspbian Stretch (Debian version 9)
+  * Version 60 of Chromium browser included
+  * Version 3.0.1 of Sonic Pi included
+  * Version 6.1.1 of RealVNC included
+  * Version 0.17.4 of NodeRED included
+  * Bluetooth audio routed via ALSA rather than Pulseaudio
+  * SenseHAT extension added to Scratch 2
+  * Various desktop applications modified to prompt for sudo password if needed
+  * lxinput control options for mouse speed simplified
+  * lxpanel plugins moved into separate packages
+  * Wireless firmware for Pi 3 and Pi 0W modified to address Broadpwn exploit
+  * Latest kernel and firmware
+  * Various small tweaks, bug fixes and theme modifications
+2017-07-05:
+  * New kernel and firmware
+  * Filesystem created without the metadata_csum feature
+2017-06-21:
+  * Scratch 2 application included
+  * Thonny Python IDE included
+  * New icons with thinner outlines
+  * Volume control more linear in behaviour
+  * Updated Flash player
+  * Updated RealVNC server and viewer
+  * Various tweaks and bugfixes
+  * New kernel and firmware
+2017-04-10:
+  * Wolfram Mathematica updated to version 11.0.1
+  * Adobe Flash Player updated to version 25.0.0.127
+  * Use PARTUUID to support USB boot
+2017-03-02:
   * Updated kernel and firmware (final Pi Zero W support)
+  * Wolfram Mathematica updated to version 11
+  * NOOBS installs now checks for presence of 'ssh' file on the NOOBS partition.
 2017-02-16:
   * Chromium browser updated to version 56
   * Adobe Flash Player updated to version 24.0.0.221

export-noobs/prerun.sh

@@ -2,14 +2,14 @@
 
 IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
 NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"
-unmount_image ${IMG_FILE}
+unmount_image "${IMG_FILE}"
 
-mkdir -p ${STAGE_WORK_DIR}
-cp ${WORK_DIR}/export-image/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img ${STAGE_WORK_DIR}/
+mkdir -p "${STAGE_WORK_DIR}"
+cp "${WORK_DIR}/export-image/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img" "${STAGE_WORK_DIR}/"
 
-rm -rf ${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}
+rm -rf "${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"
 
-PARTED_OUT=$(parted -s ${IMG_FILE} unit b print)
+PARTED_OUT=$(parted -s "${IMG_FILE}" unit b print)
 BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
 | cut -d" " -f 2 | tr -d B)
 BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
@@ -20,19 +20,21 @@ ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
 ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
 | cut -d" " -f 4 | tr -d B)
 
-BOOT_DEV=$(losetup --show -f -o ${BOOT_OFFSET} --sizelimit ${BOOT_LENGTH} ${IMG_FILE})
-ROOT_DEV=$(losetup --show -f -o ${ROOT_OFFSET} --sizelimit ${ROOT_LENGTH} ${IMG_FILE})
+BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGTH}" "${IMG_FILE}")
+ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGTH}" "${IMG_FILE}")
 echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH"
 echo "/:     offset $ROOT_OFFSET, length $ROOT_LENGTH"
 
-mkdir -p ${STAGE_WORK_DIR}/rootfs
-mkdir -p ${NOOBS_DIR}
+mkdir -p "${STAGE_WORK_DIR}/rootfs"
+mkdir -p "${NOOBS_DIR}"
 
-mount $ROOT_DEV ${STAGE_WORK_DIR}/rootfs
-mount $BOOT_DEV ${STAGE_WORK_DIR}/rootfs/boot
+mount "$ROOT_DEV" "${STAGE_WORK_DIR}/rootfs"
+mount "$BOOT_DEV" "${STAGE_WORK_DIR}/rootfs/boot"
 
-bsdtar --format gnutar --use-compress-program pxz -C ${STAGE_WORK_DIR}/rootfs/boot -cpf ${NOOBS_DIR}/boot.tar.xz .
-umount ${STAGE_WORK_DIR}/rootfs/boot
-bsdtar --format gnutar --use-compress-program pxz -C ${STAGE_WORK_DIR}/rootfs --one-file-system -cpf ${NOOBS_DIR}/root.tar.xz .
+ln -sv "/lib/systemd/system/apply_noobs_os_config.service" "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service"
 
-unmount_image ${IMG_FILE}
+bsdtar --numeric-owner --format gnutar --use-compress-program pxz -C "${STAGE_WORK_DIR}/rootfs/boot" -cpf "${NOOBS_DIR}/boot.tar.xz" .
+umount "${STAGE_WORK_DIR}/rootfs/boot"
+bsdtar --numeric-owner --format gnutar --use-compress-program pxz -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf "${NOOBS_DIR}/root.tar.xz" .
+
+unmount_image "${IMG_FILE}"

scripts/common

@@ -1,5 +1,5 @@
 log (){
-	date +"[%T] $@" | tee -a "${LOG_FILE}"
+	date +"[%T] $*" | tee -a "${LOG_FILE}"
 }
 export -f log
 
@@ -15,10 +15,13 @@ bootstrap(){
 		local BOOTSTRAP_CMD=debootstrap
 	fi
 
-	capsh --drop=cap_setfcap -- -c "${BOOTSTRAP_CMD} --components=main,contrib,non-free \
+	capsh --drop=cap_setfcap -- "${BOOTSTRAP_CMD}" --components=main,contrib,non-free \
 		--arch armhf \
-		--no-check-gpg \
-		$1 $2 $3"
+		--keyring "${STAGE_DIR}/files/raspberrypi.gpg" \
+		"$1" "$2" "$3" || true
+	if [ -d "$2/debootstrap" ]; then
+		rmdir "$2/debootstrap"
+	fi
 }
 export -f bootstrap
 
@@ -28,7 +31,7 @@ copy_previous(){
 		false
 	fi
 	mkdir -p "${ROOTFS_DIR}"
-	rsync -aHAXx "${PREV_ROOTFS_DIR}/" "${ROOTFS_DIR}/"
+	rsync -aHAXx --exclude var/cache/apt/archives "${PREV_ROOTFS_DIR}/" "${ROOTFS_DIR}/"
 }
 export -f copy_previous
 
@@ -85,7 +88,7 @@ on_chroot() {
 		mount --bind /sys "${ROOTFS_DIR}/sys"
 	fi
 
-	capsh --drop=cap_setfcap "--chroot=${ROOTFS_DIR}/" -- "$@"
+	capsh --drop=cap_setfcap "--chroot=${ROOTFS_DIR}/" -- -e "$@"
 }
 export -f on_chroot
 

scripts/dependencies_check

@@ -1,5 +1,5 @@
 # dependencies_check
-# $@	Dependnecy files to check
+# $@	Dependency files to check
 #
 # Each dependency is in the form of a tool to test for, optionally followed by
 # a : and the name of a package if the package on a Debian-ish system is not
@@ -10,18 +10,18 @@ dependencies_check()
 
 	for depfile in "$@"; do
 		if [[ -e "$depfile" ]]; then
-			deps="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < ${BASE_DIR}/depends)"
+			deps="$(sed -f "${SCRIPT_DIR}/remove-comments.sed" < "${BASE_DIR}/depends")"
 
 		fi
 		for dep in $deps; do
-			if ! hash ${dep%:*} 2>/dev/null; then
+			if ! hash "${dep%:*}" 2>/dev/null; then
 				missing="${missing:+$missing }${dep#*:}"
 			fi
 		done
 	done
 
 	if [[ "$missing" ]]; then
-		echo "Reqired dependencies not installed"
+		echo "Required dependencies not installed"
 		echo
 		echo "This can be resolved on Debian/Raspbian systems by installing:"
 		echo "$missing"

stage0/00-configure-apt/00-run.sh

@@ -0,0 +1,17 @@
+#!/bin/bash -e
+
+install -m 644 files/sources.list "${ROOTFS_DIR}/etc/apt/"
+install -m 644 files/raspi.list "${ROOTFS_DIR}/etc/apt/sources.list.d/"
+
+if [ -n "$APT_PROXY" ]; then
+	install -m 644 files/51cache "${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache"
+	sed "${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache" -i -e "s|APT_PROXY|${APT_PROXY}|"
+else
+	rm -f "${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache"
+fi
+
+on_chroot apt-key add - < files/raspberrypi.gpg.key
+on_chroot << EOF
+apt-get update
+apt-get dist-upgrade -y
+EOF

stage0/00-configure-apt/files/raspi.list

@@ -0,0 +1,3 @@
+deb http://archive.raspberrypi.org/debian/ stretch main ui staging
+# Uncomment line below then 'apt-get update' to enable 'apt-get source'
+#deb-src http://archive.raspberrypi.org/debian/ stretch main ui

stage0/00-configure-apt/files/sources.list

@@ -0,0 +1,3 @@
+deb http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi
+# Uncomment line below then 'apt-get update' to enable 'apt-get source'
+#deb-src http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi

stage0/01-configure-apt/00-run.sh

@@ -1,17 +0,0 @@
-#!/bin/bash -e
-
-install -m 644 files/sources.list ${ROOTFS_DIR}/etc/apt/
-install -m 644 files/raspi.list ${ROOTFS_DIR}/etc/apt/sources.list.d/
-
-if [ -n "$APT_PROXY" ]; then
-	install -m 644 files/51cache ${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache
-	sed ${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache -i -e "s|APT_PROXY|${APT_PROXY}|"
-else
-	rm -f ${ROOTFS_DIR}/etc/apt/apt.conf.d/51cache
-fi
-
-on_chroot apt-key add - < files/raspberrypi.gpg.key
-on_chroot << EOF
-apt-get update
-apt-get dist-upgrade -y
-EOF

stage0/01-configure-apt/files/raspi.list

@@ -1,3 +0,0 @@
-deb http://archive.raspberrypi.org/debian/ jessie main ui staging
-# Uncomment line below then 'apt-get update' to enable 'apt-get source'
-#deb-src http://archive.raspberrypi.org/debian/ jessie main ui

stage0/01-configure-apt/files/sources.list

@@ -1,3 +0,0 @@
-deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
-# Uncomment line below then 'apt-get update' to enable 'apt-get source'
-#deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi

stage0/01-locale/00-packages

@@ -0,0 +1 @@
+locales

stage0/02-firmware/01-packages

@@ -1 +1,2 @@
 raspberrypi-bootloader
+raspberrypi-kernel

stage0/prerun.sh

@@ -1,4 +1,5 @@
 #!/bin/bash -e
-if [ ! -d ${ROOTFS_DIR} ]; then
-	bootstrap jessie ${ROOTFS_DIR} http://mirrordirector.raspbian.org/raspbian/
+
+if [ ! -d "${ROOTFS_DIR}" ]; then
+	bootstrap stretch "${ROOTFS_DIR}" http://raspbian.raspberrypi.org/raspbian/
 fi

stage1/00-boot-files/00-run.sh

@@ -1,4 +1,4 @@
 #!/bin/bash -e
 
-install -m 644 files/cmdline.txt ${ROOTFS_DIR}/boot/
-install -m 644 files/config.txt ${ROOTFS_DIR}/boot/
+install -m 644 files/cmdline.txt "${ROOTFS_DIR}/boot/"
+install -m 644 files/config.txt "${ROOTFS_DIR}/boot/"

stage1/00-boot-files/files/cmdline.txt

@@ -1 +1 @@
-dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
+dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

stage1/00-boot-files/files/config.txt

@@ -1,5 +1,5 @@
 # For more options and information see
-# http://www.raspberrypi.org/documentation/configuration/config-txt.md
+# http://rpf.io/configtxt
 # Some settings may impact device functionality. See link above for details
 
 # uncomment if you get no picture on HDMI for a default "safe" mode

stage1/01-sys-tweaks/00-run.sh

@@ -1,9 +1,8 @@
 #!/bin/bash -e
 
-install -d ${ROOTFS_DIR}/etc/systemd/system/[email protected]
-install -m 644 files/noclear.conf ${ROOTFS_DIR}/etc/systemd/system/[email protected]/noclear.conf
-install -m 744 files/policy-rc.d ${ROOTFS_DIR}/usr/sbin/policy-rc.d #TODO: Necessary in systemd?
-install -v -m 644 files/fstab ${ROOTFS_DIR}/etc/fstab
+install -d "${ROOTFS_DIR}/etc/systemd/system/[email protected]"
+install -m 644 files/noclear.conf "${ROOTFS_DIR}/etc/systemd/system/[email protected]/noclear.conf"
+install -v -m 644 files/fstab "${ROOTFS_DIR}/etc/fstab"
 
 on_chroot << EOF
 if ! id -u pi >/dev/null 2>&1; then

stage1/01-sys-tweaks/files/fstab

@@ -1,3 +1,3 @@
 proc            /proc           proc    defaults          0       0
-/dev/mmcblk0p1  /boot           vfat    defaults          0       2
-/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
+BOOTDEV  /boot           vfat    defaults          0       2
+ROOTDEV  /               ext4    defaults,noatime  0       1

stage1/01-sys-tweaks/files/policy-rc.d

@@ -1,2 +0,0 @@
-#!/bin/sh
-exit 101

stage1/02-net-tweaks/00-packages

@@ -0,0 +1 @@
+netbase

stage1/02-net-tweaks/00-patches/02-persistant-net.diff

@@ -1,13 +0,0 @@
-Index: jessie-stage1/rootfs/lib/udev/rules.d/75-persistent-net-generator.rules
-===================================================================
---- jessie-stage1.orig/rootfs/lib/udev/rules.d/75-persistent-net-generator.rules
-+++ jessie-stage1/rootfs/lib/udev/rules.d/75-persistent-net-generator.rules
-@@ -22,7 +22,7 @@ IMPORT{cmdline}="net.ifnames"
- ENV{net.ifnames}=="1",			GOTO="persistent_net_generator_end"
- 
- # device name whitelist
--KERNEL!="eth*|ath*|wlan*[0-9]|msh*|ra*|sta*|ctc*|lcs*|hsi*", \
-+KERNEL!="ath*|msh*|ra*|sta*|ctc*|lcs*|hsi*", \
- 					GOTO="persistent_net_generator_end"
- 
- # ignore Xen virtual interfaces

stage1/02-net-tweaks/00-patches/series

@@ -1,2 +1 @@
 01-hosts.diff
-02-persistant-net.diff

stage1/02-net-tweaks/00-run.sh

@@ -1,9 +1,6 @@
 #!/bin/bash -e
 
-install -m 644 files/ipv6.conf ${ROOTFS_DIR}/etc/modprobe.d/ipv6.conf
-install -m 644 files/interfaces ${ROOTFS_DIR}/etc/network/interfaces
-install -m 644 files/hostname ${ROOTFS_DIR}/etc/hostname
+install -m 644 files/ipv6.conf "${ROOTFS_DIR}/etc/modprobe.d/ipv6.conf"
+install -m 644 files/hostname "${ROOTFS_DIR}/etc/hostname"
 
-on_chroot << EOF
-dpkg-divert --add --local /lib/udev/rules.d/75-persistent-net-generator.rules
-EOF
+ln -sf /dev/null "${ROOTFS_DIR}/etc/systemd/network/99-default.link"

stage1/prerun.sh

@@ -1,4 +1,5 @@
 #!/bin/bash -e
-if [ ! -d ${ROOTFS_DIR} ]; then
+
+if [ ! -d "${ROOTFS_DIR}" ]; then
 	copy_previous
 fi

stage2/00-copies-and-fills/00-run.sh

@@ -1,3 +0,0 @@
-#!/bin/bash -e
-
-touch ${ROOTFS_DIR}/spindle_install

stage2/00-copies-and-fills/02-run.sh

@@ -1,3 +0,0 @@
-#!/bin/bash -e
-
-rm -f ${ROOTFS_DIR}/spindle_install

stage2/01-sys-tweaks/00-packages

@@ -1,4 +1,4 @@
-ssh locales less fbset sudo psmisc strace module-init-tools ed ncdu crda
+ssh less fbset sudo psmisc strace ed ncdu crda
 console-setup keyboard-configuration debconf-utils parted unzip
 build-essential manpages-dev python bash-completion gdb pkg-config
 python-rpi.gpio v4l-utils
@@ -6,7 +6,7 @@ avahi-daemon
 lua5.1
 luajit
 hardlink ca-certificates curl
-fake-hwclock ntp nfs-common usbutils
+fake-hwclock nfs-common usbutils
 libraspberrypi-dev libraspberrypi-doc libfreetype6-dev
 dosfstools
 dphys-swapfile
@@ -15,3 +15,11 @@ pi-bluetooth
 apt-listchanges
 usb-modeswitch
 apt-transport-https
+libpam-chksshpwd
+rpi-update
+libmtp-runtime
+rsync
+htop
+man-db
+policykit-1
+ssh-import-id

stage2/01-sys-tweaks/00-patches/06-rc_local.diff

@@ -1,15 +0,0 @@
-Index: jessie-stage2/rootfs/etc/rc.local
-===================================================================
---- jessie-stage2.orig/rootfs/etc/rc.local
-+++ jessie-stage2/rootfs/etc/rc.local
-@@ -11,4 +11,10 @@
- #
- # By default this script does nothing.
- 
-+# Print the IP address
-+_IP=$(hostname -I) || true
-+if [ "$_IP" ]; then
-+  printf "My IP address is %s\n" "$_IP"
-+fi
-+
- exit 0

stage2/01-sys-tweaks/00-patches/07-resize-init.diff

@@ -1,5 +1,5 @@
 --- a/rootfs/boot/cmdline.txt
 +++ b/rootfs/boot/cmdline.txt
 @@ -1 +1 @@
--dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
-+dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh
+-dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
++dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh

stage2/01-sys-tweaks/00-patches/series

@@ -2,5 +2,4 @@
 02-swap.diff
 04-inputrc.diff
 05-path.diff
-06-rc_local.diff
 07-resize-init.diff

stage2/01-sys-tweaks/01-run.sh

@@ -1,15 +1,15 @@
 #!/bin/bash -e
 
-install -m 644 files/regenerate_ssh_host_keys.service	${ROOTFS_DIR}/lib/systemd/system/
-install -m 755 files/apply_noobs_os_config		${ROOTFS_DIR}/etc/init.d/
-install -m 755 files/resize2fs_once			${ROOTFS_DIR}/etc/init.d/
+install -m 755 files/resize2fs_once	"${ROOTFS_DIR}/etc/init.d/"
 
-install -d						${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d
-install -m 644 files/ttyoutput.conf			${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d/
+install -d				"${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d"
+install -m 644 files/ttyoutput.conf	"${ROOTFS_DIR}/etc/systemd/system/rc-local.service.d/"
 
-install -m 644 files/50raspi				${ROOTFS_DIR}/etc/apt/apt.conf.d/
+install -m 644 files/50raspi		"${ROOTFS_DIR}/etc/apt/apt.conf.d/"
 
-install -m 644 files/console-setup   			${ROOTFS_DIR}/etc/default/
+install -m 644 files/console-setup   	"${ROOTFS_DIR}/etc/default/"
+
+install -m 755 files/rc.local		"${ROOTFS_DIR}/etc/"
 
 on_chroot << EOF
 systemctl disable hwclock.sh
@@ -17,13 +17,24 @@ systemctl disable nfs-common
 systemctl disable rpcbind
 systemctl disable ssh
 systemctl enable regenerate_ssh_host_keys
-systemctl enable apply_noobs_os_config
+EOF
+
+if [ "${USE_QEMU}" = "1" ]; then
+	echo "enter QEMU mode"
+	install -m 644 files/90-qemu.rules "${ROOTFS_DIR}/etc/udev/rules.d/"
+	on_chroot << EOF
+systemctl disable resize2fs_once
+EOF
+	echo "leaving QEMU mode"
+else
+	on_chroot << EOF
 systemctl enable resize2fs_once
 EOF
+fi
 
 on_chroot << \EOF
 for GRP in input spi i2c gpio; do
-	groupadd -f -r $GRP
+	groupadd -f -r "$GRP"
 done
 for GRP in adm dialout cdrom audio users sudo video games plugdev input gpio spi i2c netdev; do
   adduser pi $GRP
@@ -38,4 +49,4 @@ on_chroot << EOF
 usermod --pass='*' root
 EOF
 
-rm -f ${ROOTFS_DIR}/etc/ssh/ssh_host_*_key*
+rm -f "${ROOTFS_DIR}/etc/ssh/"ssh_host_*_key*

stage2/01-sys-tweaks/files/90-qemu.rules

@@ -0,0 +1,3 @@
+KERNEL=="sda", SYMLINK+="mmcblk0"
+KERNEL=="sda?", SYMLINK+="mmcblk0p%n"
+KERNEL=="sda2", SYMLINK+="root"

stage2/01-sys-tweaks/files/apply_noobs_os_config

@@ -1,31 +0,0 @@
-#!/bin/sh
-### BEGIN INIT INFO
-# Provides:          apply_noobs_os_config
-# Required-Start:
-# Required-Stop:
-# Default-Start: 2
-# Default-Stop:
-# Short-Description: Apply config from /boot/os_config.json
-# Description:
-### END INIT INFO
-
-. /lib/lsb/init-functions
-
-set -e
-
-case "$1" in
-  start)
-    log_daemon_msg "Applying config from /boot/os_config.json (if it exists)"
-    if raspi-config --apply-os-config; then
-      update-rc.d apply_noobs_os_config remove
-      rm /etc/init.d/apply_noobs_os_config
-      log_end_msg 0
-    else
-      log_end_msg 1
-    fi
-    ;;
-  *)
-    echo "Usage: $0 start" >&2
-    exit 3
-    ;;
-esac

stage2/01-sys-tweaks/files/rc.local

@@ -0,0 +1,20 @@
+#!/bin/sh -e
+#
+# rc.local
+#
+# This script is executed at the end of each multiuser runlevel.
+# Make sure that the script will "exit 0" on success or any other
+# value on error.
+#
+# In order to enable or disable this script just change the execution
+# bits.
+#
+# By default this script does nothing.
+
+# Print the IP address
+_IP=$(hostname -I) || true
+if [ "$_IP" ]; then
+  printf "My IP address is %s\n" "$_IP"
+fi
+
+exit 0

stage2/01-sys-tweaks/files/regenerate_ssh_host_keys.service

@@ -1,11 +0,0 @@
-[Unit]
-Description=Regenerate SSH host keys
-
-[Service]
-Type=oneshot
-ExecStartPre=/bin/sh -c "if [ -e /dev/hwrng ]; then dd if=/dev/hwrng of=/dev/urandom count=1 bs=4096; fi"
-ExecStart=/usr/bin/ssh-keygen -A
-ExecStartPost=/bin/rm /lib/systemd/system/regenerate_ssh_host_keys.service ; /usr/sbin/update-rc.d regenerate_ssh_host_keys remove
-
-[Install]
-WantedBy=multi-user.target

stage2/01-sys-tweaks/files/resize2fs_once

@@ -12,7 +12,7 @@
 case "$1" in
   start)
     log_daemon_msg "Starting resize2fs_once"
-    ROOT_DEV=`grep -Eo 'root=[[:graph:]]+' /proc/cmdline | cut -d '=' -f 2-` &&
+    ROOT_DEV=$(findmnt / -o source -n) &&
     resize2fs $ROOT_DEV &&
     update-rc.d resize2fs_once remove &&
     rm /etc/init.d/resize2fs_once &&

stage2/02-net-tweaks/00-packages

@@ -1,3 +1,4 @@
-wpasupplicant wireless-tools firmware-atheros firmware-brcm80211 firmware-libertas firmware-ralink firmware-realtek
+wpasupplicant wireless-tools firmware-atheros firmware-brcm80211 firmware-libertas firmware-misc-nonfree firmware-realtek
 raspberrypi-net-mods
 dhcpcd5
+net-tools

stage2/02-net-tweaks/01-run.sh

@@ -1,8 +1,8 @@
 #!/bin/bash -e
 
-install -v -d						${ROOTFS_DIR}/etc/systemd/system/dhcpcd.service.d
-install -v -m 644 files/wait.conf			${ROOTFS_DIR}/etc/systemd/system/dhcpcd.service.d/
+install -v -d					"${ROOTFS_DIR}/etc/systemd/system/dhcpcd.service.d"
+install -v -m 644 files/wait.conf		"${ROOTFS_DIR}/etc/systemd/system/dhcpcd.service.d/"
 
-install -v -d                                           ${ROOTFS_DIR}/etc/wpa_supplicant
-install -v -m 600 files/wpa_supplicant.conf             ${ROOTFS_DIR}/etc/wpa_supplicant/
+install -v -d					"${ROOTFS_DIR}/etc/wpa_supplicant"
+install -v -m 600 files/wpa_supplicant.conf	"${ROOTFS_DIR}/etc/wpa_supplicant/"
 

stage2/02-net-tweaks/files/wait.conf

@@ -1,3 +1,3 @@
 [Service]
 ExecStart=
-ExecStart=/sbin/dhcpcd -q -w
+ExecStart=/usr/lib/dhcpcd5/dhcpcd -q -w

stage2/02-net-tweaks/files/wpa_supplicant.conf

@@ -1,3 +1,2 @@
-country=GB
 ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
 update_config=1

stage2/03-accept-mathematica-eula/00-debconf

@@ -0,0 +1,2 @@
+# Do you accept the Wolfram - Raspberry Pi® Bundle License Agreement?
+wolfram-engine  shared/accepted-wolfram-eula    boolean true

stage2/03-cleanup/00-run.sh

@@ -1,5 +0,0 @@
-#!/bin/bash -e
-
-on_chroot << EOF
-apt-get clean
-EOF

stage2/03-set-timezone/02-run.sh

@@ -0,0 +1,8 @@
+#!/bin/bash -e
+
+echo "Europe/London" > "${ROOTFS_DIR}/etc/timezone"
+rm "${ROOTFS_DIR}/etc/localtime"
+
+on_chroot << EOF
+dpkg-reconfigure -f noninteractive tzdata
+EOF

stage2/EXPORT_IMAGE

@@ -1 +1,4 @@
--lite
+IMG_SUFFIX="-lite"
+if [ "${USE_QEMU}" = "1" ]; then
+	export IMG_SUFFIX="${IMG_SUFFIX}-qemu"
+fi

stage2/EXPORT_NOOBS

@@ -0,0 +1,2 @@
+NOOBS_NAME="Raspbian Lite"
+NOOBS_DESCRIPTION="A port of Debian with no desktop environment"

stage2/prerun.sh

@@ -1,4 +1,5 @@
 #!/bin/bash -e
-if [ ! -d ${ROOTFS_DIR} ]; then
+
+if [ ! -d "${ROOTFS_DIR}" ]; then
 	copy_previous
 fi

stage3/00-install-packages/00-packages

@@ -1,10 +1,14 @@
 gstreamer1.0-x gstreamer1.0-omx gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-alsa gstreamer1.0-libav
-xpdf gtk2-engines alsa-utils
+qpdfview gtk2-engines alsa-utils
 desktop-base
-git-core rpi-update
+git
 omxplayer
 raspberrypi-artwork
 policykit-1
 gvfs
 rfkill
 chromium-browser rpi-chromium-mods
+gldriver-test
+fonts-droid-fallback
+fonts-liberation2
+obconf

stage3/00-install-packages/01-run.sh

@@ -6,4 +6,3 @@ update-alternatives --install /usr/bin/x-www-browser \
 update-alternatives --install /usr/bin/gnome-www-browser \
   gnome-www-browser /usr/bin/chromium-browser 86
 EOF
-

stage3/01-tweaks/00-run.sh

@@ -1,3 +1,3 @@
 #!/bin/bash -e
 
-rm -f ${ROOTFS_DIR}/etc/systemd/system/dhcpcd.service.d/wait.conf
+rm -f "${ROOTFS_DIR}/etc/systemd/system/dhcpcd.service.d/wait.conf"

stage3/prerun.sh

@@ -1,4 +1,5 @@
 #!/bin/bash -e
-if [ ! -d ${ROOTFS_DIR} ]; then
+
+if [ ! -d "${ROOTFS_DIR}" ]; then
 	copy_previous
 fi

stage4/00-install-packages/00-debconf

@@ -1,4 +1,2 @@
 # Enable realtime process priority?
 jackd2  jackd/tweak_rt_limits   boolean true
-# Do you accept the Wolfram - Raspberry Pi® Bundle License Agreement?
-wolfram-engine  shared/accepted-wolfram-eula    boolean true

stage4/00-install-packages/00-packages

@@ -1,36 +1,27 @@
 java-common oracle-java8-jdk
-libreoffice-sdbc-hsqldb
-sonic-pi
 python idle python3-pygame python-pygame python-tk
 python3 idle3 python3-tk
 python3-pgzero
 python-serial python3-serial
 python-picamera python3-picamera
-debian-reference-en dillo x2x
-wolfram-engine
+debian-reference-en dillo
 raspberrypi-net-mods raspberrypi-ui-mods
-smartsim penguinspuzzle
 python-pip python3-pip
 python3-numpy
 pypy
-python3-pifacecommon python3-pifacedigitalio python3-pifacedigital-scratch-handler python-pifacecommon python-pifacedigitalio
-minecraft-pi python-minecraftpi
 alacarte rc-gui sense-hat
-claws-mail
 tree
-scratch nuscratch
-greenfoot bluej
-nodered
 libgl1-mesa-dri libgles1-mesa libgles2-mesa xcompmgr
 geany
 piclone
-pulseaudio-module-bluetooth
 wiringpi pigpio python-pigpio python3-pigpio raspi-gpio python-gpiozero python3-gpiozero python3-rpi.gpio
 python-spidev python3-spidev
 python-twython python3-twython
 python-smbus python3-smbus
 python-flask python3-flask
 python-picraft python3-picraft
-python3-codebug-tether python3-codebug-i2c-tether
-python-sense-emu python3-sense-emu sense-emu-tools python-sense-emu-doc
 pprompt
+piwiz
+rp-prefapps
+libav-tools
+vlc

stage4/00-install-packages/00-packages-nr

@@ -1,4 +1,3 @@
-libreoffice libreoffice-gtk
 timidity
 pi-package
-realvnc-vnc-server realvnc-vnc-viewer
+realvnc-vnc-server

stage4/00-install-packages/01-packages

@@ -0,0 +1,22 @@
+python-automationhat python3-automationhat
+python-blinkt python3-blinkt
+python-cap1xxx python3-cap1xxx
+python-drumhat python3-drumhat
+python-envirophat python3-envirophat
+python-explorerhat python3-explorerhat
+python-fourletterphat python3-fourletterphat
+python-microdotphat python3-microdotphat
+python-mote python3-mote
+python-motephat python3-motephat
+python-phatbeat python3-phatbeat
+python-pianohat python3-pianohat
+python-piglow python3-piglow
+python-rainbowhat python3-rainbowhat
+python-scrollphat python3-scrollphat
+python-scrollphathd python3-scrollphathd
+python-sn3218 python3-sn3218
+python-skywriter python3-skywriter
+python-touchphat python3-touchphat
+python-buttonshim python3-buttonshim
+python-unicornhathd  python3-unicornhathd
+python-pantilthat python3-pantilthat

stage4/00-install-packages/01-run.sh

@@ -1,3 +0,0 @@
-#!/bin/bash -e
-
-rm -f "${ROOTFS_DIR}/root/.vnc/private.key"

stage4/00-install-packages/02-packages

@@ -0,0 +1,4 @@
+hunspell-en-gb
+hyphen-en-gb
+wamerican
+wbritish

stage4/01-console-autologin/00-run.sh

@@ -1,3 +1,4 @@
 #!/bin/bash -e
 
-ln -sf /etc/systemd/system/[email protected] ${ROOTFS_DIR}/etc/systemd/system/getty.target.wants/[email protected]
+ln -sf /etc/systemd/system/[email protected] \
+	"${ROOTFS_DIR}/etc/systemd/system/getty.target.wants/[email protected]"

stage4/02-extras/00-run.sh

@@ -1,31 +1,5 @@
 #!/bin/bash -e
 
-HASH=`wget https://api.github.com/repos/KenT2/python-games/git/refs/heads/master -qO -| grep \"sha\" | cut -f 2 -d ':' | cut -f 2 -d \"`
-
-if [ -f files/python_games.hash ]; then
-	HASH_LOCAL=`cat files/python_games.hash`
-fi
-
-if [ ! -e files/python_games.tar.gz ] || [ "$HASH" != "$HASH_LOCAL"  ]; then
-	wget "https://github.com/KenT2/python-games/tarball/master" -O files/python_games.tar.gz
-	echo $HASH > files/python_games.hash
-fi
-
-ln -sf pip3 ${ROOTFS_DIR}/usr/bin/pip-3.2
-
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/python_games
-tar xvf files/python_games.tar.gz -C ${ROOTFS_DIR}/home/pi/python_games --strip-components=1
-chown 1000:1000 ${ROOTFS_DIR}/home/pi/python_games -Rv
-chmod +x ${ROOTFS_DIR}/home/pi/python_games/launcher.sh
-
-install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/Documents"
-install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/Documents/BlueJ Projects"
-install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/Documents/Greenfoot Projects"
-install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/Documents/Scratch Projects"
-rsync -a --chown=1000:1000 ${ROOTFS_DIR}/usr/share/doc/BlueJ/ "${ROOTFS_DIR}/home/pi/Documents/BlueJ Projects"
-rsync -a --chown=1000:1000 ${ROOTFS_DIR}/usr/share/doc/Greenfoot/ "${ROOTFS_DIR}/home/pi/Documents/Greenfoot Projects"
-rsync -a --chown=1000:1000 ${ROOTFS_DIR}/usr/share/scratch/Projects/Demos/ "${ROOTFS_DIR}/home/pi/Documents/Scratch Projects"
-
 #Alacarte fixes
 install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/.local"
 install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/.local/share"

stage4/02-extras/files/.gitignore

@@ -1,2 +0,0 @@
-python_games.hash
-python_games.tar.gz

stage4/03-cleanup/00-run.sh

@@ -1,5 +0,0 @@
-#!/bin/bash -e
-
-on_chroot << EOF
-apt-get clean
-EOF

stage4/03-magpi/00-run.sh

@@ -0,0 +1,16 @@
+#!/bin/sh -e
+
+magpi_regex="MagPi[[:digit:]]*.pdf"
+magpi_loc="https://www.raspberrypi.org/magpi-issues"
+magpi_latest="$(curl "$magpi_loc/?C=M;O=D" -s | grep "$magpi_regex" -m 1 -o | head -n 1)"
+
+if [ ! -f "files/$magpi_latest" ]; then
+	find files/ -regextype grep -regex "files/$magpi_regex" -delete
+	wget "$magpi_loc/$magpi_latest" -O "files/$magpi_latest"
+fi
+
+file "files/$magpi_latest" | grep -q "PDF document"
+
+install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/MagPi"
+install -v -o 1000 -g 1000 -m 644 "files/$magpi_latest" "${ROOTFS_DIR}/home/pi/MagPi/"
+

stage4/03-magpi/files/.gitignore

@@ -0,0 +1 @@
+*.pdf

stage4/EXPORT_IMAGE

@@ -0,0 +1,4 @@
+IMG_SUFFIX=""
+if [ "${USE_QEMU}" = "1" ]; then
+	export IMG_SUFFIX="${IMG_SUFFIX}-qemu"
+fi

stage4/EXPORT_NOOBS

@@ -0,0 +1,2 @@
+NOOBS_NAME="Raspbian"
+NOOBS_DESCRIPTION="A port of Debian with the Raspberry Pi Desktop"

stage4/prerun.sh

@@ -1,4 +1,5 @@
 #!/bin/bash -e
-if [ ! -d ${ROOTFS_DIR} ]; then
+
+if [ ! -d "${ROOTFS_DIR}" ]; then
 	copy_previous
 fi

stage5/00-install-extras/00-packages

@@ -0,0 +1,15 @@
+python3-thonny
+sonic-pi
+scratch nuscratch scratch2
+smartsim
+
+minecraft-pi python-minecraftpi
+python-sense-emu sense-emu-tools python-sense-emu-doc
+
+wolfram-engine
+claws-mail
+greenfoot bluej
+nodered
+realvnc-vnc-viewer
+
+python-games

stage5/00-install-libreoffice/00-packages

@@ -0,0 +1,3 @@
+libreoffice-pi
+libreoffice-help-en-gb
+libreoffice-l10n-en-gb

stage5/EXPORT_IMAGE

@@ -0,0 +1,4 @@
+IMG_SUFFIX="-full"
+if [ "${USE_QEMU}" = "1" ]; then
+	export IMG_SUFFIX="${IMG_SUFFIX}-qemu"
+fi

stage5/EXPORT_NOOBS

@@ -0,0 +1,2 @@
+NOOBS_NAME="Raspbian Full"
+NOOBS_DESCRIPTION="A port of Debian with desktop and recommended applications"

stage5/prerun.sh

@@ -0,0 +1,5 @@
+#!/bin/bash -e
+
+if [ ! -d "${ROOTFS_DIR}" ]; then
+	copy_previous
+fi