export-image: increase free space padding

fixes #65

.dockerignore

@@ -0,0 +1,3 @@
+output/
+work/
+deploy/

.gitignore

@@ -1,6 +1,7 @@
 deploy/*
 work/*
 config
+postrun.sh
 SKIP
 .pc
 *-pc

Dockerfile

@@ -0,0 +1,12 @@
+FROM debian:jessie
+
+RUN apt-get -y update && \
+    apt-get -y install \
+        git vim parted \
+        quilt realpath qemu-user-static debootstrap zerofree pxz zip dosfstools \
+        bsdtar libcap2-bin rsync grep \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY . /pi-gen/
+
+VOLUME [ "/pi-gen/work", "/pi-gen/deploy"]

README.md

@@ -1,12 +1,26 @@
-#TODO
+# pi-gen
 
+_Tool used to create the raspberrypi.org Raspbian images_
+
+
+### TODO
 1. Documentation
 
-#Dependencies
 
-`quilt kpartx realpath qemu-user-static debootstrap zerofree pxz zip dosfstools`
+## Dependencies
 
-#Config
+On Debian-based systems:
+
+```bash
+apt-get install quilt parted realpath qemu-user-static debootstrap zerofree pxz zip \
+dosfstools bsdtar libcap2-bin grep rsync
+```
+
+The file `depends` contains a list of tools needed.  The format of this
+package is `<tool>[:<debian-package>]`.
+
+
+## Config
 
 Upon execution, `build.sh` will source the file `config` in the current
 working directory.  This bash shell fragment is intended to set needed
@@ -27,23 +41,67 @@ The following environment variables are supported:
    will not be included in the image, making it safe to use an `apt-cacher` or
    similar package for development.
 
+ * `BASE_DIR`  (Default: location of `build.sh`)
+
+   **CAUTION**: Currently, changing this value will probably break build.sh
+
+   Top-level directory for `pi-gen`.  Contains stage directories, build
+   scripts, and by default both work and deployment directories.
+
+ * `WORK_DIR`  (Default: `"$BASE_DIR/work"`)
+
+   Directory in which `pi-gen` builds the target system.  This value can be
+   changed if you have a suitably large, fast storage location for stages to
+   be built and cached.  Note, `WORK_DIR` stores a complete copy of the target
+   system for each build stage, amounting to tens of gigabytes in the case of
+   Raspbian.
+
+ * `DEPLOY_DIR`  (Default: `"$BASE_DIR/deploy"`)
+
+   Output directory for target system images and NOOBS bundles.
+
+
 A simple example for building Raspbian:
 
 ```bash
 IMG_NAME='Raspbian'
 ```
 
-#Stage Anatomy
+
+## Docker Build
+
+```bash
+vi config         # Edit your config file. See above.
+./build-docker.sh
+```
+
+If everything goes well, your finished image will be in the `deploy/` folder.
+You can then remove the build container with `docker rm pigen_work`
+
+If something breaks along the line, you can edit the corresponding scripts, and
+continue:
+
+```bash
+CONTINUE=1 ./build-docker.sh
+```
+
+There is a possibility that even when running from a docker container, the
+installation of `qemu-user-static` will silently fail when building the image
+because `binfmt-support` _must be enabled on the underlying kernel_. An easy
+fix is to ensure `binfmt-support` is installed on the host machine before
+starting the `./build-docker.sh` script (or using your own docker build
+solution).
 
 
+## Stage Anatomy
 
-#Raspbian Stage Overview
+### Raspbian Stage Overview
 
 The build of Raspbian is divided up into several stages for logical clarity
 and modularity.  This causes some initial complexity, but it simplifies
 maintenance and allows for more easy customization.
 
- - Stage 0, bootstrap.  The primary purpose of this stage is to create a
+ - **Stage 0** - bootstrap.  The primary purpose of this stage is to create a
    usable filesystem.  This is accomplished largely through the use of
    `debootstrap`, which creates a minimal filesystem suitable for use as a
    base.tgz on Debian systems.  This stage also configures apt settings and
@@ -51,7 +109,7 @@ maintenance and allows for more easy customization.
    minimal core is installed but not configured, and the system will not quite
    boot yet.
 
- - Stage 1, truly minimal system.  This stage makes the system bootable by
+ - **Stage 1** - truly minimal system.  This stage makes the system bootable by
    installing system files like `/etc/fstab`, configures the bootloader, makes
    the network operable, and installs packages like raspi-config.  At this
    stage the system should boot to a local console from which you have the
@@ -60,7 +118,7 @@ maintenance and allows for more easy customization.
    really usable yet in a traditional sense yet.  Still, if you want minimal,
    this is minimal and the rest you could reasonably do yourself as sysadmin.
 
- - State 2, lite system.  This stage produces the Raspbian-Lite image.  It
+ - **Stage 2** - lite system.  This stage produces the Raspbian-Lite image.  It
    installs some optimized memory functions, sets timezone and charmap
    defaults, installs fake-hwclock and ntp, wifi and bluetooth support,
    dphys-swapfile, and other basics for managing the hardware.  It also
@@ -75,13 +133,33 @@ maintenance and allows for more easy customization.
    you were looking for something between truly minimal and Raspbian-lite,
    here's where you start trimming.
 
- - Stage 3, desktop system.  Here's where you get the full desktop system
+ - **Stage 3** - desktop system.  Here's where you get the full desktop system
    with X11 and LXDE, web browsers, git for development, Raspbian custom UI
    enhancements, etc.  This is a base desktop system, with some development
    tools installed.
 
- - Stage 4, complete Raspbian system.  More development tools, an email
+ - **Stage 4** - complete Raspbian system.  More development tools, an email
    client, learning tools like Scratch, specialized packages like sonic-pi and
    wolfram-engine, system documentation, office productivity, etc.  This is
    the stage that installs all of the things that make Raspbian friendly to
    new users.
+
+
+### Stage specification
+
+If you wish to build up to a specified stage (such as building up to stage 2
+for a lite system), place an empty file named `SKIP` in each of the `./stage`
+directories you wish not to include.
+
+Then remove the `EXPORT*` files from `./stage4` (if building up to stage 2) or
+from `./stage2` (if building a minimal system).
+
+```bash
+# Example for building a lite system
+touch ./stage3/SKIP ./stage4/SKIP
+rm stage4/EXPORT*
+```
+
+If you wish to build further configurations upon (for example) the lite
+system, you can also delete the contents of `./stage3` and `./stage4` and
+replace with your own contents in the same format.

build-docker.sh

@@ -0,0 +1,73 @@
+#!/bin/bash -e
+DOCKER="docker"
+set +e
+$DOCKER ps >/dev/null 2>&1
+if [ $? != 0 ]; then
+	DOCKER="sudo docker"
+fi
+if ! $DOCKER ps >/dev/null; then
+	echo "error connecting to docker:"
+	$DOCKER ps
+	exit 1
+fi
+set -e
+
+config_mount=
+if [ -f config ]; then
+	config_mount="-v $(pwd)/config:/pi-gen/config:ro"
+	source config
+fi
+
+CONTAINER_NAME=${CONTAINER_NAME:-pigen_work}
+CONTINUE=${CONTINUE:-0}
+
+if [ "$*" != "" ] || [ -z "${IMG_NAME}" ]; then
+	if [ -z "${IMG_NAME}" ]; then
+		echo "IMG_NAME not set in 'build'" 1>&2
+		echo 1>&2
+	fi
+	cat >&2 <<EOF
+Usage:
+    build-docker.sh [options]
+Optional environment arguments: ( =<default> )
+    CONTAINER_NAME=pigen_work  set a name for the build container
+    CONTINUE=0                  continue from a previously started container
+EOF
+	exit 1
+fi
+
+CONTAINER_EXISTS=$($DOCKER ps -a --filter name="$CONTAINER_NAME" -q)
+CONTAINER_RUNNING=$($DOCKER ps --filter name="$CONTAINER_NAME" -q)
+if [ "$CONTAINER_RUNNING" != "" ]; then
+	echo "The build is already running in container $CONTAINER_NAME. Aborting."
+	exit 1
+fi
+if [ "$CONTAINER_EXISTS" != "" ] && [ "$CONTINUE" != "1" ]; then
+	echo "Container $CONTAINER_NAME already exists and you did not specify CONTINUE=1. Aborting."
+	echo "You can delete the existing container like this:"
+	echo "  docker rm $CONTAINER_NAME"
+	exit 1
+fi
+
+$DOCKER build -t pi-gen .
+if [ "$CONTAINER_EXISTS" != "" ]; then
+	trap "echo 'got CTRL+C... please wait 5s';docker stop -t 5 ${CONTAINER_NAME}_cont" SIGINT SIGTERM
+	time $DOCKER run --rm --privileged \
+		--volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \
+		pi-gen \
+		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
+	cd /pi-gen; ./build.sh;
+	rsync -av work/*/build.log deploy/" &
+	wait
+else
+	trap "echo 'got CTRL+C... please wait 5s'; docker stop -t 5 ${CONTAINER_NAME}" SIGINT SIGTERM
+	$DOCKER run --name "${CONTAINER_NAME}" --privileged \
+		-v $(pwd)/deploy:/pi-gen/deploy \
+		${config_mount} \
+		pi-gen \
+		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
+	cd /pi-gen; ./build.sh &&
+	rsync -av work/*/build.log deploy/" &
+	wait
+fi
+echo "Done! Your image(s) should be in deploy/"

build.sh

@@ -67,10 +67,10 @@ EOF
 			./${i}-run.sh
 			log "End ${SUB_STAGE_DIR}/${i}-run.sh"
 		fi
-		if [ -f ${i}-run-chroot ]; then
+		if [ -f ${i}-run-chroot.sh ]; then
-			log "Begin ${SUB_STAGE_DIR}/${i}-run-chroot"
+			log "Begin ${SUB_STAGE_DIR}/${i}-run-chroot.sh"
-			on_chroot < ${i}-run-chroot
+			on_chroot < ${i}-run-chroot.sh
-			log "End ${SUB_STAGE_DIR}/${i}-run-chroot"
+			log "End ${SUB_STAGE_DIR}/${i}-run-chroot.sh"
 		fi
 	done
 	popd > /dev/null
@@ -132,8 +132,8 @@ export IMG_DATE=${IMG_DATE:-"$(date -u +%Y-%m-%d)"}
 
 export BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 export SCRIPT_DIR="${BASE_DIR}/scripts"
-export WORK_DIR="${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"
+export WORK_DIR=${WORK_DIR:-"${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"}
-export DEPLOY_DIR="${BASE_DIR}/deploy"
+export DEPLOY_DIR=${DEPLOY_DIR:-"${BASE_DIR}/deploy"}
 export LOG_FILE="${WORK_DIR}/build.log"
 
 export CLEAN
@@ -148,6 +148,8 @@ export PREV_STAGE_DIR
 export ROOTFS_DIR
 export PREV_ROOTFS_DIR
 export IMG_SUFFIX
+export NOOBS_NAME
+export NOOBS_DESCRIPTION
 export EXPORT_DIR
 export EXPORT_ROOTFS_DIR
 
@@ -172,13 +174,21 @@ done
 CLEAN=1
 for EXPORT_DIR in ${EXPORT_DIRS}; do
 	STAGE_DIR=${BASE_DIR}/export-image
-	IMG_SUFFIX=$(cat ${EXPORT_DIR}/EXPORT_IMAGE)
+	source "${EXPORT_DIR}/EXPORT_IMAGE"
 	EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename ${EXPORT_DIR})/rootfs
 	run_stage
 	if [ -e ${EXPORT_DIR}/EXPORT_NOOBS ]; then
+		source ${EXPORT_DIR}/EXPORT_NOOBS
 		STAGE_DIR=${BASE_DIR}/export-noobs
 		run_stage
 	fi
 done
 
+if [ -x postrun.sh ]; then
+	log "Begin postrun.sh"
+	cd "${BASE_DIR}"
+	./postrun.sh
+	log "End postrun.sh"
+fi
+
 log "End ${BASE_DIR}"

depends

@@ -1,8 +1,12 @@
 quilt
+parted
+realpath
 qemu-arm-static:qemu-user-static
 debootstrap
-kpartx zerofree
+zerofree
 pxz zip
 mkdosfs:dosfstools
 capsh:libcap2-bin
 bsdtar
+grep
+rsync

export-image/03-set-partuuid/00-run.sh

@@ -0,0 +1,12 @@
+#!/bin/bash -e
+IMG_FILE="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img"
+
+IMGID="$(fdisk -l ${IMG_FILE} | sed -n 's/Disk identifier: 0x\([^ ]*\)/\1/p')"
+
+BOOT_PARTUUID="${IMGID}-01"
+ROOT_PARTUUID="${IMGID}-02"
+
+sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" ${ROOTFS_DIR}/etc/fstab
+sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" ${ROOTFS_DIR}/etc/fstab
+
+sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" ${ROOTFS_DIR}/boot/cmdline.txt

export-image/04-finalise/01-run.sh

@@ -18,6 +18,31 @@ if [ -e ${ROOTFS_DIR}/etc/ld.so.preload.disabled ]; then
         mv ${ROOTFS_DIR}/etc/ld.so.preload.disabled ${ROOTFS_DIR}/etc/ld.so.preload
 fi
 
+rm -f ${ROOTFS_DIR}/etc/apt/sources.list~
+rm -f ${ROOTFS_DIR}/etc/apt/trusted.gpg~
+
+rm -f ${ROOTFS_DIR}/etc/passwd-
+rm -f ${ROOTFS_DIR}/etc/group-
+rm -f ${ROOTFS_DIR}/etc/shadow-
+rm -f ${ROOTFS_DIR}/etc/gshadow-
+
+rm -f ${ROOTFS_DIR}/var/cache/debconf/*-old
+rm -f ${ROOTFS_DIR}/var/lib/dpkg/*-old
+
+rm -f ${ROOTFS_DIR}/usr/share/icons/*/icon-theme.cache
+
+rm -f ${ROOTFS_DIR}/var/lib/dbus/machine-id
+
+true > ${ROOTFS_DIR}/etc/machine-id
+
+ln -nsf /proc/mounts ${ROOTFS_DIR}/etc/mtab
+
+for _FILE in $(find ${ROOTFS_DIR}/var/log/ -type f); do
+	true > ${_FILE}
+done
+
+rm -f "${ROOTFS_DIR}/root/.vnc/private.key"
+
 update_issue $(basename ${EXPORT_DIR})
 install -m 644 ${ROOTFS_DIR}/etc/rpi-issue ${ROOTFS_DIR}/boot/issue.txt
 install files/LICENSE.oracle ${ROOTFS_DIR}/boot/

export-image/prerun.sh

@@ -8,19 +8,19 @@ rm -f ${IMG_FILE}
 rm -rf ${ROOTFS_DIR}
 mkdir -p ${ROOTFS_DIR}
 
-BOOT_SIZE=$(du -sh ${EXPORT_ROOTFS_DIR}/boot -B M | cut -f 1 | tr -d M)
+BOOT_SIZE=$(du --apparent-size -s ${EXPORT_ROOTFS_DIR}/boot --block-size=1 | cut -f 1)
-TOTAL_SIZE=$(du -sh ${EXPORT_ROOTFS_DIR} -B M | cut -f 1 | tr -d M)
+TOTAL_SIZE=$(du --apparent-size -s ${EXPORT_ROOTFS_DIR} --exclude var/cache/apt/archives --block-size=1 | cut -f 1)
 
-IMG_SIZE=$(expr $BOOT_SIZE \* 2 \+ $TOTAL_SIZE \+ 512)M
+IMG_SIZE=$((BOOT_SIZE + TOTAL_SIZE + (800 * 1024 * 1024)))
 
-fallocate -l ${IMG_SIZE} ${IMG_FILE}
+truncate -s ${IMG_SIZE} ${IMG_FILE}
-fdisk ${IMG_FILE} > /dev/null 2>&1 <<EOF
+fdisk -H 255 -S 63 ${IMG_FILE} <<EOF
 o
 n
 
 
 8192
-+`expr $BOOT_SIZE \* 3`M
++$((BOOT_SIZE * 2 /512))
 p
 t
 c
@@ -34,15 +34,27 @@ p
 w
 EOF
 
-LOOP_DEV=`kpartx -asv ${IMG_FILE} | grep -E -o -m1 'loop[[:digit:]]+' | head -n 1`
+PARTED_OUT=$(parted -s ${IMG_FILE} unit b print)
-BOOT_DEV=/dev/mapper/${LOOP_DEV}p1
+BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
-ROOT_DEV=/dev/mapper/${LOOP_DEV}p2
+| cut -d" " -f 2 | tr -d B)
+BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
+| cut -d" " -f 4 | tr -d B)
 
-mkdosfs -n boot -S 512 -s 16 -v $BOOT_DEV > /dev/null
+ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
+| cut -d" " -f 2 | tr -d B)
+ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
+| cut -d" " -f 4 | tr -d B)
+
+BOOT_DEV=$(losetup --show -f -o ${BOOT_OFFSET} --sizelimit ${BOOT_LENGTH} ${IMG_FILE})
+ROOT_DEV=$(losetup --show -f -o ${ROOT_OFFSET} --sizelimit ${ROOT_LENGTH} ${IMG_FILE})
+echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH"
+echo "/:     offset $ROOT_OFFSET, length $ROOT_LENGTH"
+
+mkdosfs -n boot -F 32 -v $BOOT_DEV > /dev/null
 mkfs.ext4 -O ^huge_file $ROOT_DEV > /dev/null
 
 mount -v $ROOT_DEV ${ROOTFS_DIR} -t ext4
 mkdir -p ${ROOTFS_DIR}/boot
 mount -v $BOOT_DEV ${ROOTFS_DIR}/boot -t vfat
 
-rsync -aHAXx ${EXPORT_ROOTFS_DIR}/ ${ROOTFS_DIR}/
+rsync -aHAXx --exclude var/cache/apt/archives ${EXPORT_ROOTFS_DIR}/ ${ROOTFS_DIR}/

export-noobs/00-release/00-run.sh

@@ -5,7 +5,7 @@ NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"
 install -v -m 744	files/partition_setup.sh	${NOOBS_DIR}/
 install -v		files/partitions.json		${NOOBS_DIR}/
 install -v		files/os.json			${NOOBS_DIR}/
-install -v		files/Raspbian.png		${NOOBS_DIR}/
+install -v		files/OS.png			${NOOBS_DIR}/
 install -v		files/release_notes.txt		${NOOBS_DIR}/
 
 tar -v -c -C		files/marketing			-f ${NOOBS_DIR}/marketing.tar .
@@ -19,13 +19,18 @@ ROOT_SIZE=$(expr ${ROOT_SIZE} / 1000000 \+ 1)
 BOOT_NOM=$(expr ${BOOT_SIZE} \* 3)
 ROOT_NOM=$(expr ${ROOT_SIZE} \+ 400)
 
+mv "${NOOBS_DIR}/OS.png" "${NOOBS_DIR}/$(echo ${NOOBS_NAME} | sed 's/ /_/g').png"
+
 sed ${NOOBS_DIR}/partitions.json -i -e "s|BOOT_SIZE|${BOOT_SIZE}|"
 sed ${NOOBS_DIR}/partitions.json -i -e "s|ROOT_SIZE|${ROOT_SIZE}|"
 
 sed ${NOOBS_DIR}/partitions.json -i -e "s|BOOT_NOM|${BOOT_NOM}|"
 sed ${NOOBS_DIR}/partitions.json -i -e "s|ROOT_NOM|${ROOT_NOM}|"
 
-sed ${NOOBS_DIR}/release_notes.txt -i -e "s|UNRELEASED|${IMG_DATE}|"
 sed ${NOOBS_DIR}/os.json -i -e "s|UNRELEASED|${IMG_DATE}|"
+sed ${NOOBS_DIR}/os.json -i -e "s|NOOBS_NAME|${NOOBS_NAME}|"
+sed ${NOOBS_DIR}/os.json -i -e "s|NOOBS_DESCRIPTION|${NOOBS_DESCRIPTION}|"
+
+sed ${NOOBS_DIR}/release_notes.txt -i -e "s|UNRELEASED|${IMG_DATE}|"
 
 cp -a ${NOOBS_DIR} ${DEPLOY_DIR}/

export-noobs/00-release/files/os.json

@@ -1,8 +1,8 @@
 {
-    "description": "A community-created port of Debian jessie for the Raspberry Pi",
+    "description": "NOOBS_DESCRIPTION",
     "feature_level": 35120124,
-    "kernel": "4.4",
+    "kernel": "4.9",
-    "name": "Raspbian",
+    "name": "NOOBS_NAME",
     "password": "raspberry",
     "release_date": "UNRELEASED",
     "supported_hex_revisions": "2,3,4,5,6,7,8,9,d,e,f,10,11,12,14,19,1040,1041,0092,0093,2082",

export-noobs/00-release/files/partition_setup.sh

@@ -12,10 +12,22 @@ mkdir -p /tmp/1 /tmp/2
 mount "$part1" /tmp/1
 mount "$part2" /tmp/2
 
-sed /tmp/1/cmdline.txt -i -e "s|root=/dev/[^ ]*|root=${part2}|"
+sed /tmp/1/cmdline.txt -i -e "s|root=[^ ]*|root=${part2}|"
 sed /tmp/2/etc/fstab -i -e "s|^.* / |${part2}  / |"
 sed /tmp/2/etc/fstab -i -e "s|^.* /boot |${part1}  /boot |"
 
+if [ -f /mnt/ssh ]; then
+  cp /mnt/ssh /tmp/1/
+fi
+
+if [ -f /mnt/ssh.txt ]; then
+  cp /mnt/ssh.txt /tmp/1/
+fi
+
+if [ -f /settings/wpa_supplicant.conf ]; then
+  cp /settings/wpa_supplicant.conf /tmp/1/
+fi
+
 if ! grep -q resize /proc/cmdline; then
   sed -i 's| quiet init=/usr/lib/raspi-config/init_resize.sh||' /tmp/1/cmdline.txt
 fi

export-noobs/00-release/files/partitions.json

@@ -3,6 +3,7 @@
         {
             "filesystem_type": "FAT",
             "label": "boot",
+            "mkfs_options": "-F 32",
             "partition_size_nominal": BOOT_NOM,
             "uncompressed_tarball_size": BOOT_SIZE,
             "want_maximised": false

export-noobs/00-release/files/release_notes.txt

@@ -1,4 +1,36 @@
 UNRELEASED:
+  * Scratch 2 application included
+  * Thonny Python IDE included
+  * New icons with thinner outlines
+  * Volume control more linear in behaviour
+  * Updated Flash player
+  * Updated RealVNC server and viewer
+  * Various tweaks and bugfixes
+  * New kernel and firmware
+2017-04-10:
+  * Wolfram Mathematica updated to version 11.0.1
+  * Adobe Flash Player updated to version 25.0.0.127
+  * Use PARTUUID to support USB boot
+2017-03-02:
+  * Updated kernel and firmware (final Pi Zero W support)
+  * Wolfram Mathematica updated to version 11
+  * NOOBS installs now checks for presence of 'ssh' file on the NOOBS partition.
+2017-02-16:
+  * Chromium browser updated to version 56
+  * Adobe Flash Player updated to version 24.0.0.221
+  * RealVNC Server and Viewer updated to version 6.0.2 (RealVNC Connect)
+  * Sonic Pi updated to version 2.11
+  * Node-RED updated to version 0.15.3
+  * Scratch updated to version 120117
+  * Detection of SSH enabled with default password moved into PAM
+  * Updated desktop GL driver to support use of fake KMS option
+  * Raspberry Pi Configuration and raspi-config allow setting of fixed HDMI resolution
+  * raspi-config allows enabling of serial hardware independent of serial terminal
+  * Updates to kernel and firmware
+  * Various minor bug fixes and usability and appearance tweaks
+2017-01-11:
+  * Re-release of the 2016-11-25 image with a FAT32-formatted boot partition
+2016-11-25:
   * SSH disabled by default; can be enabled by creating a file with name "ssh" in boot partition
   * Prompt for password change at boot when SSH enabled with default password unchanged
   * Adobe Flash Player included

export-noobs/prerun.sh

@@ -9,9 +9,21 @@ cp ${WORK_DIR}/export-image/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}.img ${STAGE_WOR
 
 rm -rf ${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}
 
-LOOP_DEV=`kpartx -asv ${IMG_FILE} | grep -E -o -m1 'loop[[:digit:]]+' | head -n 1`
+PARTED_OUT=$(parted -s ${IMG_FILE} unit b print)
-BOOT_DEV=/dev/mapper/${LOOP_DEV}p1
+BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
-ROOT_DEV=/dev/mapper/${LOOP_DEV}p2
+| cut -d" " -f 2 | tr -d B)
+BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 1'| xargs echo -n \
+| cut -d" " -f 4 | tr -d B)
+
+ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
+| cut -d" " -f 2 | tr -d B)
+ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^ 2'| xargs echo -n \
+| cut -d" " -f 4 | tr -d B)
+
+BOOT_DEV=$(losetup --show -f -o ${BOOT_OFFSET} --sizelimit ${BOOT_LENGTH} ${IMG_FILE})
+ROOT_DEV=$(losetup --show -f -o ${ROOT_OFFSET} --sizelimit ${ROOT_LENGTH} ${IMG_FILE})
+echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH"
+echo "/:     offset $ROOT_OFFSET, length $ROOT_LENGTH"
 
 mkdir -p ${STAGE_WORK_DIR}/rootfs
 mkdir -p ${NOOBS_DIR}

scripts/common

@@ -1,10 +1,11 @@
 log (){
-	date +"[%T] $@" | tee -a ${LOG_FILE}
+	date +"[%T] $@" | tee -a "${LOG_FILE}"
 }
 export -f log
 
 bootstrap(){
-	local ARCH=$(dpkg --print-architecture)
+	local ARCH
+	ARCH=$(dpkg --print-architecture)
 
 	export http_proxy=${APT_PROXY}
 
@@ -15,19 +16,19 @@ bootstrap(){
 	fi
 
 	capsh --drop=cap_setfcap -- -c "${BOOTSTRAP_CMD} --components=main,contrib,non-free \
-		--arch armhf\
+		--arch armhf \
-		--no-check-gpg \
+		--keyring "${STAGE_DIR}/files/raspberrypi.gpg" \
-		$1 $2 $3"
+		$1 $2 $3" || rmdir "$2/debootstrap"
 }
 export -f bootstrap
 
 copy_previous(){
-	if [ ! -d ${PREV_ROOTFS_DIR} ]; then
+	if [ ! -d "${PREV_ROOTFS_DIR}" ]; then
 		echo "Previous stage rootfs not found"
 		false
 	fi
-	mkdir -p ${ROOTFS_DIR}
+	mkdir -p "${ROOTFS_DIR}"
-	rsync -aHAXx ${PREV_ROOTFS_DIR}/ ${ROOTFS_DIR}/
+	rsync -aHAXx --exclude var/cache/apt/archives "${PREV_ROOTFS_DIR}/" "${ROOTFS_DIR}/"
 }
 export -f copy_previous
 
@@ -38,10 +39,11 @@ unmount(){
 		DIR=$1
 	fi
 
-	while mount | grep -q $DIR; do
+	while mount | grep -q "$DIR"; do
-		local LOCS=`mount | grep $DIR | cut -f 3 -d ' ' | sort -r`
+		local LOCS
+		LOCS=$(mount | grep "$DIR" | cut -f 3 -d ' ' | sort -r)
 		for loc in $LOCS; do
-			umount $loc
+			umount "$loc"
 		done
 	done
 }
@@ -50,42 +52,47 @@ export -f unmount
 unmount_image(){
 	sync
 	sleep 1
-	local LOOP_DEV=$(losetup -j ${1} | cut -f1 -d':')
+	local LOOP_DEVICES
+	LOOP_DEVICES=$(losetup -j "${1}" | cut -f1 -d':')
+	for LOOP_DEV in ${LOOP_DEVICES}; do
 		if [ -n "${LOOP_DEV}" ]; then
-		local MOUNTED_DIR=$(mount | grep $(basename ${LOOP_DEV}) | head -n 1 | cut -f 3 -d ' ')
+			local MOUNTED_DIR
-		if [ -n "${MOUNTED_DIR}" ]; then
+			MOUNTED_DIR=$(mount | grep "$(basename "${LOOP_DEV}")" | head -n 1 | cut -f 3 -d ' ')
-			unmount $(dirname ${MOUNTED_DIR})
+			if [ -n "${MOUNTED_DIR}" ] && [ "${MOUNTED_DIR}" != "/" ]; then
+				unmount "$(dirname "${MOUNTED_DIR}")"
 			fi
 			sleep 1
-		kpartx -ds ${LOOP_DEV}
+			losetup -d "${LOOP_DEV}"
-		losetup -d ${LOOP_DEV}
 		fi
+	done
 }
 export -f unmount_image
 
 on_chroot() {
-	if ! mount | grep -q `realpath ${ROOTFS_DIR}/proc`; then
+	if ! mount | grep -q "$(realpath "${ROOTFS_DIR}"/proc)"; then
-		mount -t proc proc ${ROOTFS_DIR}/proc
+		mount -t proc proc "${ROOTFS_DIR}/proc"
 	fi
 
-	if ! mount | grep -q `realpath ${ROOTFS_DIR}/dev`; then
+	if ! mount | grep -q "$(realpath "${ROOTFS_DIR}"/dev)"; then
-		mount --bind /dev ${ROOTFS_DIR}/dev
+		mount --bind /dev "${ROOTFS_DIR}/dev"
 	fi
 	
-	if ! mount | grep -q `realpath ${ROOTFS_DIR}/dev/pts`; then
+	if ! mount | grep -q "$(realpath "${ROOTFS_DIR}"/dev/pts)"; then
-		mount --bind /dev/pts ${ROOTFS_DIR}/dev/pts
+		mount --bind /dev/pts "${ROOTFS_DIR}/dev/pts"
 	fi
 
-	if ! mount | grep -q `realpath ${ROOTFS_DIR}/sys`; then
+	if ! mount | grep -q "$(realpath "${ROOTFS_DIR}"/sys)"; then
-		mount --bind /sys ${ROOTFS_DIR}/sys
+		mount --bind /sys "${ROOTFS_DIR}/sys"
 	fi
 
-	capsh --drop=cap_setfcap --chroot=${ROOTFS_DIR}/ -- "$@"
+	capsh --drop=cap_setfcap "--chroot=${ROOTFS_DIR}/" -- "$@"
 }
 export -f on_chroot
 
 update_issue() {
-	local GIT_HASH=$(git rev-parse HEAD)
+	local GIT_HASH
-	echo -e "Raspberry Pi reference ${IMG_DATE}\nGenerated using pi-gen, https://github.com/RPi-Distro/pi-gen, ${GIT_HASH}, ${1}" > ${ROOTFS_DIR}/etc/rpi-issue
+	GIT_HASH=$(git rev-parse HEAD)
+	echo -e "Raspberry Pi reference ${IMG_DATE}\nGenerated using pi-gen, https://github.com/RPi-Distro/pi-gen, ${GIT_HASH}, ${1}" > "${ROOTFS_DIR}/etc/rpi-issue"
 }
 export -f update_issue
+

stage0/01-locale/00-packages

@@ -0,0 +1 @@
+locales

stage1/00-boot-files/files/cmdline.txt

@@ -1 +1 @@
-dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
+dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

stage1/00-boot-files/files/config.txt

@@ -1,5 +1,5 @@
 # For more options and information see
-# http://www.raspberrypi.org/documentation/configuration/config-txt.md
+# http://rpf.io/configtxt
 # Some settings may impact device functionality. See link above for details
 
 # uncomment if you get no picture on HDMI for a default "safe" mode

stage1/01-sys-tweaks/files/fstab

@@ -1,3 +1,3 @@
 proc            /proc           proc    defaults          0       0
-/dev/mmcblk0p1  /boot           vfat    defaults          0       2
+BOOTDEV  /boot           vfat    defaults          0       2
-/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
+ROOTDEV  /               ext4    defaults,noatime  0       1

stage2/01-sys-tweaks/00-packages

@@ -1,4 +1,4 @@
-ssh locales less fbset sudo psmisc strace module-init-tools ed ncdu crda
+ssh less fbset sudo psmisc strace module-init-tools ed ncdu crda
 console-setup keyboard-configuration debconf-utils parted unzip
 build-essential manpages-dev python bash-completion gdb pkg-config
 python-rpi.gpio v4l-utils
@@ -14,3 +14,6 @@ raspberrypi-sys-mods
 pi-bluetooth
 apt-listchanges
 usb-modeswitch
+apt-transport-https
+libpam-chksshpwd
+rpi-update

stage2/01-sys-tweaks/00-patches/03-console-setup.diff

@@ -1,17 +0,0 @@
-Index: jessie-stage2/rootfs/etc/default/console-setup
-===================================================================
---- jessie-stage2.orig/rootfs/etc/default/console-setup
-+++ jessie-stage2/rootfs/etc/default/console-setup
-@@ -6,9 +6,9 @@ ACTIVE_CONSOLES="/dev/tty[1-6]"
- 
- CHARMAP="UTF-8"
- 
--CODESET="Lat15"
--FONTFACE="Fixed"
--FONTSIZE="8x16"
-+CODESET="guess"
-+FONTFACE=""
-+FONTSIZE=""
- 
- VIDEOMODE=
- 

stage2/01-sys-tweaks/00-patches/07-resize-init.diff

@@ -1,5 +1,5 @@
 --- a/rootfs/boot/cmdline.txt
 +++ b/rootfs/boot/cmdline.txt
 @@ -1 +1 @@
--dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
+-dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
-+dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh
++dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh

stage2/01-sys-tweaks/00-patches/series

@@ -1,6 +1,5 @@
 01-useradd.diff
 02-swap.diff
-03-console-setup.diff
 04-inputrc.diff
 05-path.diff
 06-rc_local.diff

stage2/01-sys-tweaks/01-run.sh

@@ -9,6 +9,7 @@ install -m 644 files/ttyoutput.conf			${ROOTFS_DIR}/etc/systemd/system/rc-local.
 
 install -m 644 files/50raspi				${ROOTFS_DIR}/etc/apt/apt.conf.d/
 
+install -m 644 files/console-setup   			${ROOTFS_DIR}/etc/default/
 
 on_chroot << EOF
 systemctl disable hwclock.sh

stage2/01-sys-tweaks/files/console-setup

@@ -0,0 +1,16 @@
+# CONFIGURATION FILE FOR SETUPCON
+
+# Consult the console-setup(5) manual page.
+
+ACTIVE_CONSOLES="/dev/tty[1-6]"
+
+CHARMAP="UTF-8"
+
+CODESET="guess"
+FONTFACE=""
+FONTSIZE=""
+
+VIDEOMODE=
+
+# The following is an example how to use a braille font
+# FONT='lat9w-08.psf.gz brl-8x8.psf'

stage2/01-sys-tweaks/files/resize2fs_once

@@ -12,7 +12,7 @@
 case "$1" in
   start)
     log_daemon_msg "Starting resize2fs_once"
-    ROOT_DEV=`grep -Eo 'root=[[:graph:]]+' /proc/cmdline | cut -d '=' -f 2-` &&
+    ROOT_DEV=$(findmnt / -o source -n) &&
     resize2fs $ROOT_DEV &&
     update-rc.d resize2fs_once remove &&
     rm /etc/init.d/resize2fs_once &&

stage2/03-cleanup/00-run.sh

@@ -1,5 +0,0 @@
-#!/bin/bash -e
-
-on_chroot << EOF
-apt-get clean
-EOF

stage2/EXPORT_IMAGE

@@ -1 +1 @@
--lite
+IMG_SUFFIX="-lite"

stage2/EXPORT_NOOBS

@@ -0,0 +1,2 @@
+NOOBS_NAME="Raspbian Lite"
+NOOBS_DESCRIPTION="A port of Debian jessie for the Raspberry Pi (minimal version)"

stage3/00-install-packages/00-packages

@@ -1,7 +1,7 @@
 gstreamer1.0-x gstreamer1.0-omx gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-alsa gstreamer1.0-libav
 xpdf gtk2-engines alsa-utils
 desktop-base
-git-core rpi-update
+git
 omxplayer
 raspberrypi-artwork
 policykit-1

stage4/00-install-packages/00-debconf

@@ -1,4 +1,2 @@
 # Enable realtime process priority?
 jackd2  jackd/tweak_rt_limits   boolean true
-# Do you accept the Wolfram - Raspberry Pi® Bundle License Agreement?
-wolfram-engine  shared/accepted-wolfram-eula    boolean true

stage4/00-install-packages/00-packages

@@ -7,7 +7,6 @@ python3-pgzero
 python-serial python3-serial
 python-picamera python3-picamera
 debian-reference-en dillo x2x
-wolfram-engine
 raspberrypi-net-mods raspberrypi-ui-mods
 smartsim penguinspuzzle
 python-pip python3-pip
@@ -27,7 +26,6 @@ piclone
 pulseaudio-module-bluetooth
 wiringpi pigpio python-pigpio python3-pigpio raspi-gpio python-gpiozero python3-gpiozero python3-rpi.gpio
 python-spidev python3-spidev
-realvnc-vnc-server realvnc-vnc-viewer
 python-twython python3-twython
 python-smbus python3-smbus
 python-flask python3-flask
@@ -35,3 +33,5 @@ python-picraft python3-picraft
 python3-codebug-tether python3-codebug-i2c-tether
 python-sense-emu python3-sense-emu sense-emu-tools python-sense-emu-doc
 pprompt
+scratch2
+python3-thonny

stage4/00-install-packages/00-packages-nr

@@ -1,3 +1,4 @@
 libreoffice libreoffice-gtk
 timidity
 pi-package
+realvnc-vnc-server realvnc-vnc-viewer

stage4/00-install-packages/01-packages

@@ -0,0 +1,19 @@
+python-automationhat python3-automationhat
+python-blinkt python3-blinkt
+python-cap1xxx python3-cap1xxx
+python-drumhat python3-drumhat
+python-envirophat python3-envirophat
+python-explorerhat python3-explorerhat
+python-fourletterphat python3-fourletterphat
+python-microdotphat python3-microdotphat
+python-mote python3-mote
+python-motephat python3-motephat
+python-phatbeat python3-phatbeat
+python-pianohat python3-pianohat
+python-piglow python3-piglow
+python-rainbowhat python3-rainbowhat
+python-scrollphat python3-scrollphat
+python-scrollphathd python3-scrollphathd
+python-sn3218 python3-sn3218
+python-skywriter python3-skywriter
+python-touchphat python3-touchphat

stage4/00-install-packages/01-run.sh

@@ -1,3 +0,0 @@
-#!/bin/bash -e
-
-rm -f "${ROOTFS_DIR}/root/.vnc/private.key"

stage4/02-extras/00-run.sh

@@ -31,28 +31,3 @@ install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/.local"
 install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/.local/share"
 install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/.local/share/applications"
 install -v -o 1000 -g 1000 -d "${ROOTFS_DIR}/home/pi/.local/share/desktop-directories"
-
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config/pcmanfm
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config/pcmanfm/LXDE-pi
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config/openbox
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config/lxsession
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.themes
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config/gtk-3.0
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/.config/lxpanel
-install -v -o 1000 -g 1000 -d ${ROOTFS_DIR}/home/pi/Desktop
-
-install -v -m 644 -o 1000 -g 1000 ${ROOTFS_DIR}/etc/xdg/pcmanfm/LXDE-pi/pcmanfm.conf ${ROOTFS_DIR}/home/pi/.config/pcmanfm/LXDE-pi/
-install -v -m 644 -o 1000 -g 1000 ${ROOTFS_DIR}/etc/xdg/pcmanfm/LXDE-pi/desktop-items-0.conf ${ROOTFS_DIR}/home/pi/.config/pcmanfm/LXDE-pi/
-
-install -v -m 644 -o 1000 -g 1000 ${ROOTFS_DIR}/etc/xdg/openbox/lxde-pi-rc.xml ${ROOTFS_DIR}/home/pi/.config/openbox/
-
-rsync -a --chown=1000:1000 ${ROOTFS_DIR}/etc/xdg/lxsession/LXDE-pi ${ROOTFS_DIR}/home/pi/.config/lxsession/
-rsync -a --chown=1000:1000 ${ROOTFS_DIR}/usr/share/themes/PiX ${ROOTFS_DIR}/home/pi/.themes/
-
-install -v -m 644 -o 1000 -g 1000 ${ROOTFS_DIR}/usr/share/raspi-ui-overrides/gtk.css ${ROOTFS_DIR}/home/pi/.config/gtk-3.0/
-
-install -v -m 644 -o 1000 -g 1000 ${ROOTFS_DIR}/usr/share/raspi-ui-overrides/Trolltech.conf ${ROOTFS_DIR}/home/pi/.config/
-
-install -v -m 644 -o 1000 -g 1000 ${ROOTFS_DIR}/etc/xdg/lxpanel/launchtaskbar.cfg ${ROOTFS_DIR}/home/pi/.config/lxpanel/
-rsync -a --chown=1000:1000 ${ROOTFS_DIR}/etc/xdg/lxpanel/profile/LXDE-pi ${ROOTFS_DIR}/home/pi/.config/lxpanel/

stage4/03-cleanup/00-run.sh

@@ -1,5 +0,0 @@
-#!/bin/bash -e
-
-on_chroot << EOF
-apt-get clean
-EOF

stage4/EXPORT_IMAGE

@@ -0,0 +1 @@
+IMG_SUFFIX="-4GB"

stage5/00-install-mathematica/00-debconf

@@ -0,0 +1,2 @@
+# Do you accept the Wolfram - Raspberry Pi® Bundle License Agreement?
+wolfram-engine  shared/accepted-wolfram-eula    boolean true

stage5/00-install-mathematica/00-packages

@@ -0,0 +1 @@
+wolfram-engine

stage5/EXPORT_IMAGE

@@ -0,0 +1 @@
+IMG_SUFFIX=""

stage5/EXPORT_NOOBS

@@ -0,0 +1,2 @@
+NOOBS_NAME="Raspbian"
+NOOBS_DESCRIPTION="A port of Debian jessie for the Raspberry Pi (full desktop version)"

stage5/prerun.sh

@@ -0,0 +1,4 @@
+#!/bin/bash -e
+if [ ! -d ${ROOTFS_DIR} ]; then
+	copy_previous
+fi