Monday December 10 2018

Booting MS Windows, FreeBSD, and Linux off of the same drive.

First off, you’re probably thinking “WHY!?” The answer? Because I can. Also, because Virtualization sucks. Plus my laptop only has room for one 2.5” SATA 3 (6gb/s) drive, or a slow SATA 2 (3gb/s) mSATA drive–which I do not plan on using

Why does virtualization suck?

Well let’s say I want to run Windows for a particular piece of software, and it’s resource intensive. CPU wise you’re going to end up with a pretty decent performance hit if you’re running it in a VM. Say nothing of the almost non-existant support for accelerated graphics inside of a virtual machine if I dare try to do something graphics intensive such as play a video game.

Next up is networking, if you’re accustomed to most major desktop virtualization software you might know that it does some NAT trickery to make it “just work”–that is until it doesn’t, or need say IPv6. It’s a disaster and doesn’t solve the problem of having incompatible or poorly written non-portable software in the first place. Something I cover in more depth in future posts, so stay tuned.

Moving on,

What is covered here is setting up a laptop with a single disk to boot Windows, FreeBSD, and Linux via UEFI using the GPT partitioning scheme.

Installing Microsoft Windows

Fresh MS Windows install. I’m sure you’ll be able to handle this one. Just be sure to install it in GPT/UEFI mode. An easy way to do this is to download the ISO from Microsoft, and then write it to a flash drive ( be sure to select UEFI only mode ) via the Rufus tool.

Be sure to set the partition size accordingly and leave plenty of room for the other operating systems. Windows requires about 60GB minimum these days. I just gave it 200gb on my X230’s 1TB ssd.

Once it’s installed you may wish to set the system time to use UTC–especially if you’re installing a lot of Unix-like operating systems that expect that functionality.

Simply write this out to a registry file:


Windows Registry Editor Version 5.00


You should then be able to double click on it, ignore the warning and install it.

Installing FreeBSD

This one is a bit tricky. The quick version is that you’ll boot to the install medium and perform a by hand installation.

The partitioning is the only real gotcha, so-to-speak:

# gpart show ada0
=>        34  1953525101  ada0  GPT  (932G)
          34        2014        - free -  (1.0M)
        2048     1021952     1  ms-recovery  (499M)
     1024000      204800     2  efi  (100M)
     1228800       32768     3  ms-reserved  (16M)
     1261568   410118144     4  ms-basic-data  (196G)
   411379712     2097152     5  freebsd-ufs  (1.0G)
   413476864  1048576000     6  freebsd-zfs  (500G)

Above you can see how my partitions are setup, I was able to create them via:

# gpart add -t freebsd-ufs -s 1G ada0
# gpart add -t freebsd-zfs -s 500G ada0

Then (optionally) encrypt and create your filesystems:

# geli init -b -e AES-XTS -l 256 -s 4096 /dev/ada0p6
# geli attach /dev/ada0p6
# zpool create -O compress=lz4 -O canmount=off -R /mnt zroot ada0p6.eli
# zfs create -o canmount=off zroot/ROOT
# zfs create -o mountpoint=/ zroot/ROOT/default
# zfs create -o mountpoint=/home zroot/home
# for _n in usr var ; do
	zfs create -o canmount=off -o mountpoint=/$_n zroot/$_n
# zfs create zroot/usr/ports
# zfs create zroot/var/log
# newfs -L boot /dev/ada0p5

Then go ahead and mount the filesystems:

# mkdir /mnt/boot_real && mount /dev/ufs/boot /mnt/boot_real
# tar -C /mnt -xJf /usr/freebsd-dist/base.txz
# tar -C /mnt -xJf /usr/freebsd-dist/kernel.txz
# mv /mnt/boot /mnt/boot_real/ && ln -s boot_real/boot /mnt/boot
# mkdir /mnt/boot/efi && mount -t msdosfs /dev/ada0p2 /mnt/boot/efi

Move Microsoft’s main bootloader out of the way, we’re going to use this to get going with FreeBSD. ( You’ll still be able to select Windows from the boot menu as it writes an EFI bootloader variable )

# mv /mnt/boot/efi/EFI/Boot/bootx64.efi /mnt/boot/efi/EFI/Boot/msft_bootx64.efi
# cp /mnt/boot/boot1.efi /mnt/boot/efi/EFI/Boot/bootx64.efi

Adjust boot/loader.conf for our ZFS root and encrypted root partition:

# ed /mnt/boot/loader.conf

Adjust /etc/rc.conf in much the same way: ( #TODO: add other flags here, and sysctl variables )

# ed /etc/rc.conf

Set our system’s timezone

# ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime

Make sure our filesystems are in the etc/fstab ( Note that ZFS is not managed through the fstab )

# ed etc/fstab
/dev/ufs/boot	/boot_real	ufs	rw	0	0
# Read only, if we want to reference it.
# Otherwise we can do `mount -u -o rw /boot/efi` to write to it
/dev/ada0p2	/boot/efi	msdosfs	ro	0	0


# sync && reboot

And that should take care of most of the install of FreeBSD. There are still a lot of other little pieces that need to be put in place if you want a usable desktop experience. Stay tuned for that post as well.

Installing Linux

These days I find myself using Gentoo most of the time. It’s stable, easy to specify the build options I want, it’s not subject to the bloat of systemd and binary package are trivial.

For my x230 I purposely wanted to do something different I’ve used Void Linux in the past for a few things. It’s a somewhat unique distribution that uses runit as the default init scheme, libressl as the TLS implementation, and optionally musl as the system’s libc. I’m currently using the musl libc variant although the setup should be similar if you go with glibc

First boot to some sort of Linux distribution, this actually matters very little as we’ll be installing via a chroot.

Partition the disk

# gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.4

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Model: CT1000MX500SSD1 
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): C0C481E6-25C4-4837-8610-3F2D5DCAE3B6
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1023999   499.0 MiB   2700  Basic data partition
   2         1024000         1228799   100.0 MiB   EF00  EFI system partition
   3         1228800         1261567   16.0 MiB    0C01  Microsoft reserved ...
   4         1261568       411379711   195.6 GiB   0700  Basic data partition
   5       411379712       413476863   1024.0 MiB  A503  
   6       413476864      1462052863   500.0 GiB   A503  
   7      1462052864      1464150015   1024.0 MiB  8300  
   8      1464150016      1883580415   200.0 GiB   8300  

Basically, create a 1GB boot partition and a 200GB partition for the OS much like you did with FreeBSD a moment ago.

Setup the filesystems, encryption, and LVM.

There will be free space in the volume group with the configuration below, this makes snapshotting easy.

# mkfs.ext4 /dev/sda7
# cryptsetup luksFormat /dev/sda8
# cryptsetup luksOpen /dev/sda8 root
# pvcreate /dev/mapper/root
# vgcreate vg /dev/mapper/root
# lvcreate vg --name root --size 10G
# lvcreate vg --name home --size 100G
# mkfs.xfs /dev/vg/root
# mkfs.xfs /dev/vg/home

Then go ahead and mount the filesystems:

# mount /dev/vg/root /mnt
# mkdir /mnt/{boot,home}
# mount /dev/vg/home /mnt/home
# mount /dev/sda7 /mnt/boot
# mkdir /mnt/boot/efi
# mount /dev/sda2 /mnt/boot/efi

Download and extract the minimal root filesystem

Be sure to check the website for a mirror near you and change the URL accordingly. You might need to browse the live/current/ path to figure out the current ROOTFS tarball

# cd /mnt
# wget
# tar xJf void-*ROOTFS*

Chroot into the new system and setup the basics

# cd /mnt
# mount -t proc none proc
# mount -o bind /sys sys
# mount -o bind /dev dev
# chroot /mnt /bin/bash
bash-4.4# export PS1="(CHROOT) # "

Setup nameservers

(CHROOT) # echo nameserver > /etc/resolv.conf

Setup the package repository to point at your local mirror

(CHROOT) # echo repository= \
	> /etc/xbps.d/00-repository-main.conf

Install the base system, lvm2, and a few other useful packages

(CHROOT) # xbps-install -Syu \
		base-system lvm2 curl wget ed grub gptfdisk lzop cryptsetup \
		lzip openvpn nmap pv rrdtool rsync sipcalc whois xz wireless_tools \
		net-tools wpa_supplicant grub-x86_64-efi

Configure bootloader, initrd and /etc/fstab

Kind of a hack to setup the fstab, but it works:

(CHROOT) # cat /proc/mounts | grep xfs  >> /etc/fstab

The Void Linux kernel by default does not build in the necessary module to handle booting off of a logical volume with associated snapshots. To make matters worse, it does not load it automatically even if you put it in the initial ramdisk. Thankfully dracut has an option to force load kernel modules


# echo 'force_drivers="dm_snapshot"' >> /etc/dracut.conf

Then go ahead and rebuild the initrd:

(CHROOT) # dracut -f --kver $(ls -t /lib/modules | sed 1q)

Now to check the generated initrd and make sure that it has LVM and Cryptsetup

(CHROOT) # lsinitrd /boot/initrd* | grep lvm >/dev/null || echo "No LVM support"
(CHROOT) # lsinitrd /boot/initrd* | grep cryptsetup >/dev/null || echo "No crypt support"

Instlalling grub

It’s pretty straightforward:

(CHROOT) # grub-install --target=x86_64-efi --efi-directory=/boot/efi \
  --bootloader-id void

Configuring grub

We’re going to need to tell the grub configuration generator how to get to the filesystem. Adjust /etc/default/grub like so:

GRUB_CMDLINE_LINUX_DEFAULT="rd.luks.uuid=$LUKS_UUID rd.luks.allow-discards rd.luks.crypttab=0"

where $LUKS_UUID is actually your UUID, you can get this via:

(CHROOT) # blkid -o export /dev/sda8 | grep '^UUID'

You can leave or remove the extra arguments that are already there. ( Not listed above) You can change the loglevel option if you want to have a little bit more verbose boot.

You will need to remove the rd.luks.allow-discards if you’re not booting to a solid state drive that supports trim. There are also some security concerns with enabling it centering around the fact that it will espose what data is and is not encrypted.

Finally, create the grub configuration file

(CHROOT) # grub-mkconfig -o /boot/grub/grub.cfg

Installing rEFInd

Now that we have more operating systems than we eaisly can select from the boot menu it’s about time we fix that problem by using a UEFI boot manager.

Why not just set some EFI variables and be done with it? Because UEFI is buggy. And on many systems it’s not that simple–or even respected.

I downloaded rEFInd from here I used the “A binary zip file”

I didn’t use the install scripts, instead I just copied the binary over to the EFI directory like so:

$ cd ~/Downloads/
$ bsdtar -xf refind-bin-0.11.4
$ cd refind-bin-0.11.4
$ sudo cp refind/refind_x64.efi /boot/efi/EFI/Boot/bootx64.efi

I then proceeded to create a configuration file like so:

timeout 20

menuentry Void {
    loader /EFI/void/grubx64.efi
    icon /EFI/refind/icons/os_void.png

menuentry FreeBSD {
    loader /EFI/freebsd/bootx64.efi
    icon /EFI/refind/icons/os_freebsd.png

menuentry "Windows 10" {
    loader \EFI\Microsoft\Boot\bootmgfw.efi
    icon \EFI\refind\icons\os_win.png

If you would like the icons and themes as mentioned above, then you can copy over the `refind/icons/’ folder:

$ sudo cp -R refind/icons /boot/efi/EFI/Boot/

If you’re on a system that supports EFI variables you can always add a special entry for it as well:

$ sudo -i
# cd /boot/efi/EFI
# mkdir refind
# cp -R Boot/* refind/
# efibootmgr --create --disk /dev/sda --part 2 --loader \
	/EFI/refind/bootx64.efi --label "rEFInd Boot Manager"

This will obviously not carry over to other systems you install the drive to so I highly recommend that you use the default EFI/Boot directory.

Final thoughts

This is obviously not a common configuration. Very few people have any desire, let alone a need to run multiple operating systems on the same machine, let alone three of them.

I run Linux and FreeBSD production servers for some clients. Although this configuration is not strictly necessary–I could always login to another host via ssh. It does offer a little bit more experience with the operating system in question when you run it with a desktop environment–especially on a laptop.

If I didn’t know how to setup this kind of thing I would be much worse at my job. When shit hits the fan, everything is broken–it’s 04:00 and you’re trying to restore from a backup after you repeatedly told the client to upgrade the hardware before it failed–you don’t have time to place blame or complain. You just have to figure out the damn bootloader and get the system running. Besides, clients are a lot more receptive to the expensive service call and a plan to prevent the problem in the future if you resolve it quickly and without much fuss.

So go on, mess with your systems, multi boot. You’ll learn something and it might come in handy if you work on these kinds of things.

If anyone is interested in how my desktop is setup, feel free to send me a message and I’ll do a writeup on i3, the applications I use, etc. I don’t think it’s anything special however and as such do not plan on writing about it at this time.