Some helpful links I used through this process:
- https://www.cyberciti.biz/hardware/cryptsetup-add-enable-luks-disk-encryption-keyfile-linux/
- https://wiki.archlinux.org/title/LVM
- https://linuxconfig.org/partition-encryption
- https://www.linux.com/training-tutorials/how-full-encrypt-your-linux-system-lvm-luks/
I’ve been considering offloading some of my critical LAN services to the cloud for some time now, but I’ve been hesitant for two reasons: 1) much of it is sensitive data, and 2) cloud storage is expensive. But, two things clicked recently and catapulted me into this project: 1) trying installation from a custom ISO for the first time by setting up OPNsense as an always-on cloud DNS server (in other words, realizing I could install a custom OS with full-disk encryption) and 2) discovering Vultr’s block storage option - $2.50/mo per 100gb if using their HDD storage - which is totally reasonable. So for the price of a monthly streaming subscription, I could have an always-on cloud server to run some of my critical services, as opposed to crossing my fingers and hoping my home internet doesn’t go out at an inopportune time.
I finally got some time off which gave me a chance to run through the initial installation process.
Debian installer with full-disk encryption
- Create a VPS using a Debian iso instead of the default available Debian provided by Vultr
- Go through guided installer, configure LVM with encryption
Pretty straightforward, no CLI required. I’ve done this a million times at this point, just not with FDE for quite some time.
One thing to keep in mind about FDE in the cloud: I will need to enter a password in the web console access upon every reboot, so I might end up with unexpected downtime. This is something I’ll keep an eye on.
Encrypting block storage with LUKS and LVM
Now though, I want to encrypt attached block storage and have it mount automatically at boot. I don’t really want to enter two passwords, and the boot device is already encrypted, so I can safely store a keyfile for the other mountpoint.
(And I could have gotten away with not using it, but I figured it was a good time to review LVM as well.)
cryptsetup
This is the initial encryption of the new drive, after creating the block storage and attaching it to the running VM. It showed up as /dev/vdb for me after attaching it.
# Create and lock down keyfile
openssl genrsa -out .keyfile 4096
chmod 400 .keyfile
# Format with password - this will serve as a backup in case we lose the keyfile
cryptsetup luksFormat /dev/vdb
# Add keyfile so that it will automatically unlock at boot
cryptsetup luksAddKey /dev/vdb .keyfile
# Test opening the drive
cryptsetup luksOpen /dev/vdb block --key-file .keyfile
lvm
This is creating the “partitioning” using the more flexible lvm or logical volume manager, as opposed to partitioning it directly with a tool like fdisk.
# Create physical volume (pv)
pvcreate /dev/mapper/block
# Create volume group (vg)
vgcreate vgblock /dev/mapper/block
# Create logical volume (lv, "partition")
lvcreate -n lv01_data -l +100%FREE vgblock
Format, mount, and mount automatically at boot
Pretty standard, done this a lot by now.
mkfs.ext4 /dev/mapper/vgblock-lv01_data
mkdir /block
mount /dev/mapper/vgblock-lv01_data /block
Edit the following two files to unlock the drive and mount it at boot:
# /etc/crypttab
# ...
block UUID=<block disk UUID> /path/to/.keyfile luks
# /etc/fstab
# ...
/dev/mapper/vgblock-lv01_data /block ext4 defaults 1 2
Reboot and find out!
Final thoughts
Writing this a few days after deploying it, and it has served me very well thus far. No complaints, haven’t had any downtime.
I’ve also done several things that I’ve already written about at length (I ended up referencing my previous posts a lot, actually):
- Set up automatic encrypted backups to Backblaze B2 (script below)
- Made sure Docker waits until the block storage is mounted
- Configured it as a “local” server
- Set up Tailscale, locked public interfaces down completely
- Set up Docker and several containers
- Configured Nginx reverse proxy and generated wildcard certificate using DNS-01 challenge
- Added entries to my local DNS server to point to the new server
- Set up [[setting_up_protonmail_bridge_on_lan_server|Protonmail Bridge with
socatto port forward]] - Tried out Obsidian LiveSync and WebDAV share
I’ve got a few more things on the list - mainly a cloud document storage, maybe file storage, and transferring my bookmark manager + RSS reader. I also want to see if I can replicate to my local NAS over an SMB share - I don’t see it being that difficult.
Backups script
This is assuming rclone is set up as documented here.
#!/bin/bash
SOURCE_DIRS=("/block/docker" "/etc/nginx")
LOCAL_BUP_DIR=/block/bups/
DEST=b2-crypt:<bucket-name>/vps
for dir in ${SOURCE_DIRS[@]}; do
echo "Backing up $dir..." | /usr/bin/logger -t backups_script
BUPFILE=$LOCAL_BUP_DIR$(echo $dir | rev | cut -d\/ -f1 | rev).$(date -I).tar.gz
tar -czf $BUPFILE $dir 2>&1 | /usr/bin/logger -t backups_script
done
for dir in ${SOURCE_DIRS[@]}; do
echo "Removing stale backups for $dir..." | /usr/bin/logger -t backups_script
find $LOCAL_BUP_DIR -name "$(echo $dir | rev | cut -d\/ -f1 | rev).*.tar.gz" -mtime +7 -delete | /usr/bin/logger -t backups_script
done
rclone sync $LOCAL_BUP_DIR $DEST \
--fast-list \
| /usr/bin/logger -t backups_script
echo "Backups completed." | /usr/bin/logger -t backups_script
EOF