Provisioning a NixOS vps cheaply

I daily-drive a macbook. Of course it’s configured with nix-darwin ;) but sometimes it’s nice to be able to experiment with modules exclusive to NixOS. Maybe even have some always-on service running to use like a DNS server or a self-hosted wiki page.

Typically, people achieve this by using a cheap vps, like a DigitalOcean Droplet for example. They start at 512MB of ram and 1 vCPU for $4 per month. But unfortunately, this isn’t actually enough ram to run commands like nixos-rebuild.

So should you upgrade your specs? Yes, but not as much as you probably think. With just 1GB of ram, you can build a new NixOS configuration without the Droplet crashing. The trick? Using swap memory to get an extra few GBs of "fake" memory. I personally use 3GB of swap for my NixOS Droplets.

Why not build locally and push a closure?

This is actually quite hard when all you have is an aarch64 laptop and an x86_64 vps. As you can imagine, you'd need to cross-compile between different architectures, and I do NOT really want to mess with that right now 😔.

The bootstrap problem

This is one of those situations where the first time you do it, you'll run into a pretty hard problem. First of all, how do you create a Droplet that runs NixOS? It turns out that DigitalOcean lets you upload a custom OS image that you can use to provision your Droplet, and there are Nixpkgs helper functions that create an entire DigitalOcean image pretty easily. So you copy-paste this snippet, build, upload the image, provision a Droplet, and grab the public IP so you can use a tool like deploy-rs to push new configurations to the Droplet. Except that wait, in order to push a configuration that creates a 3GB swap, you already needed to have 3GB of swap available. Huh.

The solution isn't too hard: when creating the image, make sure to use a configuration that configures the swap already.

Now you know how to cheap out on a vps for playing with NixOS too :>. In practice, I ended up using these Droplets to become part of my K3s Kubernetes cluster, in which 1GB (even with swap!) wasn't enough ram for the control-plane node to not corrupt etcd. D: