Provision NixOS Server

Workflow Overview

• Gather requirements from user

• Create Proxmox container

• Set up SSH access

• Create Colmena init configuration

• Deploy init config and update to static IP

• Copy infrastructure key for SOPS

• Configure application (if applicable)

• Deploy full configuration

Step 1: Gather Requirements

Ask user for:

• Hostname: Server name (e.g., woodpecker )

• Container ID: Proxmox container ID (e.g., 122 )

• Proxmox server: Target Proxmox host (e.g., thrall )

• Storage: local-lvm (fast) or cephpool1 (distributed, slower)

• Memory: RAM in MB (default: 4096 )

• Disk size: In GB (default: 100 )

• Application: What will run on this server

Verify soft-secrets exist for the new host (host.<hostname>.admin_ip_address , etc.) or ask user to create them.

Step 2: Create Proxmox Container

PROXMOX_SERVER=<server> HOSTNAME=<hostname> CONTAINER_ID=<id> STORAGE=local-lvm MEMORY=4096 DISK_SIZE_IN_GB=100

ssh $PROXMOX_SERVER "pct create $CONTAINER_ID
--arch amd64 local:vztmpl/nixos-system-x86_64-linux.tar.xz
--ostype unmanaged
--description nixos
--hostname $HOSTNAME
--net0 name=eth0,bridge=vmbr3,ip=dhcp,firewall=1
--storage $STORAGE
--memory $MEMORY
--rootfs $STORAGE:$DISK_SIZE_IN_GB
--unprivileged 1
--features nesting=1
--cmode console
--onboot 1
--start 1"

Timeout note: Use longer timeouts (5+ minutes) for cephpool1 storage.

Step 3: Set Up SSH Access

Fresh NixOS containers require full paths. Run via pct exec :

ssh $PROXMOX_SERVER "pct exec $CONTAINER_ID -- /run/current-system/sw/bin/bash -c '
mkdir -p ~/.ssh &&
curl -s https://github.com/fred-drake.keys > ~/.ssh/authorized_keys &&
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'"

Get DHCP IP address:

ssh $PROXMOX_SERVER "pct exec $CONTAINER_ID -- /run/current-system/sw/bin/ip addr show eth0 | grep 'inet '"

Step 4: Create Colmena Init Configuration

Create these files (see references/colmena-host-template.md and references/nixos-config-template.md):

• mkdir -p modules/nixos/host/<hostname>

• Create modules/nixos/host/<hostname>/configuration.nix

• Create colmena/hosts/<hostname>.nix

• Update colmena/default.nix with imports

Initial deploy config: Use DHCP IP and root user:

deployment = { targetHost = "<DHCP_IP>"; targetUser = "root"; };

Stage files and build:

git add colmena/hosts/<hostname>.nix modules/nixos/host/<hostname>/ colmena/default.nix colmena build --impure --on <hostname>-init

Step 5: Deploy Init and Update IP

Deploy (will hang when network restarts due to IP change):

colmena apply --impure --on <hostname>-init

Kill the hanging command, then update colmena/hosts/<hostname>.nix :

deployment = { targetHost = soft-secrets.host.<hostname>.admin_ip_address; targetUser = "default"; };

Verify with another deploy:

colmena apply --impure --on <hostname>-init

Step 6: Copy Infrastructure Key

Required for SOPS secret decryption:

ssh default@<NEW_IP> "mkdir -p ~/.ssh && chmod 700 ~/.ssh" scp /.ssh/id_infrastructure default@<NEW_IP>:/id_infrastructure ssh default@<NEW_IP> "chmod 600 ~/id_infrastructure"

Age public key (for .sops.yaml): age1rnarwmx5yqfhr3hxvnnw2rxg3xytjea7dhtg00h72t26dn6csdxqvsryg5

If secrets fail to decrypt, user needs to add this key to .sops.yaml and run sops updatekeys on the secret files.

Step 7: Configure Application

See references/app-templates.md for patterns.

Add Container Images

• Edit apps/fetcher/containers.toml

• Run just update-container-digests

• Stage: git add apps/fetcher/containers.toml apps/fetcher/containers-sha.nix

Create Application Config

Create apps/<appname>.nix with:

• Nginx proxy with SSL (if web-facing)

• PostgreSQL container (if database needed)

• Application container(s)

• tmpfiles rules for data directories

Create Secrets Config

Create modules/secrets/<hostname>.nix referencing SOPS files.

User must create SOPS files in secrets repo with:

• postgresql-env.sops (POSTGRES_PASSWORD, POSTGRES_USER, POSTGRES_DB)

• <appname>-env.sops (app-specific secrets)

Update Colmena Full Config

In colmena/hosts/<hostname>.nix , add to full configuration imports:

../../modules/secrets/<hostname>.nix ../../apps/<appname>.nix

Step 8: Deploy Full Configuration

just update-secrets # Get latest secrets git add <all-new-files> colmena apply --impure --on <hostname>

Common Issues

SOPS decrypt fails: Age key not in .sops.yaml - user must add key and re-encrypt

Nginx duplicate directive: Don't add proxy_http_version when using proxyWebsockets = true

PostgreSQL 18 fails: Mount at /var/lib/postgresql not /var/lib/postgresql/data

Container can't reach postgres: Use 0.0.0.0:5432:5432 for port binding, host.containers.internal in connection string