Sekiro infrastructure

This repository manages the Phase 1 NixOS service guest on the Debian server sekiro. It is written for a human operator: Ansible owns the Debian and systemd-nspawn boundary, while Nix owns the services inside the guest.

The deployed guest is sekiro-nixos at 10.231.0.2 behind the host bridge nixbr0 (10.231.0.1/24). It has outbound NAT but no public ingress. Debian Caddy is still the public edge and proxies only Grafana and Prometheus into the guest.

What is running where

Workload	Current owner	Rollback retained on Debian
Fishnet	NixOS guest	Yes, masked Quadlet and original state
Grafana	NixOS guest	Yes, masked unit and /var/lib/grafana
Prometheus	NixOS guest	Yes, masked unit and /var/lib/prometheus
VictoriaMetrics	NixOS guest	Yes, masked unit and /var/lib/victoria-metrics
node_exporter	Debian and NixOS guest	Both intentionally active
RPZ metrics parser	NixOS guest	Original Debian unit retained
Caddy, PostgreSQL, Redis, SSH, DNS, Tailscale, Factorio, Tangled, backups	Debian	Not adopted in Phase 1

Prometheus identifies the exporters separately: sekiro-host is scraped at 10.231.0.1:9100, and sekiro-nixos is scraped at 127.0.0.1:9100 inside the guest.

Grafana configuration

Grafana's persistent SQLite database at /var/lib/grafana/grafana.db contains user-managed state such as the Prometheus data source. File-provisioned dashboards are separate, tracked Nix configuration under nix/modules/services/grafana/provisioning/dashboards/. Before Grafana starts, its service materializes those tracked sources as regular files at /var/lib/grafana/provisioning/dashboards/, the configured provisioning root. Update the tracked JSON, then deploy the guest; do not edit that generated guest path directly.

Operator setup

Use the pinned repository toolchain, not a globally installed Ansible:

UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-galaxy \
  collection install -r ansible/collections/requirements.yml

Routine access uses ~/.ssh/sekiro-deploy. The inventory already selects that key and the sekiro-deploy account. Direct guest access goes through Debian:

ssh -o IdentitiesOnly=yes -i ~/.ssh/sekiro-deploy \
  -o 'ProxyCommand=ssh -o IdentitiesOnly=yes -i ~/.ssh/sekiro-deploy -W %h:%p sekiro-deploy@140.245.245.33' \
  sekiro-deploy@10.231.0.2

Create the ignored local file secrets/sekiro.ini with mode 0600:

[fishnet]
key =

[grafana]
secret_key =

Fill values from the password manager. A blank Fishnet value preserves the already-provisioned runtime key. The Grafana key must remain the same as the key that wrote the migrated database. To import the effective Debian value without displaying it, run:

UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-playbook \
  ansible/playbooks/import-grafana-secret.yml

Ansible materializes supplied values as root-only files under /srv/sekiro-nixos/secrets/. They never enter the Nix store.

Validate and deploy

Run local validation first:

UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen yamllint ansible
UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-lint ansible
UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-playbook \
  --syntax-check ansible/playbooks/site.yml

The normal sequence is:

# Manage only the Debian/nspawn boundary.
UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-playbook \
  ansible/playbooks/site.yml

# Build, copy, validate, and activate the NixOS generation.
UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-playbook \
  ansible/playbooks/deploy-guest.yml

# These migration playbooks are safe to rerun after their first cutover.
UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-playbook \
  ansible/playbooks/migrate-fishnet.yml
UV_CACHE_DIR=/private/tmp/sekiro-uv-cache uv run --frozen ansible-playbook \
  ansible/playbooks/migrate-monitoring.yml

The deployment does not build inside the service guest. A rootless Podman Quadlet runs the Nix builder with sandboxing enabled, a read-only container root, no workload state or service secrets, NoNewPrivileges=true, and only the capabilities proven necessary for multi-user Nix builds. The resulting closure is copied into the guest, tested with dry-activate, test, and the private smoke endpoint, recorded as a Nix generation, and then switched.

Health checks

On Debian:

sudo machinectl status sekiro-nixos
networkctl status nixbr0
curl --fail http://10.231.0.2:18080/
curl --fail http://10.231.0.2:3000/api/health
curl --fail http://10.231.0.2:9090/-/ready
curl --fail http://10.231.0.2:8428/health
curl --fail https://grafana.rishav.io/api/health
curl --fail https://prom.rishav.io/-/ready

After a guest restart, the smoke endpoint becomes ready before the stateful monitoring services. Wait for all three private service endpoints before interpreting a temporary public 502 as a fault.

LAN DNS for Tailscale services

grafana.rishav.io, prom.rishav.io, and related private services ultimately resolve to Sekiro's 100.120.57.16 Tailscale address. OpenWrt dnsmasq keeps DNS rebind protection enabled, with a narrow exception for this domain:

ssh openwrt "uci show dhcp.@dnsmasq[0] | grep rebind"

The expected value includes both the pre-existing plex.direct exception and rishav.io. To restore the latter if it is lost:

ssh openwrt 'uci add_list dhcp.@dnsmasq[0].rebind_domain="rishav.io"; \
  uci commit dhcp; /etc/init.d/dnsmasq reload'

Do not disable rebind protection globally. Verify the router, macOS resolver, and service in that order:

ssh openwrt 'nslookup grafana.rishav.io 127.0.0.1'
dscacheutil -q host -a name grafana.rishav.io
curl --fail https://grafana.rishav.io/api/health

Roll back a Nix generation

List and select the desired recorded generation from Debian:

leader=$(sudo machinectl show sekiro-nixos --property=Leader --value)
sudo nsenter --target "$leader" --mount --uts --ipc --net --pid --root --wd \
  /run/current-system/sw/bin/nix-env \
  --profile /nix/var/nix/profiles/system --list-generations

sudo nsenter --target "$leader" --mount --uts --ipc --net --pid --root --wd \
  /run/current-system/sw/bin/nix-env \
  --profile /nix/var/nix/profiles/system --switch-generation GENERATION
sudo nsenter --target "$leader" --mount --uts --ipc --net --pid --root --wd \
  /nix/var/nix/profiles/system/bin/switch-to-configuration switch

Verify the private and public health checks afterward. The machine root's /sbin/init follows /nix/var/nix/profiles/system/init, so the selected generation also remains the nspawn boot default.

Restore Debian monitoring

The tested rollback order is:

1. Stop guest grafana, prometheus, and victoriametrics.
2. Unmask and start Debian grafana-server, prometheus, and victoria-metrics.
3. Wait for 127.0.0.1:3003/api/health, 127.0.0.1:9090/-/ready, and the retained VictoriaMetrics endpoint to become ready.
4. Change only the two Caddy upstreams back to 127.0.0.1:3003 and 127.0.0.1:9090.
5. Run sudo caddy validate --config /etc/caddy/Caddyfile, reload Caddy, and verify both public endpoints.

To return to NixOS, start the three guest services and wait for their direct health endpoints, restore the Caddy upstreams to 10.231.0.2:3000 and 10.231.0.2:9090, validate and reload Caddy, then stop and mask the retained Debian units. Do not recopy Debian data: the marker /srv/sekiro-nixos/data/.monitoring-migrated-v1 protects live guest state.

Fishnet rollback similarly stops guest Fishnet, unmasks and starts the retained Debian fishnet.service, and leaves the guest data isolated.

Known Debian reboot issue

A full-host reboot exposed an existing ordering race outside this repository's Phase 1 ownership. Rootful Podman creates podman0; tangled-loopback adds 11.0.0.1; Redis binds 10.88.0.1; and Caddy binds the Tangled address. If those units run before podman0 exists, Tangled, Redis, and Caddy can fail even though the NixOS guest boots correctly.

After confirming podman0 exists, the observed recovery order is:

sudo systemctl start tangled-loopback.service
sudo systemctl start redis-server.service
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl start caddy.service

Their unit ordering is deliberately not changed here because these workloads are excluded from Phase 1.