# Setting up a remote Borg repository on HexOS (TrueNAS SCALE)

Target: a Borg repository target on the HexOS box (built on TrueNAS SCALE), reachable
over the WireGuard split-tunnel, for backing up `/srv/nas` from `vortex-n`.

HexOS has no native rsync/SFTP/backup-receiver UI, and the base OS doesn't allow
persistent package installs — everything here runs as a container on top of TrueNAS
SCALE's Apps system. SSH isn't exposed directly in the HexOS UI either — it may need
to be reached through the underlying TrueNAS admin panel (look for an
"Advanced"/"Developer" toggle in HexOS settings if the TrueNAS UI isn't obviously
available).

## Option A (preferred): BorgWarehouse

[BorgWarehouse](https://borgwarehouse.com/) is a web UI purpose-built for running a
central Borg repository server. It provisions the dedicated system user, SSH key, and
storage quota automatically per repository, and hands back a ready-to-paste `borg`
client command — no hand-editing `authorized_keys` with forced-command strings.

It ships as a **Community-train TrueNAS SCALE app**
(`apps.truenas.com/catalog/borgwarehouse_community/`). Since HexOS runs on TrueNAS
SCALE, it should install the same way, provided HexOS's Apps screen still exposes
access to the Community catalog train — check whether it needs to be added as a
catalog first, since HexOS's simplified Apps UI may hide this.

### 1. Add/confirm the Community catalog is available

In the Apps section, check for a catalog/train selector. If Community isn't listed,
look for an "Add Catalog" option (standard TrueNAS SCALE terminology) — this may or
may not be exposed in HexOS's simplified UI.

### 2. Plan the datasets before installing

The app needs separate storage locations for:
- App config
- SSH keys
- SSH host keys
- Borg repository data (this is the one that needs real capacity — size for the full
  `/srv/nas` mount, currently ~350GB and growing)
- Logs
- Temp files

Each can be an auto-created (`ixVolume`) or an existing host path — pointing the repo
data storage at an explicit dataset (e.g. `<pool>/backups/nas`) rather than an
auto-created one keeps it easy to find/monitor later.

### 3. Install the app

- WebUI port and SSH port are both configurable at install (defaults `30406`/`30407`
  in the catalog listing) — only expose the SSH port on the interface reachable
  through the WireGuard split-tunnel, not on any publicly forwarded port.
- Main container runs as root (needed to provision system users/SSH keys per
  repository dynamically) with a set of Linux capabilities
  (CHOWN/DAC_OVERRIDE/FOWNER/SETUID/SETGID/SYS_CHROOT) — inherent to what it
  automates, not a red flag by itself.
- Complete the first-run setup wizard (creates the initial admin account — no default
  credentials to clean up afterward).

### 4. Create the repository through the WebUI

- Add a new repository, pointing it at the repo-data storage location from step 2.
- Set a storage quota matching planned capacity.
- **Enable append-only mode** — protects the repo from a compromised or malicious
  client being able to delete/overwrite existing archives, only add new ones. This
  is the same protection the manual recipe below achieves via
  `borg serve --append-only`, just as a UI toggle here.
- Copy the generated ready-to-paste SSH client command — this becomes the
  `BORG_REPO`/`BORG_RSH` setup on the `vortex-n` side.

### 5. Verify from the Arch box

Once the WireGuard interface is up and routing to the HexOS box's LAN, the
repository-creation screen's client command should work directly:

```bash
borg init -e repokey-blake2 ssh://<generated-user>@<tunnel-ip>:<ssh-port>/./<repo-path>
```

## Option B (fallback): manual `linuxserver/openssh-server` container

If the Community app catalog isn't reachable on HexOS, this achieves the same result
by hand.

### 1. Create datasets

```
<pool>/applications/borg/config
<pool>/applications/borg/custom-container-init-scripts
<pool>/backups/nas
```

### 2. Create a dedicated system user for the account holding the repo

Credentials → Local Users → Add:
- Username: `borg`
- Disable password: yes
- Home directory: the `<pool>/backups/nas` dataset from step 1
- Shell: `nologin`
- Note the UID it gets assigned (defaults around 3000) — needed for the container's
  PUID below.

### 3. Write the container init script that installs Borg

At `<pool>/applications/borg/custom-container-init-scripts/install-borg.sh`:

```bash
echo "**** installing borgbackup ****"
apk add --no-cache borgbackup
```

Set it root-owned (`chown root:root`) so the container's init system will actually
run it.

### 4. Launch an SSH server container

Apps → Discover Apps → look for a "Custom App" / "Launch Docker Image" option (native
to TrueNAS SCALE's Apps screen — check if HexOS's simplified Apps UI still exposes it;
if not, Dockge or Portainer deployed as a custom app is the next fallback). Image:
`lscr.io/linuxserver/openssh-server:latest`.

**Volumes:**

| Host path | Container path |
|---|---|
| `<pool>/applications/borg/config` | `/config` |
| `<pool>/applications/borg/custom-container-init-scripts` (read-only) | `/custom-cont-init.d` |
| `<pool>/backups/nas` | `/var/local/borg` |

**Environment:**

```
PUID=<the borg user's UID from step 2>
PGID=<matching GID>
USER_NAME=borg
PASSWORD_ACCESS=false
SUDO_ACCESS=false
LISTEN_PORT=22
```

Only expose this container's port on the interface reachable through the WireGuard
split-tunnel — not on any publicly forwarded port.

### 5. Generate a dedicated key on the Arch box for this connection

Don't reuse an existing personal key — a purpose-scoped one makes revocation trivial
later:

```bash
ssh-keygen -t ed25519 -f /root/.ssh/borg_hexos_ed25519 -N "" -C "borg-remote@vortex-n"
```

### 6. Lock the authorized_keys entry to only ever run `borg serve`

In `<pool>/applications/borg/config/.ssh/authorized_keys` (create the `.ssh` dir with
`700`, the file with `600`, owned by the borg UID/GID):

```
command="borg serve --restrict-to-path /var/local/borg/nas --append-only",restrict ssh-ed25519 AAAA... borg-remote@vortex-n
```

- `restrict` blocks port-forwarding, X11, agent-forwarding, and PTY allocation — the
  connection can do nothing but speak the Borg protocol.
- `--append-only` means even a compromised local host can create new archives but
  can't delete or overwrite existing ones remotely.
- `--restrict-to-path` confines the session to that one directory, so this key can't
  wander the rest of the pool.

### 7. Create the repo directory the key is scoped to

```bash
sudo -u borg mkdir -p /var/local/borg/nas   # from inside the container, or the
                                              # equivalent host path
```

### 8. Verify reachability from the Arch box

```bash
ssh -i /root/.ssh/borg_hexos_ed25519 borg@<tunnel-ip> -p <LISTEN_PORT>
```

This should hang waiting for the Borg protocol handshake (not drop to a shell) —
Ctrl-C out, that's the expected/correct behavior confirming the forced command is
active. A real check:

```bash
export BORG_REPO="ssh://borg@<tunnel-ip>:<LISTEN_PORT>/var/local/borg/nas"
export BORG_RSH="ssh -i /root/.ssh/borg_hexos_ed25519"
borg init -e repokey-blake2 "$BORG_REPO"
```

## Next steps (not done yet)

Once either option succeeds, the remaining work is on the `vortex-n` side:

- Wire up `nas-remote-sync.sh` (or extend `borg-backup.sh` with a second `BORG_REPO`
  target) covering `/srv/nas` as a whole.
- Guard it with a shared `flock` against the existing monthly `borg-backup.service`
  (same lock file, e.g. `/run/lock/nas-backup.lock`) so the two can never run
  concurrently against the local repo/filesystem.
- Drop in the actual WireGuard config once delivered, and bring the tunnel up only
  for the sync window (`wg-quick up`/`down` wrapped around the run) rather than
  leaving it always-on.

See `project_terramaster_nas_buildout` memory and `~/nas/README.md` for the broader
NAS buildout context this fits into.