Backup and restore

What an MSSP backs up, how often, and how to restore. SocTalk holds three layers of state; each has its own backup and restore path.

This page expands on Daily Operations, Database restore, which is the same procedure documented at runbook level. Use this page to plan the strategy; use ops for the keystrokes.

What to back up

1. Postgres (the control plane)

soctalk-system-postgres-0 holds:

• Tenant rows + lifecycle events
• Users, sessions, roles
• Investigations, cases, runs, proposals
• Settings (LLM, integrations, branding)
• Append-only audit_log and event-sourced case_events
• Outbox rows pending executor consumption

Loss tolerance: zero. A lost Postgres = lost audit history, no recoverable investigations.

2. Kubernetes Secrets in soctalk-system

Secret (chart-rendered name)	What's in it
soctalk-system-llm-api-key	LLM provider API key (install-wide default)
soctalk-system-bootstrap-admin	Initial admin email + password (if install.bootstrapAdmin.password set in values)
soctalk-system-jwt-signing-key	Session token signing key
soctalk-system-adapter-signing-key	Adapter-token signing key
soctalk-system-postgres-admin-creds	Postgres soctalk_admin (migrations) creds
soctalk-system-postgres-app-creds	Postgres soctalk_app (runtime) creds
soctalk-system-postgres-mssp-creds	Postgres soctalk_mssp (cross-tenant queries) creds
soctalk-slack-creds	Slack tokens (env-provided; not chart-rendered)
soctalk-thehive-creds	TheHive API key (env-provided)
soctalk-cortex-creds	Cortex API key (env-provided)

A regenerated set of Secrets is recoverable, but in-flight sessions break and integration creds need re-pasting.

3. Per-tenant PVCs

For each tenant-<slug> namespace:

PVC	What's in it
wazuh-indexer-data	All Wazuh alert and event history
wazuh-manager-data	Wazuh agent registrations + manager state
cortex-data	Cortex Elasticsearch (if Cortex enabled)
thehive-data	TheHive Cassandra (if TheHive enabled)

poc-profile tenants use local-path, which has no real persistence guarantee: a node restart can lose data. persistent-profile tenants use whatever StorageClass the install marks default; back up according to that provisioner's documentation.

Cadence

Layer	Suggested cadence	Retention
Postgres logical backup (pg_dump)	daily	30 days
Postgres WAL archiving	continuous	7 days
Kubernetes Secrets snapshot	weekly + on every rotation	90 days
Per-tenant PVCs	matches your customer SLA (typically daily for compliance work)	per-contract

Compliance customers (PCI, HIPAA, SOC 2) often require longer retention. Treat the above as the floor.

Postgres backup

pg_dump (logical)

Runs against the live database, no downtime. Slower restore than physical backup but compresses well and is portable.

kubectl -n soctalk-system exec soctalk-system-postgres-0 -- \
  pg_dump -U soctalk_app -d soctalk -Fc -Z 9 \
  > soctalk-$(date +%Y%m%d).pgdump

Pipe to your usual offsite store (S3, GCS, Azure Blob).

WAL archiving (point-in-time)

Not wired through the chart in this release. The soctalk-system chart does not expose a postgres.archiveCommand value, so PITR requires a Postgres deployment outside the chart's bundled StatefulSet. Two paths:

1. Run Postgres externally (managed RDS / Cloud SQL / Azure Database for PostgreSQL). Configure WAL archiving / PITR per the provider's docs. Pointing the chart at an external Postgres is not wired through values in V1: the chart hard-codes the bundled StatefulSet's connection details into the role-creds Secrets. Today this means either running your own helm overlay that patches the API Deployment's DATABASE_URL env, or modifying soctalk-system-postgres-app-creds / -mssp-creds / -admin-creds after install. A postgres.external values knob is on the roadmap.
2. Sidecar archiver in your own helm overlay (e.g., spilo or wal-g as a sidecar). Out of scope for the chart; runs as a separate Deployment that streams WAL to object storage.

Either way, the SocTalk side is unchanged, the data plane treats Postgres as an external dependency. Wiring a chart-side archiveCommand is tracked for a future release.

Restore (Postgres)

See the runbook. Summary:

1. Scale API to zero so nothing is writing (the V1 chart bundles the orchestrator into the API pod, one Deployment).
2. pg_restore the dump (clean DB first).
3. If using WAL: replay WAL to the desired point-in-time.
4. Scale API back up.

After restore, the API pod (which embeds the orchestrator in the V1 chart) may need a kick to re-pick up pending runs:

kubectl -n soctalk-system rollout restart deploy soctalk-system-api

Secrets backup

K8s Secrets are tedious to back up safely because of the secret material. Two patterns:

Sealed Secrets (recommended)

Install Bitnami sealed-secrets once per cluster. Convert your Secrets to SealedSecret resources; commit those to git. The cluster's controller decrypts them at install time. Loss of a Secret is recoverable from git.

Velero with restic / kopia

Velero backs up Kubernetes resources (including Secrets) plus PVCs to object storage. Use the in-tree CSI snapshotter for PVCs and standard resource backup for Secrets.

velero backup create soctalk-system-daily \
  --include-namespaces soctalk-system \
  --snapshot-volumes \
  --schedule "0 2 * * *"

Per-tenant PVC backup

persistent-profile tenants use real StorageClass; use that provisioner's snapshot tools:

• Longhorn: built-in scheduled backups to S3
• Rook/Ceph: RBD snapshots or cephfs-mirror
• CSI cloud volumes (EBS/Persistent Disk/Azure Disk): native snapshot APIs

For Velero users, velero backup create tenant-<slug>-daily --include-namespaces tenant-<slug> --snapshot-volumes covers both the PVCs and the K8s objects in one go.

Per-tenant restore

1. Decommission the existing tenant (if any), this deletes the namespace.
2. Restore the PVCs to a fresh namespace from the snapshot.
3. Onboard a tenant with the same slug and profile via POST /api/mssp/tenants/onboard: provisioning is idempotent on the namespace, so the Helm install will adopt the restored PVCs.
4. Verify Wazuh sees existing agents (no re-enrollment needed if PVC restore was clean).

If only the data plane is corrupted (not the SocTalk control plane), the simpler path is helm rollback tenant-<slug> then restore the PVCs in-place.

Restore drill

Run a restore drill quarterly. Pick a non-prod cluster or a temporarily-quiesced tenant. Time-box at 4 h. Document what failed and update this page.

Common failures the drill catches:

• WAL gap (archiving fell behind during a node failure)
• Secrets that were rotated since the last backup
• StorageClass mismatch between cluster and snapshot
• Network policy blocking the restored pod from reaching the new Postgres

What's not covered here

• Cluster-wide disaster recovery (control plane node loss, etc.), that's Kubernetes operations, not SocTalk-specific. See your distribution's documentation.
• LLM provider credential recovery, out of scope; manage with your normal secret-rotation runbook.
• Customer-side endpoint backups, the customer's responsibility, not the MSSP's.