High availability

High availability matters most for the gateway, because it sits inline in front of AI traffic. Analytics can tolerate brief interruptions: sensors buffer events locally and replay them. Plan for HA in that order: the gateway first, then the event bus, then the data stores.

Gateway

• Run at least three replicas across zones. Use topologySpreadConstraints (set by default in the chart) and a PodDisruptionBudget with minAvailable: 2.

• Put the forward proxy behind an L4 load balancer. Proxy clients and PAC files should list a second gateway VIP as a fallback:

  return "PROXY ai-gw-a.example.com:3128; PROXY ai-gw-b.example.com:3128; DIRECT";

  Leave DIRECT out of the fallback list if your policy requires AI traffic to be inspected (fail-closed at the network level).

• Gateways hold policy and detector state in memory and refresh it from the API every 30 seconds. If the API is down, they keep enforcing the last known policy.

Fail-open vs fail-closed

Choose per policy rule:

rules:
  - name: block-phi-to-public-ai
    match: { data_class: [phi] }
    action: block
    on_error: fail_closed   # if detection cannot complete, block
  - name: log-everything-else
    action: log
    on_error: fail_open     # never interrupt users because of a monitoring fault

Event pipeline

• NATS JetStream runs as a three-node cluster with file storage, replication factor 3.
• The browser extension and endpoint agent buffer up to 72 hours of events on the device, encrypted, if they can’t reach ingestion.
• Gateways buffer to a local volume (gateway.buffer.size, default 5 GiB) before dropping metadata. Enforcement never depends on the buffer.

Data stores

Store	HA approach	RPO / RTO guidance
PostgreSQL	Managed HA (RDS Multi-AZ, Azure Flexible Server HA) or CloudNativePG with 3 instances	Seconds / minutes
ClickHouse	ReplicatedMergeTree with ClickHouse Keeper; 2 replicas × N shards	Minutes / minutes
Object storage	Provider durability; enable versioning and cross-region replication if required	Provider dependent

Backups

promptglass admin backup create --include config,identity --dest s3://acme-pg-backups/

Back up PostgreSQL with your normal tooling. ClickHouse can be rebuilt from object storage exports if you enable export.events (see REST API), but most teams snapshot it with clickhouse-backup.

Disaster recovery

For a warm standby region, install a second release with global.mode=standby, replicate PostgreSQL and object storage, and point DNS for the gateway VIPs at the standby when you fail over. The standby’s gateways enforce policies as soon as they come up.