Preparing for Coder 2.38 and the enablement of NATS

Last updated: September 24, 2026

Background

At large scale, Coder's default PostgreSQL-based pub/sub (via pg_notify) can become a bottleneck, particularly for tailnet coordinator traffic (e.g., tailnet_ready_for_handshake, peer updates), which can cause excessive load on the PostgreSQL database. To reduce the impact of this Coder is moving pub/sub functionality out of PostgreSQL and back into Coder by implementing NATS.

NATS PubSub is an embedded, in-process NATS server that replaces PostgreSQL pub/sub for tailnet traffic, removing it from pg_notify entirely and reducing PostgreSQL pub/sub call volume by approximately 80–90%. NATS communication is automatically secured using mTLS (which Coder self manages).

NATS PubSub becomes enabled by default in Coder 2.38.0. In versions 2.37.x and earlier, it is opt-in via the nats_pubsub experiment flag. As of Coder 2.38.0 if the below prerequisites are not met Coder will automatically fall back to using PostgreSQL for pub/sub (i.e. the existing behaviour).

Prerequisites

Before upgrading to Coder 2.38.0 (or later) with the desire to use the new NATS functionality, ensure the following prerequisites are met:

Port 6222/tcp must be open between all Coder control plane replicas. This is only required if you run more than one replica (HA). This port is used exclusively for inter-replica NATS cluster communication and does not need to be accessible from workspaces or workspace proxies.
1. Communication is between Coder control plane replicas only. Workspaces and Workspace Proxies do not need connectivity to/from the Coder control place replicas on port 6222, they continue to only require access to the Coder control plane via CODER_ACCESS_URL per references below [1] [2] [3].

CODER_CLUSTER_HOST must be set on each replica to its own routable IP address. This allows other replicas to dial it for NATS cluster mesh formation. Upon startup Coder will first validate if CODER_CLUSTER_HOST is populated - if unset Coder falls back to the DERP relay URL hostname. If neither is available, NATS will not cluster and Coder will fall back to PostgreSQL pub/sub.
1. If your Coder deployment has the built in DERP server disabled you must populate CODER_CLUSTER_HOST if you wish to use NATS.

For Kubernetes: The Helm chart (version 2.37.3+) automatically populates CODER_CLUSTER_HOST using status.podIP. Confirm your replicas are populating this env var using:
```
kubectl exec <coder-pod> -- bash -c 'echo $CODER_CLUSTER_HOST'
```

For ECS or other non-Kubernetes environments: Set the variable in your entrypoint script before starting the Coder server. For example, on ECS:

export CODER_CLUSTER_HOST=$(curl -s "${ECS_CONTAINER_METADATA_URI_V4}/task" | jq -r '.Containers[0].Networks[0].IPv4Addresses[0]')
# then run: coder server ...

Rollout

Once prerequisites are confirmed, follow these steps to enable and validate NATS PubSub:

Where possible we recommend starting with a dev or staging environment.
Confirm CODER_CLUSTER_HOST is correctly set on each replica (see prerequisites above). You can cross-check the value against the DERP relay URL returned by:
```
curl -sH "Coder-Session-Token: <your-token>" <CODER_URL>/api/v2/deployment/config | jq '.config.derp'
```
The IP in CODER_CLUSTER_HOST should be the replica's own routable IP, not the DERP relay URL.
On 2.37.x, opt in by setting the experiment flag:
```
CODER_EXPERIMENTS=nats_pubsub
```
On 2.38.0+, NATS is enabled by default. To opt out if needed:
```
CODER_EXPERIMENTS=no_nats_pubsub
```
Ensure replicas can reach each other on port 6222/tcp (check Kubernetes NetworkPolicies, AWS Security Groups, or equivalent firewall rules for east-west traffic between control plane pods/instances).
Upgrade the version of Coder to 2.37.3, 2.38.0 or any newer version.

If debug logging is enabled in Coder log entries such as the below can be used to confirm Coder is operating using NATS:

2026-09-23 22:39:01.783 [info]  nats_pubsub: embedded nats server started  client_url=nats://127.0.0.1:36113
2026-09-23 22:39:01.785 [debu]  nats_pubsub: subscribing on nats  event=pubsub_watchdog
2026-09-23 22:39:02.321 [debu]  coderd.nats_ca_cache.nats_ca_signing_keycache: created new key cache  feature=nats_ca
2026-09-23 22:39:02.321 [debu]  coderd.nats_ca_cache.nats_ca_signing_keycache: fetching crypto keys  feature=nats_ca
2026-09-23 22:39:02.323 [debu]  coderd.nats_ca_cache.nats_ca_signing_keycache: crypto key fetch complete  feature=nats_ca
2026-09-23 22:39:02.347 [debu]  nats_pubsub: subscribing on nats  event=provisioner_job_posted
2026-09-23 22:39:03.396 [debu]  nats_pubsub: subscribing on nats  event=chat:config_change
2026-09-23 22:39:03.396 [debu]  nats_pubsub: subscribing on nats  event=ai_providers_changed
2026-09-23 22:39:03.396 [debu]  nats_pubsub: subscribing on nats  event=chat:ownership
2026-09-23 22:39:03.398 [debu]  nats_pubsub: subscribing on nats  event=workspace_build_orchestrations:wake
2026-09-23 22:39:03.443 [debu]  nats_pubsub: subscribing on nats  event=tailnet_coordinator_heartbeat
2026-09-23 22:39:03.443 [debu]  nats_pubsub: subscribing on nats  event=tailnet_peer_update
2026-09-23 22:39:03.444 [debu]  nats_pubsub: subscribing on nats  event=tailnet_tunnel_update
2026-09-23 22:39:03.444 [debu]  nats_pubsub: subscribing on nats  event=tailnet_ready_for_handshake
2026-09-23 22:39:03.444 [info]  nats_pubsub.cluster_tls: nats cluster mTLS enabled
2026-09-23 22:39:03.444 [debu]  coderd: nats port updated  port=6222
2026-09-23 22:39:08.534 [debu]  coderd.nats_ca_cache.nats_ca_signing_keycache: request for key  feature=nats_ca  sequence=-1
2026-09-23 22:39:08.535 [debu]  nats_pubsub.cluster_tls: minted nats cluster leaf  ca_sequence=1

Alternatively / additionally, we suggest capturing the relevant NATS Prometheus metrics emitted by Coder. Search for nats on the Coder Prometheus metrics reference page. coder_nats_pubsub_connected showing a value of 1 for each Coder replica indicates NATS is being used.

If NATS is unable to be used by Coder an error level log event will be written containing the following:

embedded NATS pubsub is enabled but this replica has no cluster host;
set --cluster-host (CODER_CLUSTER_HOST) to this replica's routable IP address, or configure the DERP relay URL; falling back to PostgreSQL pubsub

Usage

Once NATS PubSub is running, use the following to validate and monitor the deployment:

Verify NATS cluster formation: Watch coder_nats_pubsub_connected in Prometheus. If routes between replicas fail to form, each replica will only see its own events — there will be no error, but peers on different replicas will be unable to discover each other, causing connection failures for a subset of users.

Compare PostgreSQL load: After enabling NATS, compare the volume of pg_notify-related calls and overall database load. You should see a significant reduction in pub/sub-related PostgreSQL calls. Note that PostgreSQL pub/sub is still used for initial bootstrapping (replica-sync and license events), so it does not disappear entirely.

Preparing for Coder 2.38 and the enablement of NATS

Background

Prerequisites

Rollout

Usage

References: