Configuration reference

The environment is authoritative. SenditCold never lets a web form overwrite a setting your environment owns, so there is no precedence to reason about and no file that silently rewrites itself behind you.

That rule has three consequences worth stating before the tables:

• Everything on this page is set in the environment of the running process. With Docker Compose that is the .env next to docker-compose.yml. With Kubernetes it is the pod spec or a secret. With a bare binary it is the shell or the systemd unit.
• Most resolved values are visible, read only, in the admin panel under Instance > Configuration > Environment (http://localhost:5174/configuration?tab=environment on a stock install). Each row shows the variable name, the value SenditCold actually resolved, where it came from (env, default, derived or unset), and whether changing it needs a restart. That page is how you answer "is my variable actually being picked up", without reading source.
• A handful of settings are stored in the database instead, because no environment variable owns them. They are listed in settings stored in the database and are the only settings editable from a browser.

Secrets are never returned by any API. The configuration page shows a sensitive key as set or unset plus a four character fingerprint of its value, which is enough to confirm that two services hold the same AUTH_SECRET without disclosing it to anyone.

Seeing what is actually set

Anything flagged on Instance health links back to the section of this page that explains the fix.

Deployment

In Gin release mode, access logs omit successful health probes and internal GET lookups, including an expected missing email-message mapping. Authentication failures, server errors, other missing resources and all mutations remain logged. Gin debug mode keeps every access log. Query strings and credential path parameters remain redacted in both modes; organization audit events are unaffected.

Secrets

Five values protect the whole instance. Compose ships a working default for each so a fresh clone boots with no configuration, and every one of those defaults is published in this repository, so they protect nothing.

Generate real ones before anyone else can reach the instance:

cat >> .env <<EOF
AUTH_SECRET=$(openssl rand -base64 32)
INTERNAL_API_TOKEN=$(openssl rand -hex 24)
SECRET_KEY_BASE=$(openssl rand -base64 64 | tr -d '\n')
KMS_LOCAL_MASTER_KEY=$(openssl rand -base64 32)
CREDENTIALS_ENCRYPTION_KEY=$(openssl rand -hex 32)
APP_ENV=prod
EOF

make gen-key prints a single fresh KMS_LOCAL_MASTER_KEY if that is all you need.

APP_ENV=prod goes last. It is what turns a published default from a logged warning into a refusal to start, so an instance that gets prod before the other five will not boot.

Values that must be identical across services, because each service reads its own copy:

Rotate instance keys

These settings are optional. An existing self-host keeps its current keys and behavior when they are unset. All list values are JSON arrays with at most four nonempty strings; [] means no additional keys. Keep them in your secret store or private .env.

Changing these settings requires restarting the affected service. Internal-token deadlines are checked on every request, so old tokens stop working at the cutoff without another restart. Tracking and forms overlap does not grant fleet broker access, and portal service overlap opens only the portal service's routes. JWT verification uses only AUTH_SECRET; encryption and unsubscribe read keys never authorize sessions.

For encryption rotation, first deploy a compatible release everywhere with the existing primary key and the replacement included in its read-key list. Confirm that all readers use that baseline before a later deployment activates the replacement primary and retains the old key for reads. Then reseal stored values with warmblyctl keys reseal: run it once without --apply to see the counts, then with --apply, and repeat until warmblyctl keys status shows nothing left under a read key. It is resumable and never overwrites a value the backend changed meanwhile. warmblyctl keys generate prints replacement values in the right format. The supported rollback floor during rotation is that compatible baseline release, using keys that read both generations. A binary rollback keeps the current compatible key configuration; reverting a primary key alone does not migrate data.

Retain old keys until stored ciphertext, retained events, pending jobs and the supported rollback window no longer need them. Keep the keys required by older backups for as long as those backups are retained. Organization DEKs and their KMS wrapping keys have a separate lifecycle; changing the instance credential key does not rotate them.

To rotate sessions while preserving existing MFA enrollments, unsubscribe links and published DNS proofs, first pin TWOFA_SECRET, UNSUBSCRIBE_SIGNING_SECRET and DOMAIN_PROOF_SECRET to their existing resolved values. Then change AUTH_SECRET and realtime's matching JWT_SECRET together. Users sign in again. Rotate the independent purpose keys separately; issued unsubscribe links remain valid for one year, so their read cutoff must cover the last previously issued link's expiry.

Consumer nodes hold CREDENTIALS_ENCRYPTION_KEY and CREDENTIALS_ENCRYPTION_READ_KEYS, and every node holds INTERNAL_API_TOKEN or NODE_BROKER_TOKEN, in node.env. Workers are never sent the credential keys. A node that keeps its env current receives each step of a rotation on its own; wait for warmblyctl fleet env-status to report every node current before activating a new credential key, and again before retiring an old key or token. A node still on the old values keeps sealing and authenticating with them. The Keys tab under Configuration in the admin panel shows the same three answers in one place: values still under a read key, open overlap windows and nodes that have not applied the current env. On AWS, the rotation workflow runs the whole procedure; see Rotate keys.

Seal mailbox credentials under workspace keys

MAILBOX_CREDENTIALS_SEALING chooses the key that seals newly stored mailbox credentials: SMTP and IMAP hosts, usernames and passwords, and OAuth access and refresh tokens.

Every release that reads this setting opens both formats, so the setting only decides what new writes use. With organization, the consumer also moves credentials already stored under the instance key onto workspace keys, in small resumable batches, and skips a mailbox whose stored value no key opens. With instance, the same consumer job seals any OAuth token still stored in plaintext under CREDENTIALS_ENCRYPTION_KEY. Workspace keys are cached in each process for five minutes and in Redis for fifteen, so a busy workspace costs one KMS decrypt per cache period, not one per mailbox.

Set it on the backend and the consumer, and only after every backend, consumer and worker runs a release that reads it. That release becomes the rollback floor: an older release cannot open a workspace-sealed credential. Keep CREDENTIALS_ENCRYPTION_KEY configured, because personal mailboxes with no workspace, mailboxes not yet moved and older backups still use it.

Addresses

Every emailed link (password reset, invitation, the first-run claim link) is built from APP_URL. Leave it unset and a self-hosted instance guesses the origin from CORS_ALLOW_ORIGINS or PUBLIC_HOST instead, and never falls back to the hosted service: a link nobody can open is a support ticket, while a working link to somebody else's dashboard carries a live reset token out of your deployment. Set it.

Customer domains behind Cloudflare for SaaS

Optional, and only for an instance whose TRACKING_DOMAIN is on a Cloudflare for SaaS zone. Cloudflare refuses a customer domain pointed at that host (error 1000 at a root, 1014 at a CNAME) until the domain is one of the zone's custom hostnames. With both values set, the backend registers a root redirect (and its www) as a custom hostname the moment it verifies and a verified tracking host within two minutes, rechecks a pending one every two minutes, and removes one a day after nothing on the instance serves it any more. While Cloudflare still needs proof of a domain, the redirect lists its _cf-custom-hostname record. The root record a redirect asks for also changes from the tracking host's addresses, which are Cloudflare's shared ones, to a CNAME to the tracking host. Unset, nothing is registered and the records are unchanged.

The origin has to route a request for any registered name to the tracking service with its Host header intact. On the AWS Fargate deployment that is tracking_serves_unmatched_hosts; see AWS Fargate.

Agency portal domains behind Cloudflare for SaaS

Optional, and only for an instance that serves agency portal domains from a Cloudflare for SaaS zone of their own, usually the zone PORTAL_DOMAIN is on, as SenditCold Cloud does. Agency portal domains CNAME to PORTAL_DOMAIN, not to the tracking host, so they get their own zone and their own settings. With both values set, the backend registers a portal domain as a custom hostname on that zone the moment it verifies (and every verified one within two minutes), rechecks a pending one every two minutes, and removes one a day after it stops being verified. While Cloudflare still needs proof of a domain, the agency brand's portal_domain lists the _cf-custom-hostname TXT record in edge_records. Workspace forms domains are never registered on this zone; they keep pointing at FORMS_DOMAIN. The tracking zone above is unaffected. Unset, no portal domain is registered anywhere, which is the self-host default: the instance issues their certificates on demand and its own portal service serves them.

Set CLOUDFLARE_SAAS_PORTAL_WORKER when the zone also holds other customer hostnames, such as tracking or redirect domains: one route for every custom hostname would send those to the portal too, and a route for each portal domain touches nothing else. A portal domain reads as registered only once its route to that Worker exists. A domain whose pattern already goes to another Worker is refused, and that route is never changed or removed. On a zone that holds portal domains only, leave it unset and route every custom hostname to the Worker once.

Moving a zone that already serves portal domains from one route for every hostname to a route for each: set the variable first and keep the old route for a day. A domain that is already active gets its own route at its next daily check, not at once, so removing the old route sooner would leave it unanswered until then.

On SenditCold Cloud the portal service is a Cloudflare Worker on that zone, configured by deploy/cloudflare/portal/wrangler.jsonc; its deployment is in deploy/cloudflare/README.md.

Network and proxy

Empty is correct for a directly exposed backend. Behind a reverse proxy it is not: with no trusted CIDR, SenditCold records the proxy's address as the client address, and the per IP login limiter, session records, audit rows and API key IP allowlists all read the wrong address. Set it to the CIDR your proxy connects from:

TRUSTED_PROXIES=10.0.0.0/8,172.16.0.0/12

TLS listeners

The backend, forms, tracking and realtime services listen on plain HTTP, and that stays the default everywhere: Compose, the installer and local development need nothing from this table. Set SERVE_TLS on a service and it also listens with TLS 1.2 or later on SERVE_TLS_PORT, for a load balancer or an internal caller that should reach it encrypted. The plain listener stays up beside it, so the container health check and any caller not yet moved keep working.

A service refuses to start on an incomplete combination rather than serving something other than what was asked for. The backend logs a line at boot when the certificate it serves expires within 30 days.

The forms and tracking services verify the backend's certificate whenever BACKEND_INTERNAL_URL is https://. BACKEND_INTERNAL_CA (PEM) or BACKEND_INTERNAL_CA_FILE replaces the system roots with exactly the certificates given, which is how they trust a backend serving a private certificate through SERVE_TLS=supplied. That certificate has to name the host in BACKEND_INTERNAL_URL.

Authentication

Full behavior, including what each registration mode does to the sign-up form, is on accounts and access.

Proxying PostHog

Content blockers drop requests to posthog.com, so analytics and error reporting quietly stop for a large share of visitors. The backend serves a reverse proxy at /ingest for this. Point each frontend at it:

Nothing else changes: the key stays the same, and the proxy forwards the payload untouched. It is on the backend rather than in each frontend because there are three of them, and a proxy copied three times drifts three ways.

The proxy forwards only what PostHog needs to read the request. Cookies and Authorization stay on this side, so a session for your instance is never handed to a third party, and the caller's address goes in X-Forwarded-For so visitors are geolocated rather than your server.

Advertising conversion measurement

Advertising measurement is hosted-only and off by default. A self-host needs no Meta account, Google Ads account, tag, token, or Google Cloud credential. The marketing site and /auth/register show a compact accept or reject control only when a public advertising ID is configured. They load no advertising script before acceptance, and authenticated /app/ routes never load the advertising tags.

The browser sends PageView on every marketing page and on /auth/register after consent. Meta also receives ViewContent on pricing, product, feature, use-case, and blog pages. The backend records CompleteRegistration, InitiateCheckout, and Purchase with stable source IDs. It writes them to an instance-local outbox and retries provider failures, so provider availability cannot fail registration, Checkout creation, or a Stripe webhook. Workspace exports exclude the outbox because it uses the source instance's operator credentials and must not replay on another instance.

Google conversions use the Data Manager API. Create one conversion action for each configured event, grant the credential access to the Ads account, and enable the Data Manager API in its Google Cloud project. Meta and Google credentials are independent. Google Tag Manager is not used.

Captcha

CAPTCHA_PROVIDER has no constant default. Unset, it resolves to turnstile when TURNSTILE_SECRET holds a value, whether from the environment or from AWS Secrets Manager, and to none when it does not, so configuring the secret is what turns captcha on and clearing it is what turns captcha off. The panel reports the resolved value with source derived.

Setting CAPTCHA_PROVIDER=turnstile explicitly while TURNSTILE_SECRET is empty is the one combination that breaks: every verification fails, which means nobody can sign in. Set the secret or set the provider back to none.

A Turnstile widget only renders on the hostnames listed for its site key in Cloudflare, and an agency's portal domain cannot be listed in advance. The dashboard on a portal domain therefore embeds a small page from the API's own host (/v1/auth/captcha-frame), which the API lets only a verified portal domain frame, and that page hands the token back to the portal page. The token is checked by the backend exactly as on the main address. For it to work, TURNSTILE_SITE_KEY must be set on the backend and the API's hostname must be on the site key's hostname list. Without them the portal sign-in asks people to sign in at the main address.

The native iOS app never renders Turnstile. It proves it is a genuine copy of the app with Apple App Attest, which stands in for the captcha on sign-in, registration and password reset. With CAPTCHA_PROVIDER=none the app sends nothing. With Turnstile on, the app offers email and password sign-in only when APP_ATTEST_APP_IDS names its App ID; otherwise it offers Sign in with Apple, Google and passkeys, which never need a captcha. A self-hosted instance that runs Turnstile and wants the iOS app to sign in with a password sets APP_ATTEST_APP_IDS to the iOS app's App ID. Verification needs no Apple account and makes no outbound call: it checks against Apple's App Attest root certificate, which ships in the binary. Each install's key is stored in the app_attest_keys table, which belongs to the instance and is not part of a workspace export. The iOS simulator cannot use App Attest, so local development keeps using TURNSTILE_BYPASS_TOKEN.

Platform mail

Platform mail is the product's own outbound: registration codes, password resets, team invitations, notification digests and login codes where those are enabled. It is separate from campaign mail, which leaves through the mailboxes you connect.

A self-hosted instance never fills those in with ours. SenditCold's own company details and website are the hosted service's, not yours, so an unset EMAIL_BRAND_* renders nothing rather than sending your users a footer naming a company they have no relationship with. Set them to your own if your jurisdiction wants an identification line on business email.

Despite the prefix, these are not only about email. GET /v1/auth/config serves the public half of them (name, website, terms, privacy, support address) and every surface that used to hardcode senditcold.com reads it: the sign-in screen's wordmark and its Terms and Privacy links, the copyable API example on the API keys page, a shared stats card, and the "powered by" line at the foot of a hosted form page. Each renders nothing when the value is unset, so a stranger filling in one of your forms is never sent to a website with no relationship to it.

log is a real transport, not a broken one: it writes every message to the backend log and delivers nothing. It exists so a fresh install can complete its first sign-in with no relay. What it costs you is password resets, invitation delivery and digests, all of which have a workaround described on accounts and access. Unless APP_ENV is dev (also development or local), the logged copy has the query string and fragment of every link replaced with ?[redacted], because that is where reset and invitation tokens travel; the rest of the message is logged as written. A transport that does not deliver never gates a sign-in or a signup on an emailed code, so no such code reaches the log.

Read a code out of the log:

docker compose -p warmbly logs backend | grep -B2 -A12 "MAIL_TRANSPORT=log"

Automatic campaign send recovery

Pre-send verification

Before a campaign sends to an address, the backend can check it: syntax, then MX, then an SMTP RCPT probe against the recipient's mail server. An address that comes back invalid is skipped rather than sent to, which turns a would-be hard bounce into a silent drop. The probe runs from the backend, never from a worker, because workers are your sending IPs.

Verdicts are re-checked after 90 days (30 for an inconclusive one), in passes of 200 contacts a minute that repeat while a backlog remains. A workspace that connected MillionVerifier under Integrations is checked through its own credits whether or not the instance-wide key is set.

EMAIL_VERIFY_HELO_HOST is not SMTP_EHLO_NAME. SMTP_EHLO_NAME is the greeting your platform mail relay sees in platform mail; this one is the greeting recipients' servers see from the verifier.

Encryption

An empty CREDENTIALS_ENCRYPTION_KEY does not fail at boot. It disables sealing, so mailbox passwords are stored unsealed. Set it before you connect a single mailbox, and back it up.

Storage

On filesystem, a remote worker writes blobs to its own disk rather than a volume the backend can read. Use s3 with a bucket both sides reach when workers run off-host.

The attachment download proxy works with filesystem and S3-compatible stores and requires no cloud account. Keep it disabled during a rolling upgrade from v0.6.72 until every backend supports the download route. Before rolling back to v0.6.72, disable it and allow the 15-minute download-link lifetime to elapse. Attachment rows, stored bytes and worker messages retain their existing format in both versions.

Event bus

Kafka event publication waits for the broker's delivery report with all in-sync replica acknowledgements, bounded by the caller's deadline and EVENTBUS_HANDLER_TIMEOUT. Local queue admission is not evidence of publication. A rejected or timed-out result keeps the original command unacknowledged for replay; a timeout does not prove the event was never delivered, so consumers still deduplicate replayed outcomes. Generic producer events are drained separately through shutdown, without consuming a publication's independent delivery report.

json needs nothing and is the default. avro resolves every event against SCHEMA_REGISTRY_URL and ships only in the -kafka images. Without CODEC_READ_BOTH=true, producers and consumers have to agree on one codec. Change it by draining the bus, or enable CODEC_READ_BOTH on every Go reader before a staged migration. The envelope subjects need BACKWARD compatibility, because a new event type adds a union branch; a publisher that meets FORWARD or FULL sets its subject to BACKWARD itself. PUBSUB_ENABLED must agree across backend, consumer and realtime.

Go services limit each decoded Avro array or map to 65,536 entries, counted across blocks, and each string or byte field to 1 MiB. Payloads exceeding these limits are rejected. The maintained encoder uses the existing schemas and Confluent framing. Queued messages within these limits remain readable without draining the bus or re-registering subjects. Keep the publishing codec unchanged until all readers support the queued formats.

Database

Existing self-hosts keep their current database configuration and startup migrations. A separate direct connection, transaction pooling and a deployment migration job are optional. They do not require cloud services or change the one-command installer defaults.

Migrations are embedded in the backend image. Self-hosts apply them on boot by default. Hosted deployments run the same image's /app/migrate binary once before rolling services, with DB_MIGRATIONS_ON_START=false on the backend. The migration binary uses PRIMARY_DB_DIRECT when configured. A failed migration blocks deployment; operators inspect its logs and schema state before retrying. Services do not automatically reverse a migration during application rollback.

Use additive schema changes that remain usable by the previous release during deployment and rollback. Stage renames, removals, data transformations and event format changes across releases. Backfill large datasets with a resumable job after compatible readers and writers are deployed.

Realtime upgrade compatibility

Existing WEBSOCKET_URL values ending at the host, /socket, or /socket/websocket remain supported, including custom path prefixes, ports and query parameters. HTTP(S) forms are converted to WS(S). Both discovery and ticket responses use the same normalization.

The dashboard can connect through /v1/realtime/socket/websocket on the API origin. No new environment variable, CSP allowlist entry or data migration is required. The API forwards only to the configured realtime gateway and keeps the browser Origin intact. Legacy Docker installs advertising a loopback gateway can fall back to the existing realtime service on the same port when the API container cannot reach its own loopback address.

Upgrade the backend, realtime and dashboard images together to the same release; the realtime update preserves per-client IP limits for API-proxied connections. Existing direct clients and older dashboard/backend combinations retain the direct URL. A custom reverse proxy should allow WebSocket upgrades on the API route as well as the direct realtime route; direct connections remain a retry fallback. Keep CHECK_ORIGIN_HOSTS and the shared AUTH_SECRET/JWT_SECRET consistent. A retry label does not prove recovery: Live means the socket is open, while Reconnecting means it is still retrying.

Cache

Redis holds rate limit counters, the organization key cache, the realtime bridge and the first-run setup token. Flushing it on an unclaimed instance destroys the claim link along with every pending auth session.

GeoIP

GEODB_PATH must be set on the backend in every environment. The file itself is optional: a missing file at that path means sessions and audit rows are recorded without a city, and nothing else changes. The consumer reads the same variable, optionally, to put a country and city on each email open and click; without it those records carry client and device only.

These are two different databases and two different files. GeoLite2-City answers where an address is; GeoLite2-ASN answers which network announces it, which is what makes the tracking service's asn: scanner entries match. Neither file contains the other's data, so a reader pointed at the wrong one resolves nothing. Both are free on one MaxMind account, as the GeoLite2-City and GeoLite2-ASN editions, and geoipupdate fetches either.

GEODB_PATH belongs on the backend and the consumer, TRACKING_SCANNER_ASN_DB on the tracking service, and each is ignored when its file is absent: a missing city database costs location labels, a missing ASN database costs asn: matching. Pointing TRACKING_SCANNER_ASN_DB at a City database is refused at boot with a line naming what it found, rather than matching nothing and saying nothing.

Getting the databases in

A file you can mount is the simplest answer and needs none of the variables below. Where there is nothing to mount from, which is most container hosts, set the matching _URL variable and the service fetches its own copy at startup.

# MaxMind's download endpoint, which always serves the current build of an
# edition, so a restart is how an instance picks up a newer one. Both halves of
# the credential come from the same page of your MaxMind account: the account
# id is the username and the licence key is the password.
GEODB_URL=https://ACCOUNT_ID:[email protected]/geoip/databases/GeoLite2-City/download?suffix=tar.gz
TRACKING_SCANNER_ASN_DB_URL=https://ACCOUNT_ID:[email protected]/geoip/databases/GeoLite2-ASN/download?suffix=tar.gz

Any URL serving a MaxMind-format database works, and the shape is read from the content rather than the file name, so a .tar.gz, a .mmdb.gz and a bare .mmdb are all accepted. A credential in the userinfo, as above, is sent as HTTP basic authentication. DB-IP's free Lite databases need no account at all and are published in the same format under dbip-city-lite and dbip-asn-lite; their country data is comparable and their city data is weaker.

Two things to know:

• Nothing waits for it. The service starts with location lookups answering Unknown and picks the database up when it arrives, so a mirror that is slow or gone costs city labels rather than a deploy's health check.
• A file that is already there is kept until it goes stale. A database less than seven days old is used as it is, with no request at all, so a mounted copy, a volume that survived a restart and a copy you dropped in by hand all win. Past seven days the file is revalidated with If-Modified-Since, which normally answers 304 Not Modified, costs nothing and does not count against MaxMind's daily allowance; only a genuinely newer database is downloaded. With no URL set nothing is ever requested and the file is yours alone.
• A refused download is retried. Four attempts with a growing wait, obeying a Retry-After when the server sends one. A throttled or briefly unreachable mirror used to cost that container its geo data for the whole of its life, because the fetch ran once at boot and never again. A refusal that will repeat, such as a bad licence key or a name that does not resolve, is not retried.
• A download that is not a database is thrown away, not installed. An expired licence key returns an error page, and storing that would disable location lookups permanently, because the next start would find a file and not retry. The bytes are opened before they are put in place, and a failure leaves the path empty and one line in the log.
• Plain http is refused for a URL carrying a credential, in the query or the userinfo, because it would put the licence key on the wire in the clear. An http mirror with nothing secret in its URL is allowed, and a redirect from https down to http is refused either way.

The tracking service is the exception to the path: it holds its ASN database in memory and never writes it out, so TRACKING_SCANNER_ASN_DB_URL works on its own and TRACKING_SCANNER_ASN_DB can stay unset.

Mirror the databases rather than pulling from MaxMind

A GeoLite2 account is allowed 30 direct downloads per 24 hours and answers 429 past that; a paid GeoIP account gets 2,000. Each consumer boot fetches GeoLite2-City and each tracking boot fetches GeoLite2-ASN, so a release costs two downloads and a restart loop costs one per restart. That is a ceiling you can reach on a busy day, and reaching it is worse than a failed download: a container that starts into a 429 has no geo data until it is replaced.

Point the URLs at your own copy instead. .github/workflows/geoip-mirror.yml runs on Tuesdays and Fridays, which is MaxMind's own publish cadence, downloads both editions once and uploads them to a private bucket. The whole fleet then reads from there, and MaxMind sees two requests a week however often you deploy. An intra-region pull of a 60 MB file is also a second or two rather than however long MaxMind takes.

It needs these repository settings, and skips itself when MAXMIND_ACCOUNT_ID is unset so a fork never fails on it:

Then set GEODB_URL and TRACKING_SCANNER_ASN_DB_URL to the mirrored objects. Both accept a credential in the userinfo or the query, so a presigned URL or an object under a non-guessable key works without the services holding bucket credentials. Only the mirror job holds the MaxMind licence key, which is the other reason to prefer this: the key stops being deployment configuration on several services and becomes one repository secret.

Keep the copies private. GeoLite2 may not be redistributed, which is also why nothing is ever baked into the published images. If you would rather have no account at all, DB-IP's Lite databases are CC-BY 4.0 and need none.

Workers

Workers hold no database connection by design. Everything relational they need arrives over the backend's internal HTTP API.

What a joining node is given

warmbly join renders /etc/warmbly/node.env from the backend's own environment, so most of this page reaches a node without being set twice. Several things are decided rather than copied:

• The crypto and blob providers are translated. An instance on KMS_PROVIDER=aws hands its nodes brokered, and one on BLOB_PROVIDER=s3 hands them brokered too, so no machine in the fleet carries a cloud credential. An instance on KMS_PROVIDER=local hands its workers brokered and keeps local for consumers, which may create a workspace's first data key. Filesystem blobs pass through unchanged.
• A worker never receives PRIMARY_DB, CREDENTIALS_ENCRYPTION_KEY, CREDENTIALS_ENCRYPTION_READ_KEYS or KMS_LOCAL_MASTER_KEY. A consumer does, because it opens Postgres itself and seals stored credentials. A worker opens only what reaches it sealed under a workspace key. An instance that keeps its DSN in a secret manager rather than the environment has none to send, and the join script says so.
• DEPLOYMENT_MODE, MAIL_ALLOW_PRIVATE_HOSTS and MAIL_TLS_INSECURE are resolved. A node cannot tell a hosted instance from a self-host on its own, so it receives the mode and the policies its control plane applies. On hosted production that means a node ignores MAIL_TLS_INSECURE and the other self-host-only opt-outs even when node.local.env sets them. A node that has not rejoined or refreshed its env since an upgrade keeps what it was given; on a hosted instance, rejoin existing nodes or write DEPLOYMENT_MODE=cloud in node.local.env.
• The backend address should be https on hosted production. Join and env refresh log a warning when ENCRYPTED_KEYS_BACKEND_URL (or APP_INTERNAL_URL) is plain http and does not name this machine. Keep a cleartext address on a private network only. A self-host keeps whatever it uses on its own network.
• Each node gets credentials of its own. NODE_LOG_TOKEN for log ingestion and NODE_CREDENTIAL for fetching its own env are issued per node on every join and never derived from the fleet-wide token.
• Addresses are inherited literally. NATS_URL, REDIS and ENCRYPTED_KEYS_BACKEND_URL are used exactly as the backend has them, and a value that only resolves inside your container network produces a node that enrols cleanly and reaches nothing.

Settings the control plane cannot know

node.env is rewritten on every join. Next to it, /etc/warmbly/node.local.env is created once and never written again, and the container reads it second, so a name repeated there wins.

That is where a per-machine value belongs: a DSN this instance does not hold, credentials for infrastructure of your own, or a deliberate override back to a direct provider.

printf 'PRIMARY_DB=%s\n' "postgres://..." >> /etc/warmbly/node.local.env
systemctl restart warmbly-consumer

Keeping a node's env current

A node joined with WARMBLY_ENV_REFRESH=true picks up configuration changes without anyone signing in to it. Every heartbeat reply carries the revision of the env the control plane would render for that node now. When it differs from what the node applied, the updater fetches the new env from the backend with the node's broker token and its own NODE_CREDENTIAL, writes node.env in one step, restarts the node and reports the new revision. If the restarted node does not stay up, the previous node.env is put back and that revision is not tried again. The backend hands each node only its own env, and logs every hand-out and refusal ("event":"broker_access", operation fleet.node_env) without its contents. A worker whose updater predates NODE_CREDENTIAL is still answered, because a worker's env carries nothing the fleet token does not already reach; a consumer's env is answered only against its credential.

warmblyctl fleet env-status lists every live node as current, stale or manual (a node that does not refresh) and exits non-zero until all of them are current. A key rotation that reaches nodes waits on it, so a manual node holds a rotation open until it is rejoined, or adopted with --env-refresh:

curl -fsSL https://<your-instance>/join.sh | sudo sh -s -- --adopt --env-refresh

--adopt installs the standard unit and updater for the node the existing node.env describes, without enrolling again. With --env-refresh it first checks that a refresh would keep every key in node.env, and refuses, listing the names, when one exists only on this machine. Keys the control plane stopped sending a worker (the credential keys and the KMS master key) are not listed: dropping them is intended. Name the others with --keep-local to move them into node.local.env. It also refuses when the control plane asks for a different version than the node runs, because the updater would move it within two minutes; set the fleet version first, or pass --accept-target. Adopting issues no NODE_CREDENTIAL, so a consumer joined before node credentials existed resumes env refresh only after it is joined again.

Mailbox connections

Needed on the backend and every worker: the backend starts the OAuth flow, and each worker refreshes the token when it expires.

Plain SMTP and IMAP mailboxes need none of this. If a worker is missing these values, the mailbox connects fine and then silently stops about an hour later, when its first access token expires.

Multiple dashboards on one backend

Several dashboard deployments can share one backend. Each points WARMBLY_API_URL at the same API and sets WARMBLY_APP_URL to its own dashboard URL. Containers generate /config.js at startup, so no second image is needed. Cloudflare Pages renders it during build:pages; set the variables on the intended Pages project and redeploy.

The customer dashboard also generates a Content Security Policy from the same deployment settings. Static bootstrap scripts and styles use build-generated hashes; scripts and styles injected as markup are not generally allowed. API, Sentry and PostHog origins are derived from their configured URLs. Before nginx starts, the dashboard reads the public /v1/auth/config endpoint at its configured API URL and allows only the exact websocket_url origin. Existing backend WEBSOCKET_URL configuration therefore needs no duplicate CSP setting. If WEBSOCKET_URL is also passed into the web container, it supplies the origin directly without an HTTP request.

pnpm build:pages performs the same discovery at build time, using curl and jq on the build host (both are included in the dashboard container). Discovery uses bounded retries and never weakens CSP if the backend is unreachable or returns invalid configuration. An unavailable backend produces a startup/build warning; restart the dashboard or rebuild Pages after it is ready. Offline/custom deployments can explicitly allow the realtime origin through WARMBLY_CSP_CONNECT_ORIGINS, for example wss://realtime.example.com. Pass it into the web service's environment, not only the Compose .env. Changing origins requires a container restart or a Pages rebuild. Pages remains fully static and needs no Function.

The Pages dashboard removes Cloudflare's default Access-Control-Allow-Origin: * header from its static responses. Dashboard assets are not a cross-origin public API. This does not change the backend's CORS_ALLOW_ORIGINS or the public image URLs used in customer emails.

Email bodies render in a separate static, script-disabled sandbox. That document permits email styling and the reader's remote-content choice without relaxing the dashboard policy. Dashboard images load from its own origin, the configured API, data URLs or temporary blob URLs. External UI images and editor previews use the backend's image-preview endpoint, which allows HTTPS public destinations only, checks every connection and redirect, and caps images at 5 MiB and requests at ten seconds. Owned public images are read through the configured blob store, so filesystem storage and private MinIO endpoints need no public-network exception. The unsafe-webhook development flag does not relax image-fetching restrictions.

This proxy is display-only. Existing records need no migration, and saved and sent email HTML retains its original image URLs, links and styles. Recipients do not use the dashboard proxy or need a session to load email images. Incoming messages keep their separate remote-content consent and do not use this proxy. Dashboard inline style attributes remain permitted for editor-generated email designs; inline scripts and arbitrary style blocks do not. Development uses fresh per-response nonces for Vite's script and stylesheet injection.

On the shared backend, keep APP_URL as the primary dashboard and add both exact dashboard origins to CORS_ALLOW_ORIGINS, for example https://app.example.com,https://assessment.example.com. Google and Microsoft mailbox connects, reauthorization, whole-domain administrator grants, integration OAuth and Slack identity verification bind the initiating browser origin to their server-side OAuth state. The callback posts only to that allowlisted origin, and a popup without an opener returns to its /oauth-return page. Missing, expired or consumed routing state shows a retry notice instead of forwarding a code to another dashboard. Valid legacy states without an origin still use the primary dashboard. Wildcards are not accepted as OAuth return origins. Provider-registered callback URIs stay on the same API.

Email/password login, OTP/2FA, signup, logout and ordinary dashboard navigation use local routes. Password-reset and invitation emails initiated from an allowlisted dashboard link back to that dashboard. Stripe checkout, credit top-ups, billing portal and warmup-plan returns use the initiating dashboard and retain safe same-origin paths. Requests without a trusted browser origin use the primary dashboard.

Each additional dashboard also needs:

• its hostname allowed on the Cloudflare Turnstile widget. A widget restricted to one hostname does not cover a sibling. Keep CAPTCHA enabled.
• its origin appended to the realtime service's CHECK_ORIGIN_HOSTS, preserving the existing origins. The API origin allowlist does not configure websocket origin checks.
• the existing WEBAUTHN_RP_ID, unchanged. Changing it invalidates every registered passkey. If it is the primary dashboard's hostname, use email/password on a sibling hostname; if it is a shared parent domain and both origins are allowed for WebAuthn, existing passkeys work on both dashboards.

This is not a global rewrite of every primary-dashboard link. Dashboard Google/Apple/OIDC login remains primary-dashboard routed. Server-generated notifications, Slack-launched links, CLI/bootstrap links and self-hosted Cloud broker returns retain their canonical instance routing. Cloud broker return origins remain bound to the registered instance.

Optional hosted static edge

The optional Worker in deploy/cloudflare/ serves public frontend files through an explicitly configured custom domain. It is independent of the application containers and is not required by self-hosts.

Deploying the checked-in Worker configuration attaches no domains or routes. Stage each hostname, verify effective TLS and frontend responses, and retain the Pages association and DNS needed for rollback before moving production traffic. The deployment procedure is in deploy/cloudflare/README.md.

Optional hosted static assets

Workers Static Assets can serve the same built frontend files directly, without a request Worker. This is a separate opt-in delivery path. The existing Pages projects remain the public build-configuration source and the rollback copy. Self-hosted containers and their configuration are unchanged.

For example, a staging entry is [{"project":"dashboard-project","worker":"dashboard-assets-stage","domains":["assets-stage.example.com"]}]. No wildcard domain is accepted. Empty domains do not attach a new domain or detach a custom domain attached by an earlier deployment. Remove an existing binding in the Worker's Settings > Domains & Routes page when needed. workers.dev and preview URLs stay disabled. The publisher preserves _headers and the dashboard's _redirects. For admin it replaces the Pages catch-all rewrite with the platform's SPA fallback and keeps the other frontends' 404 behavior. It rejects symlinks, Pages Functions, files above 25 MiB and artifacts exceeding the Free plan's 20,000-file limit.

The manual Publish released static assets workflow rebuilds an existing published stable release using its tag and exact commit SHA. It supports v0.6.28 and later releases containing the public Pages configuration importer; older tags fail the source check before publishing. It does not create a tag, release, application deployment or worker-fleet update. It uses current public Pages configuration, so a rebuilt artifact can differ from the earlier build if that configuration changed. The deployed artifact includes /.well-known/warmbly-release.json and is retained as a workflow artifact. Subsequent releases publish the same built files to Pages and configured static asset targets. Static asset requests do not invoke a Worker script; confirm the account's current limits and pricing before enabling any other Workers feature.

Before moving a production hostname, verify the staging site's certificate, TLS policy, root and nested routes, assets, response headers and redirects. Archive its existing Pages association and DNS. If verification fails after a cutover, remove the Worker's custom-domain binding and restore that Pages association and DNS. Redeploying the saved asset artifact to the same Worker reapplies its custom-domain routes; it does not restore Pages or DNS. Keep the Pages deployment and its configuration available through the rollback window. A source merge alone does not attach a domain.

Whole-domain connects

A workspace can connect every mailbox on a Microsoft 365 organization through one administrator's grant (Add account > Microsoft > Whole organization). Google Workspace domain grants are legacy and no longer offered under Add account: a grant a workspace already holds keeps working and is reached from View the grant (Check the grant when the grant is invalid) in the drawer of a mailbox it connected, and a new domain is refused with google_delegation_retired. A self-host that configured only the delegation service account can no longer set up a new Workspace domain; configure the BOX_GOOGLE_* OAuth client instead, which each domain's administrator marks Trusted. A client with the Internal user type on your own Workspace organization is exempt from Google's 100-user cap for unverified apps (Google's help). These are read by the backend only: it mints every token for those mailboxes itself and hands them to workers per use, so workers need nothing here. The complete setup, step by step, is in whole-domain mailbox connects.

Google, only for an instance whose workspaces already hold grants (a new one has no use for it):

1. In the Google Cloud console, create a project or pick the one you will keep. Its project ID is permanent.
2. Under APIs & Services > Library, enable the Gmail API and the Admin SDK API.
3. Under IAM & Admin > Service accounts, create a service account (for example warmbly-workspace). Grant it no roles on the project.
4. On its Keys tab, choose Add key > Create new key > JSON. A new Google Cloud organization blocks this by default with the Disable service account key creation policy, which an Organization Policy Administrator can lift for this project.
5. Put the key's path in GOOGLE_WORKSPACE_DELEGATION_KEY_FILE (mounted read only) or its JSON in GOOGLE_WORKSPACE_DELEGATION_KEY, and restart the backend.

Each customer's authorization is tied to the service account's Client ID, so create the service account once and rotate its keys, never the account: a recreated one has a new Client ID and every domain would have to authorize it again. The key can mint a token for any address on any domain that has authorized it, so treat it like CREDENTIALS_ENCRYPTION_KEY: it is the one credential behind every such mailbox.

A Google grant is recorded only for the workspace that proves it controls the domain, and only when the directory lists the admin address as a super administrator. The proof is one of:

• The administrator's Google sign-in, through the dashboard's own sign-in client (GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET) and its existing redirect URI (GOOGLE_REDIRECT_URI, or API_PUBLIC_URL plus /v1/auth/google/callback), with the openid and email scopes only. The callback recognizes the proof and hands it back to the dashboard window rather than signing anyone in, so no new redirect URI needs registering. When that client is not configured, the BOX_GOOGLE_* mailbox client is used instead, the same way. Neither depends on BOX_GOOGLE_OAUTH_CONNECT.
• A TXT record at _warmbly.<domain> holding a warmbly-verify= value unique to the workspace and domain, always available and the only proof when neither client is set. The value is keyed by DOMAIN_PROOF_SECRET, which defaults to AUTH_SECRET.

Microsoft 365 organizations reuse the Outlook app, BOX_OUTLOOK_CLIENT_ID and BOX_OUTLOOK_CLIENT_SECRET, and are on whenever those are set. The app registration needs three additions for them:

• Supported account types: accounts in any organizational directory (multitenant), so another organization can consent to it.
• Redirect URI: the existing one, your API base plus /addresses/outlook/callback. Admin consent returns there too.
• API permissions: the Microsoft Graph application permissions Mail.ReadWrite, Mail.Send and User.Read.All, next to the delegated ones single-mailbox sign-in already uses.

Do not grant admin consent for those in your own tenant unless you are testing with it. Consent is given per customer organization, by that organization's Global Administrator, through the dashboard. The organization a grant names is the one the administrator signed in to, read from the consent sign-in itself, never from the request. Publisher verification is optional.

Application permissions reach every mailbox in a consenting organization. We recommend that each organization scope the app to the mailboxes SenditCold should reach with an Exchange Online ApplicationAccessPolicy (or RBAC for Applications); a mailbox outside the policy is refused when it connects. The customer steps are in connect a whole Microsoft 365 organization.

GET /emails/grants/config reports what is still missing, as google_missing and microsoft_missing: the names of the settings above that are unset, and an empty list once a provider is ready.

Mailbox sync

How quickly a worker sees new mail. Workers poll every mailbox once a minute unless the provider announces changes; realtime mailbox sync explains each channel and the Google Cloud and Microsoft setup. Gmail and Graph push are off until configured, and every mailbox keeps polling either way. A joined worker receives IMAP_IDLE and SYNC_PUSH_FALLBACK_INTERVAL from the control plane; a joined consumer that should read Gmail push needs the Gmail settings in its node.local.env.

Integrations

The three Slack values come from the Slack app's Basic Information page. Set them on the backend and the consumer and restart both; the shipped docker-compose.yml passes all three to both. The request URLs are built on BACKEND_PUBLIC_URL when it is set, otherwise on API_PUBLIC_URL. See Slack app for creating the app from its manifest.

Google Sheets app

Google Sheets connects from Contacts > Sync sources, not the integrations catalog. The on-demand sync reads the selected tab and imports contacts only when you run it.

Different permissions alone do not require a separate Google Cloud project, but consent configuration and verification are shared by the OAuth clients within a project. To keep Sheets consent and verification separate from an existing Google login or Gmail setup, use a dedicated project with its own consent configuration and Web application client. A second client in the same project does not provide that isolation. Google also recommends a separate test project when evaluating new scopes for an approved app. A separate project still needs the verification required for its requested scopes.

1. In your Google Cloud project, enable the Google Sheets API.
2. Configure the project's Google Auth Platform branding and audience. For an external app in Testing, add the Google accounts that will connect as test users. For a public launch, follow Google's sensitive-scope verification requirements.
3. Create an OAuth client of type Web application, or reuse an existing Web application client in that project. Add the exact backend callback to its Authorized redirect URIs: INTEGRATIONS_OAUTH_REDIRECT_URL when set, otherwise BACKEND_PUBLIC_URL (else API_PUBLIC_URL) plus /integrations/oauth/callback. For example, https://api.example.com/integrations/oauth/callback. It is not the dashboard URL, the Google login callback, or the Gmail mailbox callback.
4. The current client requests https://www.googleapis.com/auth/spreadsheets and https://www.googleapis.com/auth/userinfo.email. Include these scopes in the consent configuration. The Sheets scope is sensitive and grants read/write access, even though contact sync does not write back. It is not the restricted Gmail scope; an existing Google login or Gmail approval does not automatically approve Sheets access.
5. Set the client's ID and secret as GOOGLE_SHEETS_OAUTH_CLIENT_ID and GOOGLE_SHEETS_OAUTH_CLIENT_SECRET in the backend's process environment, then restart or redeploy the backend. For containers, explicitly pass both variables into the service; adding them to a Compose .env alone does not forward them. Do not put the secret in dashboard build variables. Changing these settings requires a restart, not a database migration.

Neither GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET nor BOX_GOOGLE_CLIENT_ID / BOX_GOOGLE_CLIENT_SECRET is used as a fallback. If either Sheets credential is missing, Connect with Google returns 501 with an OAuth-not-configured message before contacting Google. On SenditCold Cloud, the service operator configures the client once for all workspaces; workspace members only authorize their own Google account.

After setup, connect a sync source: authorize an account that can access the spreadsheet, paste its ID, select a tab, map the columns (including email), and save. Run Sync now whenever you want to import new rows. The first row should contain column headers, with one contact per subsequent row.

HubSpot app

HubSpot mode on a self-hosted instance uses a HubSpot app you create in your own HubSpot developer account:

• Redirect URL: the value of INTEGRATIONS_OAUTH_REDIRECT_URL, or BACKEND_PUBLIC_URL (else API_PUBLIC_URL) plus /integrations/oauth/callback when it is unset
• Required scopes: oauth, crm.objects.contacts.read, crm.objects.contacts.write, crm.objects.companies.read, crm.objects.companies.write, crm.objects.deals.read, crm.objects.deals.write, crm.objects.owners.read, crm.schemas.contacts.read, crm.schemas.contacts.write
• Optional scopes: crm.lists.read (import from HubSpot lists) and sales-email-read. An account that cannot grant them still connects
• Webhooks (optional): target URL <backend>/api/v1/integrations/hubspot/webhooks, subscribed to contact.creation, contact.propertyChange, contact.deletion, deal.creation, deal.propertyChange and deal.deletion. Every HubSpot request is verified against its X-HubSpot-Signature-v3 signature over the public URL, BACKEND_PUBLIC_URL (else API_PUBLIC_URL) plus the path, so that URL must be the address HubSpot calls. Without webhooks the consumer's one-minute pull keeps everything current

APP_URL is also what the warmbly_link property on HubSpot contacts links to. A consumer that joined the fleet with warmbly join is handed the HubSpot variables and APP_URL with the rest of its environment; a worker never is.

The repository's hubspot-app/ directory is the same app as a HubSpot developer project: the app definition, the SenditCold card on contact records, the "Add to SenditCold campaign" workflow action and the webhook subscriptions. Replace https://api.senditcold.com with your backend URL in its *-hsmeta.json files and in src/app/cards/WarmblyCard.tsx, then upload it with the HubSpot CLI (hs project upload from that directory). The CLI leaves out files under a hidden directory, so upload from a path with no dot-folder in it; otherwise the card build reports its entrypoint as missing. The card is permitted to fetch only <backend>/api/v1/integrations/hubspot/app/ (permittedUrls.fetch in app-hsmeta.json). The card and the workflow action call these public routes, each verified with the app's client secret. The card routes take the portal and user from the values HubSpot appends to the query; the webhook and workflow action routes take the portal from the body and refuse a request that carries those query values:

Pipedrive app

Pipedrive mode on a self-hosted instance uses a Pipedrive app you create in the Pipedrive Developer Hub:

• Callback URL: the value of INTEGRATIONS_OAUTH_REDIRECT_URL, or BACKEND_PUBLIC_URL (else API_PUBLIC_URL) plus /integrations/oauth/callback when it is unset

• Scopes: deals, contacts and activities with full access, and users read. Leads full access (an interested reply adds a lead), contact fields full access (the SenditCold person fields) and webhooks full access (instant updates) are used when granted; without them those steps are skipped. Pipedrive takes scopes from the app, not from the request

• Webhooks: none to set up. SenditCold registers its own, per connection, at <backend>/api/v1/integrations/pipedrive/webhooks/<connection id> when a workspace switches, and removes them when it switches away. They need BACKEND_PUBLIC_URL (else API_PUBLIC_URL) to be an https address Pipedrive can reach; otherwise the consumer checks Pipedrive every five minutes instead

• App extensions (optional): the SenditCold panel on person and deal pages. Under App extensions, add a JSON panel with the API endpoint <backend>/api/v1/integrations/pipedrive/app/panel, the JSON data structure from pipedrive-app/panel.schema.json, and the Person details and Deal details locations. Add two JSON modals with the same locations: "Add to SenditCold campaign" (<backend>/api/v1/integrations/pipedrive/app/enroll, schema pipedrive-app/enroll-modal.schema.json) and "Pause in SenditCold" (<backend>/api/v1/integrations/pipedrive/app/pause, schema pipedrive-app/pause-modal.schema.json), and add both as actions of the panel. Leave every JWT secret blank: Pipedrive then signs each call with the app's client secret, which is what the backend checks, along with the user and company the token names. Pipedrive needs the backend on https with a certificate from a public authority

APP_URL is also what the Open in SenditCold person field links to. A consumer that joined the fleet with warmbly join is handed the Pipedrive variables, APP_URL and BACKEND_PUBLIC_URL (or API_PUBLIC_URL) with the rest of its environment, since the consumer is what registers the webhooks; a worker never is.

Salesforce reads SALESFORCE_OAUTH_CLIENT_ID and SALESFORCE_OAUTH_CLIENT_SECRET, from an External Client App or Connected App in your own org with PKCE required and the api, refresh_token and id scopes. One app serves production, sandbox and My Domain logins; each member picks theirs when connecting. See Salesforce self-hosting.

AI and search

Model compatibility needs no configuration. Backends disagree about the request shape: newer OpenAI models want max_completion_tokens rather than max_tokens, refuse any temperature but the default, and some refuse function tools unless reasoning is switched off, while most OpenAI compatible backends only know the older shape. The provider learns this from the first rejection of each parameter and remembers it, so a model is walked to a working shape on its first call and every later call starts there.

The OpenAI-compatible agent loop also handles catalogs larger than 128 tools automatically. It exposes read-only tool discovery and loads up to 20 matching tools at a time, rather than sending the whole catalog in one request. All permitted built-in and connected MCP tools remain discoverable, and their original permissions and approval rules still apply. Existing conversations can continue after upgrading; no database migration or AI configuration change is needed.

Set these on the backend and the consumer.

A reader answers from its own network, so it can follow a site's redirect to anything that network reaches. SenditCold refuses a page when the reader reports it landed on a private address (Jina and Crawl4AI report where they landed; DDGS and most custom readers do not), and it only hands a reader pages on public addresses. Run a self-hosted reader where it cannot reach your private services.

Workspaces can also bring their own: a Jina AI key under Integrations gives that workspace's lead finder Jina Reader and Jina Search, and a tool on a connected MCP server can be picked as the workspace's web search. See the lead finder.

Automatic inbox tagging is read by the backend, consumer, and warmblyctl. Both inbox-tagging variables are required; a key by itself does not start classification. See Automatic inbox tagging.

Lead finder

The lead finder works with nothing set: OpenStreetMap through public Overpass servers, Komoot's public Photon geocoder for typed locations, and OpenFreeMap's keyless vector tiles for the map. Every variable here is optional and read by the backend.

A workspace's own key, connected under Integrations, always takes precedence over the instance's and costs no credits.

The dashboard's content security policy allows https://tiles.openfreemap.org. A map style served from anywhere else, including your own tile server, also needs its origin (and the origins of its tiles, sprites and glyphs) in WARMBLY_CSP_CONNECT_ORIGINS on the web container, or the map stays blank.

Reading websites for emails runs from the backend, opens only public addresses on ports 80 and 443, and identifies itself as WarmblyBot/1.0. See data control for every call the lead finder makes.

Tasks and billing

Warmup replies use the same persisted SQL task identity with either queue provider. They wait at least 45 minutes after the exact parent receipt and recheck mailbox windows, roles, consent, standing, suppression and remaining budgets before native dispatch. Queue callbacks do not grant sending authority. See diagnostic generation and rollout.

Per-mailbox test_mode, test_send_enabled and test_receive_enabled separate diagnostic sending from receiving; new mailboxes default off and legacy NULL values keep prior behavior. Positive shared_daily_limit and rolling_recipient_limit optionally constrain all outbound lanes above their separate cold/diagnostic allocations. These are database-backed mailbox PATCH fields, not environment variables or global admin settings. Current participation, calendars, suppression, provider holds and durable reservations are rechecked by protocol-2 executors. See defaults, repair evidence and rollout.

The local task dispatcher runs at most eight handlers at once, lowered to one third of the backend's effective Postgres pool ceiling with a minimum of one. This leaves room for API requests and reconciliation. Work beyond that limit stays pending in Postgres without a dispatch attempt or failure stamp. The ceiling applies across polling rounds, not separately to each batch, and does not change sending authority or unknown-outcome protections.

Stripe Tax also needs account-side setup. In Tax > Settings, confirm the head office address, use the business-use SaaS product tax code (txcd_10103001) as the default, and choose whether prices include tax or have tax added at checkout. Apply the same tax code and an explicit tax behavior to every subscription product and price. A price set to inclusive or exclusive cannot be changed later, so replacing that choice requires a new Price and an update to the matching plan. Money added to a workspace balance needs no product or price in Stripe: each payment is billed as an invoice, or a Checkout line, created for the amount. The STRIPE_CREDIT_PACK_*_PRICE_ID variables are no longer read and can be removed.

Add a Tax registration only after the business is registered with that tax authority. Automatic tax returns zero with a not_collecting reason in jurisdictions where no active registration applies. Enabling Stripe Tax is not a substitute for registering or filing.

Stripe Checkout always requests the billing address needed for tax and lets business buyers provide a legal name and business tax ID. The active default billing portal configuration must allow updates to payment methods, billing email, billing address, name, and tax ID, and must show invoice history. This lets a customer correct its tax location or registration before a renewal.

Point the webhook endpoint at /webhook/stripe using API version 2026-08-26.dahlia, matching stripe-go v86. Subscribe to checkout.session.completed, checkout.session.async_payment_succeeded, checkout.session.expired, customer.subscription.created, customer.subscription.updated, customer.subscription.deleted, invoice.paid, invoice.payment_failed, invoice.finalization_failed, payment_intent.succeeded and charge.refunded. If you replace the endpoint, update STRIPE_WEBHOOK_SECRET to the new endpoint's signing secret before disabling the old endpoint. The handler accepts the previous webhook shape during an endpoint upgrade, but new endpoints should use the matching version.

Turning on Stripe Tax does not update existing subscriptions. Review and migrate them from Tax > Migrations after setting the prices' tax behavior. Stripe applies the tax change from the next billing cycle without prorating it. The Dashboard migration does not operate on sandbox subscriptions, so update those individually when testing.

Delayed sends run through the local poller, so the backend must be running for scheduled work to fire.

Observability

Error reporting

Every runtime in SenditCold can report its errors, and none of them do unless you say where. Where is the operator's choice, not a requirement of the software, so the default install reports nowhere and contacts no host.

There are two backends and you can run either, both, or neither:

• PostHog is the default. One project key carries error tracking for every service, and the same key carries the handful of product events described below, so there is one account to have rather than two.
• Sentry is still fully supported. An instance that already reports to Sentry keeps working unchanged, and setting POSTHOG_ERROR_TRACKING=false keeps a PostHog key configured for analytics while sending no exceptions there.

Set both and every error goes to both. Each variable is read by one process, so pointing the backend at a project does not make the dashboard report too.

Configuring neither backend means nothing leaves the process and no host is contacted. The dashboard, the admin panel and form pages go further and never load an SDK at all, so there is not even a script; the Go, Rust and Elixir services still record errors, to their own logs.

A PostHog key can name PostHog Cloud (US or EU) or your own PostHog through POSTHOG_HOST. A Sentry DSN can point at Sentry Cloud, a self-hosted Sentry, or any Sentry-compatible server such as GlitchTip; nothing in SenditCold assumes sentry.io.

The distinct id on a server-side PostHog exception names the process that raised it (warmbly-backend, warmbly-worker), person profiles are switched off per event, and IP geolocation is disabled, since the address would be the server's. A browser exception belongs to the signed-in person the app identified, so it sits next to that person's session replay.

Exceptions still carry the context that makes them answerable, as searchable properties:

Browser exceptions also carry a trail of steps: the routes visited and the API calls that failed just before, each with its method, path, status and request id. In the dashboard and the admin panel the exception also sits next to the session replay it happened in, when WARMBLY_POSTHOG_SESSION_REPLAY is on; see the analytics section below for what those two surfaces record.

The admin panel and the dashboard also tag events with the build they were served from. That value is baked at image build time, because it has to match the release the source maps were uploaded under, so it cannot be changed by a container variable afterwards.

Source maps

A browser stack trace is minified without them. Uploading them is optional and off unless the build is given credentials, so a fork, a self-host build and a local pnpm build emit no source maps at all and the shipped bundle is identical.

In this repository's release workflow the two project ids are repository variables and the two keys are repository secrets, passed to the image build as build secrets. Nothing in the app has to match a release name: PostHog pairs a stack frame with its map through the chunk id the CLI injected into the served file, so an event from any build resolves as long as that build's maps were uploaded.

A static host builds with pnpm build:pages instead of an image, and that runs the same upload when the same two variables are set in the host's build environment. Setting them there matters more than it looks: with no maps uploaded, PostHog falls back to fetching <bundle>.js.map from the site itself, a static host answers that with its SPA fallback, and every frame in every issue reads Invalid source map: bad json beside a minified function name.

Product analytics and session replay

These are the same two keys as the error tracking above, and they exist for the hosted service. A self-hosted instance that sets no key loads no analytics and sends no usage data: the installer never asks about them, and a build with none of them set contains no analytics script to block. An instance that does set a key gets everything below.

What each surface sends, with a key set:

The person in the dashboard and the person on a server-side event are the same account id, so a signup counted by the backend, the session that led to it and the subscription the Stripe webhook later reports are one funnel. The acquisition parameters the marketing site carries into the signup link land on that person write-once at signup. Countries come from PostHog's own GeoIP on the address the event arrived from; the /ingest proxy above forwards the visitor's address for exactly this, and the server-side events send the signup request's address along.

Session replay masks what is typed into a password field in the browser, before anything is sent. Every other input and every screen is recorded, which is what makes a replay answer "what exactly happened", and it is also why the hosted service's privacy page describes it in those words.

Push

Updates

The updater's own variables (UPDATER_MODE, UPDATER_COMMAND, UPDATER_REPO_DIR and the rest) are on Updates; it is a separate process and its environment is not on the configuration page.

Forms service

The public face of hosted forms (cmd/forms): it serves the React (TanStack) form app built from forms/, the per-form page shells, the embed loader and public submissions, on its own port so form traffic never touches the API origin. It reads its own environment; none of these appear in the admin panel except FORM_IP_RATE_LIMIT.

Portal service

The portal service serves what an agency's clients open: client connect pages, shared reports and the /embed.js loader that puts either on a website. Its source is portal/worker, and it serves the page app built from portal/. It answers two kinds of host:

• the shared address, PORTAL_DOMAIN: connect links and reports of any workspace, and nothing else. Every other path answers "not found"
• every verified agency portal domain: that agency's clients' connect links and reports, and the dashboard from DASHBOARD_ORIGIN on every other path. Any other host answers 404, or goes to FALLBACK_ORIGIN when that is set

It holds no database and no secret beyond PORTAL_EDGE_TOKEN: it asks the backend whether a host is a verified portal domain and resolves every link through the backend's internal portal routes. Connect links and report links need it. Without PORTAL_DOMAIN the backend mints neither and every other feature works as before.

On SenditCold Cloud it is a Cloudflare Worker (see above). A self-hosted instance runs the same code as its own portal container, ghcr.io/warmbly/warmbly/portal, published with every release beside the other images. The image runs it on workerd, the open-source runtime Cloudflare Workers run on, as a non-root user, listening on port 8095. The installer, the bundled docker-compose.yml and the Coolify compose file all include it. It reads its own environment:

FALLBACK_ORIGIN is what lets one proxy rule send every customer-owned domain through the portal. The installer's Caddyfile sends a custom domain's tracking paths (/t/, /c/, /tracking.js, /p, /unsubscribe/) to the tracking service and its form paths (/f/, /forms.js, /api/forms/) to the forms service, as before, and everything else to the portal. The portal serves an agency portal domain itself and hands any other host, such as a custom tracking domain's root or a root redirect, to tracking unchanged, so those answer exactly as they did before the portal existed.

Per visitor and minute it allows 30 connect link lookups, 10 sign-in starts or server-details submits, 60 sign-in result polls and 60 report reads, counted by the platform's rate-limit bindings when it has them (CONNECT_VIEWS, CONNECT_ACTIONS, CONNECT_POLLS, REPORT_VIEWS) and in the process otherwise. The backend also limits attempts per link.

The backend side:

The dashboard on a portal domain

For the dashboard an agency's portal domain serves (see white-label):

1. Set DASHBOARD_ORIGIN on the portal service.
2. Keep CHECK_ORIGIN_HOSTS set on the realtime service (the installer sets it for a domain install). Portal origins are only added to a configured list.
3. The dashboard's WARMBLY_API_URL must be an absolute URL, such as https://api.example.com, because the dashboard on a portal domain calls the API on its own address.
4. For the security check on portal sign-in, set TURNSTILE_SITE_KEY on the backend and list the API's hostname for that site key in Cloudflare (see captcha).

Upgrading a self-hosted instance

The portal is optional, and an instance upgraded without it keeps working: connect links and report links are refused (client_links_unavailable, report_links_unavailable) until both PORTAL_DOMAIN and the portal container exist, and nothing else changes. To turn it on:

• Installed with install.sh: run the installer again. It adds the portal service, PORTAL_DOMAIN (portal.<host> by default) and a generated PORTAL_EDGE_TOKEN to an existing install without touching any other value, and the regenerated Caddyfile adds the portal's site block. Point the portal host's DNS at the machine. The admin panel's Update and restart pins a release and pulls it but does not regenerate the compose file, so it does not add the portal on its own.
• The bundled docker-compose.yml: it already lists the portal service. Set PORTAL_DOMAIN and PORTAL_EDGE_TOKEN (openssl rand -hex 32) in .env, run docker compose up -d, and route the portal host to port 8095 with its Host header kept. With both unset the container serves nothing itself and passes every request to FALLBACK_ORIGIN.
• Coolify: set the same two variables and give the portal service the portal host on port 8095.

The backend reads PORTAL_DOMAIN per request and PORTAL_EDGE_TOKEN at start, so it needs a restart after the token is set; docker compose up -d recreates it when .env changes. Start the portal before or with the backend that has PORTAL_DOMAIN, because links are built on that address from the first request. During a rolling update the order does not matter otherwise: a backend from before the portal ignores both variables and builds no link, so a new portal in front of it has nothing to serve and passes every other host to FALLBACK_ORIGIN as before.

The on-demand certificate check (/tls/authorize) answers yes for PORTAL_DOMAIN and every verified portal domain. The API accepts cross-origin calls, and OAuth sign-ins return, only for portal domains that are verified at the time, checked against the database and cached for at most two minutes.

Tracking service

The Rust open and click service. It reads its own environment, so these have to be set on that container, and none of them appear in the admin panel.

Run a single copy of it. Its event deduplication, its per-source rate limits and its click-ticket caches are per process and in memory, with no shared store behind them, so a second replica does not halve the work: it doubles the budgets a flooding source gets and lets the same open be published twice. One container handles the pixel and click load of a large instance comfortably; scale the workers instead. If you must run more than one, put a sticky-by-source-address hash in front of them, which keeps a given source on one process and restores both properties.

A catalogue entry is <source> <scope> <certainty> <label>. Scope is which requests it may judge: all for a network that never carries a person's own request, clicks for one that also proxies a mail client's image fetches, where a pixel fetch is a genuine open. Certainty is how much the match settles, and defaults to certain: the source only ever filters mail, so a match is the whole verdict. probable says the source can also carry a person, which browser isolation makes true of the mail-security vendors, and a match then widens the consumer's machine window (tracking.machine_window_probable_seconds) instead of deciding. The two columns are independent. A probable entry moves events in one direction only, from counted-as-human to counted-as-automated, so it can never let a scan count as engagement; outside the window the event is judged exactly as an unrecognised one would be. The cost is the other side of that: a recipient who really does click inside the window is classified automated, and separately, a click from any recognised network is redirected without its identification ticket, because the edge has no dispatch time and cannot tell the delivery-time scan from the isolated human click the way the consumer can. So an isolated click the consumer later counts as a person's still leaves no website-visit attribution for that contact.

asn: entries need the source ASN, and there are two ways to have it. TRACKING_SCANNER_ASN_DB resolves it from the address itself and needs no ASN header and no transform rule, which is the option for an instance behind nginx, Caddy, a cloud load balancer, or exposed directly. TRACKING_SCANNER_ASN_HEADER takes it from a header your edge writes, which on Cloudflare is one transform rule. Where both are configured the header wins on any request that carries one, so adding the database never changes a verdict your edge has already given; a request the header does not cover falls through to the database.

Neither removes the need for TRACKING_TRUSTED_PROXIES behind a reverse proxy, for different reasons. The header is read only from a peer listed there, so a header with that variable empty resolves nothing at all. The database reads the client address, and with that variable empty the client address is the proxy's, so the lookup returns the proxy's network rather than the requester's. With no usable source at all, the ASN half of the catalogue is inert and the tracking service says so at boot, naming both.

Which entries ship on, and how, follows from that. Microsoft 365's Exchange Online Protection ranges and Barracuda's Email Gateway Defense blocks are the mail-filtering tier itself, where no recipient ever reads their mail, so they are certain and settle the verdict outright. Proofpoint, Mimecast and Cisco Secure Email are probable and matched by ASN: they are pure mail-security networks, but Proofpoint Isolation and Mimecast Browser Isolation render a clicked page in the vendor's own cloud and stream it to the recipient, so a click from there can have a person behind it. Isolation is usually scoped to uncategorised URLs, which is what a new outreach domain looks like. The probable window is what makes them safe to ship on: the scan runs within minutes of the send, and the isolated click runs whenever the person got to it.

The rest of Microsoft's and Google's networks stay commented out even as probable, because they are whole cloud allocations rather than mail-security networks. A recipient on Windows 365, Azure Virtual Desktop or a corporate NAT gateway sits inside them all day, so a match says almost nothing about who is asking. tracking/scanner-networks.txt documents every entry and is the place to change them.

Realtime service

The Elixir websocket service. Its runtime configuration is read only when the release boots in prod, which is how the shipped image runs. Like tracking, it reads its own environment and appears nowhere in the admin panel.

Settings stored in the database

These are the only settings a browser can change, and no environment variable owns any of them. They live in the admin panel under Instance > Configuration > Settings (/configuration?tab=settings). Reads are cached for 30 seconds in each process, so a change takes effect everywhere within that window.

The four sync.* values are read by the backend when a mailbox is loaded onto a worker (on connect, on reassignment, and by the reconciler's periodic republish), so a change reaches every mailbox within 30 minutes without a restart (the republish gives each mailbox one turn per 30 minutes, spread out so no worker gets them all at once). The fixed pacing numbers around them (burst per five minutes, hourly, backfill pace, the flood threshold and the chronic-overage rule) are compiled constants listed under Instance > Configuration > Effective limits; see Mailboxes for how the budgets behave. The four sync.* values also have a companion read view on Operations > Sync, which shows each mailbox's backfill progress and fair-use throttle against them and can clear a throttle or restart a backfill.

The eight retention.* values are read by the pruning sweeps on every pass, so shortening one takes effect on the next sweep rather than at the next restart. Deletion is permanent and there is no grace period: what already sits outside a shortened window goes on that sweep. The warmup mail window is the one that reaches into customers' mailboxes: the consumer retires each message and the worker holding the mailbox deletes it there, so shortening it empties the warmup folders of the whole instance down to the new window within a few hours. See data control for what each log holds and what a shorter window costs.

An unattended install can seed the whole document before anyone signs in, with WARMBLY_SETTINGS_BOOTSTRAP holding the same partial JSON the admin API takes:

WARMBLY_SETTINGS_BOOTSTRAP={"sync":{"backfill_days":30},"retention":{"audit_log_days":30}}

It is applied only while the settings row has never been written, so from the first save in the panel onwards the panel is authoritative and the variable is a no-op. That is what install.sh --wizard writes when you answer its retention questions.

The two deliverability.* values are read on every scheduling pass and every warmup send, so turning the gate off releases blocked mailboxes within the 30 second cache window. Turning it back on does not stop anything retroactively: a domain still has to spend its whole grace window failing first. Only a sustained failure gates, so a domain reading unknown (never checked, DNS could not answer, or a special-use domain that cannot resolve) always sends. Manual sends and unibox replies are never gated; see domain authentication.

The placement.* allowance, seed and spacing values are read when a test starts, so a change applies to the next test and never to one already sending. The four placement.batch_* values are read on every pass of the batch runner, about every 30 seconds, so a change reaches running batches within a minute; lowering the concurrency never stops a mailbox already sending, it only delays the next start. A value of zero or below resolves to the default, and one above the range to its ceiling. Which mailboxes form the instance seed panel is not a setting: an administrator marks them from the admin panel.

Changing them is audited, and every value is validated and clamped server side on write as well as on read, so a row written by an older version still resolves.

Usage pricing

The usage section is the price list a workspace balance is charged from. A charge is what the work used at these prices, plus usage.markup_percent, rounded up to the next millionth of a dollar. Edit it in the admin panel under Instance > Configuration > Settings, in the Usage pricing section, which shows what a workspace is charged for each row after the markup.

• Models are matched on the model name that ran the call: a row with the exact name first, then the longest row the name starts with (so claude-sonnet-4 prices claude-sonnet-4-6), then the * row, which prices every model without a row of its own and cannot be removed. Rows with an empty or repeated name or a negative price are dropped on save.
• Units price work that is not tokens: web_search (one web search an AI feature makes), placement_test (one paid placement test), and three for the lead finder: lead_website_read (one website read, including the people it names and their addresses), lead_person_email (one named person's email found on request) and lead_search_page (one page of a search through a source the instance pays for; the default is Google Places' Text Search list price, so set it to what your provider charges). A missing or negative unit falls back to its default.
• A change applies to charges made after it, within the 30-second settings cache. Charges already made keep their price.

Only an instance that bills (BILLING_PROVIDER=stripe) charges anything, so on a self-host with billing off this section has no effect. See Balance and usage for what workspaces see. A partial patch replaces usage.models and usage.units whole when either is present:

{"usage": {"markup_percent": 30, "units": {"web_search": 0.01, "placement_test": 0.5}}}

Variables that do not do what their name suggests

See also

Hosted AWS deployment automation accepts DEPLOY_FLEET_BAKE_SECONDS (300 through 3600, default 300) in its deployment client. The workflow supplies it from fleet_bake_seconds or the AWS_FLEET_BAKE_SECONDS GitHub repository variable. It controls healthy uptime per stage for future worker rollouts, not application startup or existing self-host settings. See AWS deployment.

• Install for the one-command install that writes all of this for you
• Data control for where each store lives and what each retention window governs
• First run for claiming a fresh instance
• Accounts and access for who may sign in and how to invite people
• Instance health for the checks that read these values back
• Troubleshooting for the errors these settings produce