Skip to content

Scalability and recovery#

Topology#

Several manent instances share the same Postgres and pick deploys from the same queue. API and workers can be scaled separately:

manent-server api    # N API instances behind a load balancer
manent-server worker               # M workers, each with a different executor_id

flowchart LR
    LB[Load balancer] --> A1["API<br/>manent-server api"]
    LB --> A2["API<br/>manent-server api"]
    A1 --> PG[("Postgres<br/>(DBOS database)")]
    A2 --> PG
    PG --> W1["worker<br/>executor_id=manent-0"]
    PG --> W2["worker<br/>executor_id=manent-1"]
    PG --> W3["worker<br/>executor_id=manent-2"]

SQLite

SQLite is fine for development and for workers on the same machine. Several nodes need Postgres.

executor_id#

Every worker must have a unique and stable dbos.executor_id (default: the hostname; DBOS__VMID takes precedence).

At startup a worker resumes only the deploys left pending with its own ID, from the step where they stopped. If a node does not come back with the same ID, its deploys stay PENDING until another worker resumes them through recovery, manual or automatic. In Kubernetes a StatefulSet is recommended, whose pods have stable names (manent-0, manent-1, …): the Helm chart uses the pod name as executor_id.

With the default, several workers on the same machine would all use the hostname as ID: give each one its own ID, stable across restarts.

DBOS__VMID=build01-w1 manent-server -c manent.yml worker
DBOS__VMID=build01-w2 manent-server -c manent.yml worker

A worker started with the ID of another one that is still active (not stopped and with a heartbeat more recent than recovery.stale_after) does not start. Otherwise, at startup, the DBOS recovery would also resume the deploys running on the other one, which would run twice. Before giving up it waits for recovery.stale_after: this covers a restart after a crash, where the previous process did not record its shutdown. The error names the host using the ID.

Several regions#

With targets in several regions, wave rollouts update one region (or one AZ) at a time. With dbos.region_routing each worker declares its dbos.region and runs only the passes on that region's targets, from the <queue>.<region> queue: SSH, downloads and checks start from the region's network. There is still a single database, reached by all regions.

Plugin queues#

Some plugins need a runtime that manent's distroless image does not have: the ansible plugin needs Python and ansible-core on the machine of the worker that runs it. Give such a plugin its own queue, and let only a dedicated pool of workers serve it:

manent.yml (every process)
plugins:
  ansible:
    external: {command: /usr/local/bin/manent-plugin-ansible, sha256: "<sha256>"}
    artifact_types: [ansible]
    queue: ansible
manent.yml (only the workers of the pool, image server-ansible)
dbos:
  plugins: [ansible]
  • At submission manent routes the deploy to the plugin's queue: by the plugin of the effective descriptor (deploy.plugin, also when it comes from the environment configuration) or, when the environment configuration cannot be read at that moment, by the artifact type (artifact_types, so each type belongs to at most one plugin with a queue). The plugin's queue wins over a tenant's queue.
  • Only the workers that list the plugin in dbos.plugins register and dequeue its queue, besides their own. As soon as a plugin has its own queue, every worker dequeues only the queues it knows (default, tenants, its region, its plugins): DBOS keeps the registered queues in the database, and without this a worker would also dequeue the plugin's. Everywhere else, the API included, the plugin is a stub that knows its name and artifact types: it never starts, so its binary and runtime are not needed there. A deploy that reached such a worker anyway fails without retries.
  • The queue has the limits of the default queue (concurrency, environment_concurrency, global_concurrency) and shows as kind plugin in manent queues and in the Platform page.
  • dbos.plugins is not supported together with dbos.region_routing yet.

Routed queues#

Some deploys need a worker the others do not: a GPU, a network segment only some machines reach, a team's credentials. Route them to a dedicated queue by tenant/application, artifact and environment, and let only a pool of workers serve it:

manent.yml (every process)
queues:
  - name: gpu
    match:
      applications: ["ml/*"]          # qualified with the tenant, as in permissions
      artifacts: [trainer, scorer]
      environments: [prod]
    concurrency: 2                    # global limit of the queue (default: dbos.global_concurrency)
manent.yml (only the workers of the pool)
dbos:
  queues: [gpu]
  exclusive: true                     # optional: only gpu, not the default and tenant queues
  • The entries are operator configuration: a descriptor cannot choose its queue. At submission the first entry whose match applies wins; match needs at least one pattern, and the empty lists match everything. The routed queue wins over a tenant's queue, a plugin's queue wins over both, because only its workers have the plugin's runtime.
  • Only the workers that list the queue in dbos.queues register and dequeue it, besides their own; with dbos.exclusive they dequeue only the queues of dbos.queues and dbos.plugins (and the approvals, which take no deploy slot). As with plugin queues, as soon as queues is set every worker dequeues only the queues it knows.
  • The queue has the concurrency and environment_concurrency of the default queue; its own concurrency is the global limit, and also caps the deploys of the queue on each worker. It shows as kind routed in manent queues and in the Platform page, also while empty: a routed queue without workers keeps its deploys queued.
  • The route is decided at submission and recorded in the request: scheduled ticks, approvals, dependency waits and recoveries keep the queue even if queues changes later. A queue name also in the tenant registry stops the workers at startup.
  • queues is not supported together with dbos.region_routing yet.

Concurrency#

The queue is partitioned by <application>/<environment>.

Option Scope Effect
environment_concurrency (default 1) all workers Concurrent deploys of the same environment of the same application. With 1, deploys of myapp/prod are serialized.
concurrency single worker Deploys running on that worker.
global_concurrency all workers Deploys running in total (0 = no limit).

Immediate, scheduled and recurring deploys and dry-runs all go through the same queue and respect the same limits.

Recovery#

What happens if a worker stops during a deploy:

  1. the deploy stays PENDING, associated with the worker's executor_id;
  2. on restart, the worker resumes it: steps already completed are not rerun, the interrupted one restarts;
  3. if the deploy resumes on a different worker, because the machine was replaced keeping the ID, and the working directory does not contain the artifact, the deploy step downloads it again and verifies its sha256.

manent workers shows each worker's state, last heartbeat and running workflows; manent activity which worker is running each deploy.

This is why steps and plugins must be idempotent. The remote-copy plugin, for example, writes to a .part file and renames it atomically: rewriting the same artifact produces the same result.

Shutdown#

worker and serve stop on SIGINT or SIGTERM. The worker stops picking up deploys and waits for those in progress up to dbos.drain_timeout (see drain). Those not finished within this time stay pending and are resumed on restart. External plugin processes are stopped after DBOS.

Concurrent startup#

On an empty Postgres, several workers started together try to create the DBOS schema at the same time. Initialization is retried automatically with an increasing wait, so there is no need to start workers one at a time. The API (api) and admin create the schema too if they find it missing, so they can start before the workers.