Cluster Agent

Configuration

Every MPC Agent config knob — identity, license, intra-cluster transport, storage, API auth, rate limits, policy webhooks, performance & limits.

Configuration

The agent reads a YAML config file at startup. Pass its path with --config; when omitted it defaults to config.yml in the working directory. Every field below maps to a top-level key in that file. Env-var interpolation via ${VAR_NAME} is supported in any string value, so secrets stay out of the config file and in your secret store.

agent start --config /etc/mpc-agent/config.yml

A starter config you can drop in and customize:

node:
  id: "node-1"
  display_name: "MPC Node 1"

identity:
  private_key_hex: "${NODE_PRIVATE_KEY}"
  allow_dynamic_nodes: false
  trusted_peers: []

license:
  certificate: "${MPC_LICENSE_CERT}"

rabbitmq:
  url: "amqp://mpc:mpc@broker:5672/%2F"

database:
  url: "sqlite:///app/data/agent.db?mode=rwc"

api:
  host: "0.0.0.0"
  port: 3000
  auth:
    auth_type: api_key
    api_keys:
      - "${API_KEY_1}"
  rate_limit:
    enabled: true
    max_per_minute: 300

security:
  session_timeout_ms: 300000
  session_expiry_window_ms: 30000
  max_concurrent_sessions: 200
  ack_timeout_ms: 60000
  session_retention_ms: 86400000
  session_sweep_interval_ms: 300000

log:
  level: info
  format: json

node

node:
  id: "node-1"           # REQUIRED — stable identifier, used in cluster routing and logs.
  display_name: "MPC Node 1"   # Optional, surfaces in /node/info responses.
  index:                 # Optional. Auto-derived from peer list on each request.

The id is the agent's stable identity within the cluster. It must be unique across every node in the same logical cluster (the cluster sharing one license, one intra-cluster transport, one ceremony session-id namespace). Changing it after the first DKG breaks every wallet that node was a party to — treat it as immutable.

index is normally auto-detected from each ceremony request's peer list (matched by node_id). Set explicitly only if you're debugging an asymmetric mis-assignment.


identity

identity:
  private_key_hex: "${NODE_PRIVATE_KEY}"      # REQUIRED. 32-byte Ed25519 seed, hex-encoded.
  allow_dynamic_nodes: false                  # If true, accept any node presenting a valid signature.
  trusted_peers:                              # Hex Ed25519 pubkeys; ignored when allow_dynamic_nodes=true.
    - "<node-1-public-key-hex>"               # include THIS node's own key (see note below)
    - "<node-2-public-key-hex>"
    - "<node-3-public-key-hex>"

Generate the seed with agent gen-secret. It prints two labelled lines — Private key : <hex> and Public key : <hex>. Use the private hex for private_key_hex and the public hex when pinning this node in other nodes' trusted_peers. Each node MUST have its own seed — sharing seeds across nodes breaks cluster identity.

allow_dynamic_nodes: true accepts any node that presents a valid Ed25519 signature — useful for permissive dev environments where you spin nodes up and down freely. In production, set false and pin the public keys of every node you trust under trusted_peers.

:::warning When you set trusted_peers to a non-empty list, it must include this node's own public key alongside the other nodes'. Each node sees its own ceremony messages as well as its peers', so a list that omits the node's own key causes it to reject its own messages and ceremonies stall. The two safe options are: list every node including this one, or leave trusted_peers: [] empty (which disables pinning entirely — every node presenting a valid signature is accepted, equivalent to allow_dynamic_nodes: true for trust purposes). :::


license

license:
  certificate: "${MPC_LICENSE_CERT}"

The certificate is an issuer-signed bundle that pins:

  • The licensee identity (your organization).
  • The allowed curves (Secp256k1, Ed25519, etc.).
  • The allowed ceremony kinds (DKG, presigning, sign, reshare, sign-pack export).

The agent verifies the cert at startup and refuses every ceremony if it can't. The cert is public-facing material — fine to put in a config file or env var.


rabbitmq

rabbitmq:
  url: "amqp://mpc:mpc@broker:5672/%2F"
  exchange: "mpc.ceremony"   # Optional. Defaults to "mpc.ceremony".

  # Optional throughput tuning. Defaults are fine for production workloads
  # up to ~10k ceremonies/day per node; raise the pool sizes for higher
  # volume. See the "Scaling tiers" section below for concrete recipes.
  prefetch: 100              # Per-consumer prefetch. Caps unacked-in-flight on the broker side.
  publish_channels: 4        # AMQP channel pool for parallel outbound publishes.
  workers: 32                # Worker pool size for inbound envelope dispatch.

Ceremony envelopes between cluster nodes flow over a message broker. The agent supports any AMQP 0.9.1 broker (any production AMQP broker works — managed alternatives like AWS MQ or CloudAMQP work without changes).

publish_channels controls how many AMQP channels the agent opens for outbound publishes. Each channel is independent at the AMQP layer — confirms on one don't block sends on another, so N channels give roughly N-way parallelism on publish throughput. Default 4 is enough for typical production loads; raise to 16-32 for high-volume deployments. the broker's per-connection channel cap (default 2047) is the hard ceiling.

workers is the inbound dispatch pool size. Each worker pulls envelopes from a shared queue and runs the per-envelope handler (commitment / share / partial-signature processing). Increasing this raises the agent's ability to handle concurrent in-flight ceremonies; the practical limit is the host's CPU + memory budget. The earlier hard cap of min(prefetch, 32) is gone — set as high as your hardware supports.

prefetch tunes the broker side: how many unacknowledged messages the broker will deliver to this consumer at once. Keep it ≥ workers so the worker pool isn't starved waiting for the broker.

Production sizing. A single broker instance handles tens of thousands of ceremonies per day without tuning. For higher volume, run a clustered broker (3 replicas is the standard pattern) or use a managed broker. The agent reconnects on backoff if the broker goes down — transient outages don't kill the process.

Security. The broker is intra-cluster only — do NOT expose it to the public internet. Treat the connection string as a secret. Inside the cluster, every envelope is end-to-end X25519-sealed before it ever hits the broker, but firewall the broker port anyway as defense-in-depth.


database

database:
  url: "sqlite:///app/data/agent.db?mode=rwc"   # REQUIRED for persistence.
  max_connections: 10                            # Optional. Default 10.

The database block is what turns persistence on. With it set, the agent persists ceremony state, key shares, presignatures, and audit logs, and runs schema migrations at startup. Omit it entirely and the agent runs stateless — nothing is persisted and agent.keyShares.list() returns an error, so DKG output can't be looked up. A persistent database is required for any real deployment.

Three URL schemes are accepted:

SchemeExample
SQLitesqlite:///app/data/agent.db?mode=rwc
PostgreSQLpostgres://user:pass@host:5432/dbname
MySQLmysql://user:pass@host:3306/dbname

For SQLite, the directory in the path (/app/data above) must already exist and be writable by the agent — mode=rwc creates the database file but not its parent directory. The published image ships a writable data directory at /app/data; point the SQLite URL there (and mount your volume at /app/data) and it works without extra setup. Sensitive fields (key shares, presignatures) are encrypted at rest with a key derived from the node identity — no separate key management is required.

Backup. Snapshot the database (or the SQLite file's directory) regularly. Restoring on a new host requires the same identity.private_key_hex and the same license.certificate — both are bound into the at-rest encryption, so restoring one node's data onto a different node's identity fails to decrypt.


storage

storage:
  backend: db   # db (default) | file | s3

Controls where encrypted key-share blobs live. The default db backend stores them inline in the database above — most deployments need nothing here. Set backend: file (with file.path) to keep blobs as one encrypted file per share on a local volume, or backend: s3 for an S3-compatible object store. All backends still require the database block for metadata; storage is not a substitute for database.


api

api:
  host: "0.0.0.0"
  port: 3000

  tls:                          # Optional. Omit when terminating at a load balancer.
    source: file                # file | secrets_manager
    cert_path: "/certs/server.pem"
    key_path:  "/certs/server.key"

    # Or, AWS Secrets Manager (secret must be JSON: {"cert_pem":"...","key_pem":"..."}):
    # source: secrets_manager
    # secret_id: "arn:aws:secretsmanager:us-east-1:123:secret:mpc-agent-tls"
    # region: "us-east-1"

  auth:
    auth_type: api_key          # api_key | jwt
    api_keys:                   # When auth_type: api_key
      - "${API_KEY_1}"

    # When auth_type: jwt:
    # auth_type: jwt
    # jwt_secret: "${JWT_SECRET}"
    # (Tokens must contain a "sub" claim and be signed with the secret.)

  rate_limit:
    enabled: true
    max_per_minute: 300         # Per API key.

The agent's HTTPS interface. Every request is authenticated (no anonymous access). Pick auth_type: api_key for the simplest setup; configure multiple keys to rotate without downtime.

For production, terminate TLS upstream (ACM + ALB on EKS is the canonical pattern) and leave api.tls unconfigured. The internal HTTP between the LB and the agent is fine because it never leaves your VPC. If you must terminate TLS on the agent itself, the tls.source: file path is standard; the secrets_manager source is a convenience for AWS deployments where cert rotation runs through Secrets Manager.

rate_limit.max_per_minute is per API key. The default 300/min is sized for individual-device clients; raise it for backend integrations that batch thousands of operations.


security

security:
  session_timeout_ms: 300000        # 5 min — how long a ceremony stays active before timing out.
  session_expiry_window_ms: 30000   # 30s grace period after timeout before the session is GC'd.
  max_concurrent_sessions: 200      # Per-node concurrent session ceiling.
  ack_timeout_ms: 60000             # 60s to collect all acks before failing the ceremony.
  session_retention_ms: 86400000    # 24h — how long terminal sessions stay in the DB. Set 0 to disable retention.
  session_sweep_interval_ms: 300000 # 5 min — how often the retention sweeper runs.

These knobs control ceremony liveness, GC, and DB growth. Defaults are tuned for embedded-wallet usage (5-minute ceremonies, 200 concurrent users per node, 24h audit trail).

For high-throughput backends, raise max_concurrent_sessions (it gates concurrency at the session-state-machine layer with near-zero memory overhead per session). For audit-heavy environments, raise session_retention_ms and let the DB grow.

session_sweep_interval_ms: 0 disables the sweeper, leaving terminal rows in the DB forever. Useful when you have an external archival pipeline that pulls terminal sessions to long-term storage; otherwise leave the default.


policy_webhooks (optional)

policy_webhooks:
  pre_dkg:
    url: "https://policy.example.com/mpc/pre-dkg"
    secret: "${POLICY_WEBHOOK_SECRET}"
  pre_sign:
    url: "https://policy.example.com/mpc/pre-sign"
    secret: "${POLICY_WEBHOOK_SECRET}"
  on_event:
    url: "https://policy.example.com/mpc/on-event"
    secret: "${POLICY_WEBHOOK_SECRET}"
  timeout_ms: 5000
  fail_open: false

Every ceremony creation can fire a webhook to your infrastructure BEFORE the ceremony proceeds. Your endpoint returns { "allow": bool, "reason": "..." } and the agent enforces.

Request envelope:

POST <url>
Content-Type: application/json
x-mpc-webhook-timestamp: <unix-ms when the agent signed>
x-mpc-webhook-nonce:     <hex of 16 random bytes — anti-replay>
x-mpc-webhook-signature: <hex of HMAC-SHA256(secret, timestamp + "." + nonce + "." + body)>

{ "ceremony_kind": "sign", "key_id": "...", "curve": "Secp256k1", ... }

Verify the signature on your end — it pins the request to the timestamp + nonce + body and rejects replays. The fail_open flag controls behavior when the webhook is unreachable / times out:

  • fail_open: false (default, recommended): ceremonies fail closed when the policy endpoint is unreachable. Conservative — prefers "no signature" over "potentially unauthorized signature."
  • fail_open: true: ceremonies proceed when the policy endpoint is unreachable. Use only in dev / non-production deployments.

on_event is fire-and-forget — phase transitions to Complete or Failed fire this hook without blocking the ceremony.


permissions (optional)

permissions:
  rules: []   # Empty = open policy (allow all curves + operations the license allows).

  # Example: only allow secp256k1 DKG + sign:
  # rules:
  #   - allow:
  #       curves: ["Secp256k1"]
  #       operations: ["dkg", "sign"]

A second layer of allow-list on top of the license. Use to lock down a node further than the license allows — e.g., a node deployed in a region that should only sign for one curve.


log

log:
  level: info        # trace | debug | info | warn | error
  format: json       # json | pretty

  # Optional distributed-tracing span export (OTLP gRPC). When unset
  # (the default), no exporter is initialised and the agent runs with
  # stderr logs only.
  otlp_endpoint: "http://otel-collector:4317"
  otlp_service_name: "mpc-agent-prod"        # default: "mpc-agent"

JSON format is recommended for production (parseable by log aggregators); pretty is human-friendly for dev. Per-module verbosity can be tuned via the standard level setting; for finer control reach for environment-level log filters supported by your runtime.

Distributed tracing (OTLP)

When log.otlp_endpoint is set, the agent exports every span over OTLP gRPC to the configured collector. Pair with Jaeger, Tempo, Honeycomb, Datadog, or any OTLP-compatible backend.

Two operational benefits the metric scrape doesn't give:

  • Drill-down from a metric spike. When ceremony_duration_seconds p99 climbs, traces show which phase (DKG commit-collection, presigning sub-share exchange, sign partial-collection) ran long for a specific session id.
  • Per-handler latency attribution. Each ceremony handler emits a span; sub-spans on broker publish, DB write, and crypto operations reveal exactly where the wall-clock time went.

Minimal local setup with Jaeger via Docker:

# docker-compose.yml fragment — add alongside the agent service
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC (agent → collector)
    environment:
      COLLECTOR_OTLP_ENABLED: "true"

Then in the agent's config:

log:
  otlp_endpoint: "http://jaeger:4317"
  otlp_service_name: "mpc-agent-dev"

Open http://localhost:16686 and search for service mpc-agent-dev. Every incoming request shows up as a root span with nested handler / broker / DB spans, and inter-node ceremony envelopes carry W3C trace context so a multi-node ceremony collapses into a single trace across services.

Production collector pattern. Run an OpenTelemetry Collector sidecar (or DaemonSet on K8s) that batches + ships spans to your trace backend. The agent talks OTLP gRPC to the local collector; the collector handles backend-specific export, retry, and TLS. This keeps the agent's transport surface narrow: it only knows OTLP, not the dozen backend-specific protocols.

Cost note. OTLP export is gated by log.level (set to info to skip debug/trace spans). At Production-Large + Enterprise XL tiers, watch the collector's queue depth — sustained back-pressure means the collector itself needs scaling, not the agent.

Failure mode. If the collector is unreachable at agent startup, OTLP is silently disabled (a single eprintln! warns) and the agent continues with stderr logs only. The agent never fails to start because of a misconfigured collector — that would turn an observability tool into a deployment dependency, which is the wrong trade.


Performance and limits

Per-ceremony cost model

A few constants set the baseline for every sizing decision:

MetricValue (per agent node)Notes
Pre-signed sign() latency< 20 ms p50When the device-side presignature pool is warm, sign skips the presigning round-trip and only collects partials.
Cold sign() latency150-400 ms p50Includes inline presigning. Network-bound by the slowest cluster node.
DKG / reshare latency300-800 ms p50Two round-trips through the intra-cluster transport; bound by the slowest participating node.
Storage growth~2 KB / completed ceremonyAudit trail + session metadata. With session_retention_ms: 86400000 (24h) and 1k ceremonies/day, the DB grows by ~2 MB/day.
Cluster scale-outlinear with node count for read; threshold-bound for ceremoniesListing / fetching scales with replicas; ceremony throughput is set by the slowest participating node since DKG / presigning / reshare all need threshold-many parties at once.

Scaling tiers

The defaults ship with the Production-Small tier in mind (embedded-wallet workloads, low-thousands of ceremonies per day). For higher volume, tune the table below. Numbers are per-node unless flagged "cluster"; multiply for cluster-level throughput.

TierTarget volumePer-node hardwareKey config knobsSustained throughput
Dev / CIad-hoc1 vCPU, 512 MB RAM, 5 GB diskdefaults~50 ceremonies/min
Production-Small (defaults)up to 10k ceremonies/day, ~500 concurrent2 vCPU, 1 GB RAM, 10 GB diskmax_concurrent_sessions: 500 (default)workers: 32 (default)publish_channels: 4 (default)prefetch: 100 (default)~5 ceremonies/s steady-state
Production-Mediumup to 100k ceremonies/day, ~5k concurrent4 vCPU, 4 GB RAM, 25 GB diskmax_concurrent_sessions: 5000workers: 64publish_channels: 8prefetch: 200database.max_connections: 25~30 ceremonies/s steady-state
Production-Largeup to 1M ceremonies/day, ~50k concurrent8 vCPU, 16 GB RAM, 100 GB SSDmax_concurrent_sessions: 50000workers: 128publish_channels: 16prefetch: 500database.max_connections: 50session_retention_ms: 21600000 (6h)~200 ceremonies/s steady-state
Enterprise XLup to 10M ceremonies/day, ~100k concurrent16 vCPU, 64 GB RAM, NVMe SSD; postgres backend recommended over SQLitemax_concurrent_sessions: 100000workers: 256publish_channels: 32prefetch: 1000storage.backend: postgres (when supported)session_retention_ms: 3600000 (1h) + external archival~1000 ceremonies/s steady-state

Validated under load

The tier numbers above are the 24/7-sustained floor with headroom. To validate the agent holds its shape under burst load far above that floor, a single Production-Small-tier node (2 vCPU / 1 GB RAM, default config) was driven through the SDK's load-test harness against a 3-node local cluster.

Burst sweep — N parallel create() → sign() round-trips fired all at once against one node, then repeated at growing N:

Concurrent burstWall timeThroughputLock-wait p50Lock-wait p99Lock-wait max
200.16 s375 /s< 1 µs< 1 µs2.5 µs
500.22 s676 /s< 1 µs20 µs90 µs
1000.34 s888 /s< 1 µs< 1 µs40 µs
2000.82 s734 /s< 1 µs< 1 µs90 µs
5001.74 s861 /s< 1 µs< 1 µs310 µs
10003.67 s816 /s< 1 µs< 1 µs420 µs

Each row is a fresh burst of full DKG + presigning + signing — i.e. 3 × N ceremonies completing in the wall-time column. Throughput peaks around 100 concurrent (CPU-bound on dev hardware), stays flat through 1000 concurrent, and never surfaces a lock-wait p99 above 10 microseconds — meaning the per-session Mutex is not a bottleneck even at 50× the documented Production-Small ceiling.

End-to-end scenario suite — run in parallel against the same 3-node cluster, each verifying its signatures end-to-end (every signature checked against the signer's published public key):

ScenarioWhat it exercises
20 parallel create()Concurrent DKG kickoffs, signer-ID uniqueness, public-key derivation under contention
50 parallel sign() across 10 signersThe per-session lock and the worker pool under fan-in load (5 signs per signer)
100 sequential signs with auto-refilling presignature poolPool depletion → background refill → continued signing without depletion errors
5 parallel reshare() on the same keyIdCross-replica reshare contention: exactly one ceremony wins, the rest fail cleanly
10 parallel sign() on a 2-of-N multi-agent signerMulti-agent inbound envelope routing, threshold partial-signature aggregation under burst

Cluster aggregates after the full run (single node-1):

CounterValue
DKG ceremonies started / completed2,517 / 2,517
Presigning ceremonies started / completed2,862 / 2,862
Signing ceremonies started / completed2,809 / 2,809
Reshare ceremonies started / completed20 / 20
Ceremonies aborted (any rule)0
Envelopes dropped (any reason)0
Rate-limit rejections0
HTTP 5xx responses0
Lock-wait samples observed13,607
Lock-wait p99 across all samples1.25 µs

What this means for sizing. A correctly-configured node tolerates short bursts ~150× its sustained-throughput tier. The documented per-tier Sustained throughput column is the number to plan capacity against (it assumes load is roughly even across the day); burst absorption above that is built in. The metric to watch in production is mpc_agent_active_sessions against the max_concurrent_sessions cap — that's the saturating dimension, not raw CPU.

Production-Medium reference config

A drop-in config.yml snippet for the 100k/day tier:

rabbitmq:
  url: "amqp://mpc:mpc@broker:5672/%2F"
  prefetch: 200
  publish_channels: 8
  workers: 64

security:
  max_concurrent_sessions: 5000
  session_timeout_ms: 300000
  session_retention_ms: 43200000   # 12h — halves audit DB growth vs default
  session_sweep_interval_ms: 60000 # 1m — keep terminal-row count low

api:
  rate_limit:
    enabled: true
    max_per_minute: 5000           # for backend integrations doing batch ops

database:
  max_connections: 25

Production-Large reference config

For the 1M/day tier, the bottleneck shifts from the agent's worker pool to the broker. Run a 3-replica AMQP broker cluster (or use a managed broker — AWS MQ Multi-AZ, CloudAMQP "Production" plan) and raise the agent's pool sizes:

rabbitmq:
  url: "amqp://mpc:mpc@broker-1:5672,broker-2:5672,broker-3:5672/%2F"
  prefetch: 500
  publish_channels: 16
  workers: 128

security:
  max_concurrent_sessions: 50000
  session_timeout_ms: 300000
  session_retention_ms: 21600000   # 6h
  session_sweep_interval_ms: 30000 # 30s

api:
  rate_limit:
    enabled: true
    max_per_minute: 50000

database:
  max_connections: 50

Observability — knowing which knob to turn

The agent exposes Prometheus metrics on GET /metrics (auth-exempt, same as /health and /ready). Scrape every 15-30 s and wire to Grafana / Datadog / your alerting stack. Five metrics map directly to the scaling-tier table above:

MetricTypeWhat it tells youWhen to act
mpc_agent_active_sessionsgaugeIn-flight ceremony countClimbing toward security.max_concurrent_sessions → raise the cap or scale out. Sustained >80% utilisation = you're one burst away from rejecting requests with HTTP 429.
mpc_agent_ceremony_completed_total{kind,curve}counterThroughput per ceremony kindrate(mpc_agent_ceremony_completed_total[5m]) is your sustained ceremonies/second. Cross-check against the "Sustained throughput" column in the scaling-tier table.
mpc_agent_ceremony_duration_seconds{kind,curve}histogramPer-ceremony wall-clock latencyp99 climbing alongside throughput indicates a saturated dimension — usually broker (publish_duration_seconds will also climb) or worker pool.
mpc_agent_publish_duration_secondshistogramAMQP basic_publish + confirm round-tripp99 > 50 ms = the publish pool is undersized for the load. Bump rabbitmq.publish_channels from 4 → 8 → 16.
mpc_agent_envelopes_received_total{kind}counterInbound envelope volume by message kindCross-check against envelopes_dropped_total per reason to spot rate-limit drops, untrusted-peer drops, or replay attempts.
mpc_agent_worker_pool_sizegaugeStatic worker count (from rabbitmq.workers)Constant — useful only as the denominator in the saturation panel below.
mpc_agent_worker_queue_depthgaugeEnvelopes queued between the AMQP consumer dispatch and the worker poolPersistent non-zero → worker pool saturated. Bump rabbitmq.workers (default 32; double per tier). Spike-then-drain during bursts is normal; sustained queue depth >0 for >1 minute is the action signal.
mpc_agent_http_requests_total{method,route,status_class}counterPer-route HTTP request volume, split by 2xx / 4xx / 5xxrate(...{status_class="4xx"}[5m]) climbing = caller is misusing the API (auth failures, validation errors). rate(...{status_class="5xx"}[5m]) climbing = the cluster itself is unhealthy — pair with ceremony_aborted_total for attribution.
mpc_agent_http_request_duration_seconds{method,route}histogramPer-route end-to-end HTTP latencyp99 on the /presignatures/:id/sign route is the most user-visible signal — climbing here means agent saturation, not network.
mpc_agent_db_pool_size / mpc_agent_db_pool_idlegaugesDatabase connection pool size + idle countdb_pool_idle / db_pool_size near 0 sustained → pool is fully busy, raise database.max_connections. The docs' scaling tiers bump this from 10 (default) to 25 (Medium), 50 (Large).
mpc_agent_session_lock_wait_secondshistogramTime waiting to acquire the per-session Mutex before processing an envelopeSub-microsecond at low load (p99 typically < 10µs). Climbing p99 = a hot ceremony has many concurrent inbound envelopes serialising on its session lock. There's no agent-side fix without splitting the session state machine; the practical mitigation is caller-side back-pressure (don't fire 100 parallel signs on the same LoadedKeyShare handle).

The full registry — including aborts (mpc_agent_ceremony_aborted_total{kind,rule}), drops (mpc_agent_envelopes_dropped_total{reason}), and rate-limit rejections (mpc_agent_rate_limited_total) — is emitted by the same /metrics endpoint and discoverable by scraping # HELP lines.

A Grafana panel that lights up the scaling-tier match:

# Throughput at this node (compare to "Sustained throughput" column)
sum by (kind) (rate(mpc_agent_ceremony_completed_total[5m]))

# Session-cap utilisation (alert when > 0.8)
mpc_agent_active_sessions / on() group_left() <max_concurrent_sessions_const>

# Publish-pool saturation indicator (alert when p99 > 50 ms)
histogram_quantile(0.99, rate(mpc_agent_publish_duration_seconds_bucket[5m]))

# Ceremony p99 latency by kind
histogram_quantile(0.99, sum by (le, kind) (rate(mpc_agent_ceremony_duration_seconds_bucket[5m])))

# Worker-pool saturation (alert when sustained > 0.5)
mpc_agent_worker_queue_depth / mpc_agent_worker_pool_size

# HTTP error rate by route (alert when 5xx rate > 0)
sum by (route) (rate(mpc_agent_http_requests_total{status_class="5xx"}[5m]))

# Sign-path p99 latency (the most user-visible signal)
histogram_quantile(0.99,
  rate(mpc_agent_http_request_duration_seconds_bucket{route="/presignatures/:id/sign"}[5m])
)

# Database pool saturation (alert when sustained < 0.2)
mpc_agent_db_pool_idle / mpc_agent_db_pool_size

Tuning notes

  • Channel pool vs worker pool. publish_channels and workers tune different dimensions: the publish pool sets the outbound concurrency (how many AMQP basic_publish round-trips can be in flight at once), the worker pool sets the inbound concurrency (how many incoming envelopes the agent processes in parallel). Both grew bottleneck-bound in earlier builds — the publish side because every send serialised on a single channel mutex, the worker side because of an artificial min(prefetch, 32) cap. Both caps are gone; raise the knobs as the host can support.
  • Worker pool sizing. Start at 2-4× host vCPU. The work each worker does is mostly I/O-bound (AMQP round-trips, DB writes) so the pool can exceed vCPU count without thrashing.
  • session_retention_ms is the audit DB growth knob. At 1M ceremonies/day with the default 24h retention, terminal-session rows hold ~2 GB; dropping to 6h holds ~500 MB. Pair short retention with an external archival pipeline (the sweeper exposes the deleted rows via the audit log before purging) if you need long-term forensics.
  • SQLite ceiling. SQLite tops out around ~1-5k writes/second per database file because of the WAL serialisation. At Production-Large+ volumes plan for a postgres / managed-storage backend (config field exists; native postgres backend lands when needed).
  • Sub-millisecond per-publish wins compound. At 1M ceremonies/day with 2 inbound + 2 outbound envelopes per ceremony, the cluster moves ~50 envelopes/second average — but bursts hit 5-10× that. Channel pool + worker pool sizing matters more than peak vCPU.

Configuration validation

The agent validates the config at startup and refuses to boot on any inconsistency. Common errors:

ErrorCause
license: invalid signatureThe certificate isn't signed by the issuer key this build was compiled against. Check that you're using the right build for the right environment (dev cert + dev binary, prod cert + prod binary).
identity: private_key_hex must be 32 bytesWrong key format. Use agent gen-secret to generate.
intra-cluster transport: connection refusedThe broker isn't reachable. Check the URL, port, and the broker's container state. The agent retries connect on a backoff so transient broker outages don't kill the process.
api: port already in useAnother process is bound to api.port. Change the port or stop the conflicting process.

The full validation runs as a dedicated subcommand:

agent validate-config --config /etc/mpc-agent/config.yml

Prints a config summary and exits 0 on success; exits 1 with the validation error on failure. Wire it into your deploy pipeline so a broken config never reaches a running node.


OpenAPI / Swagger UI

The agent serves an interactive OpenAPI spec and Swagger UI for its REST API — useful when you're driving the agent over raw HTTP rather than through the SDK. It is disabled by default; enable it explicitly:

swagger:
  enabled: true            # default false
  path: /swagger-ui        # UI path; raw spec served at <path>/openapi.json
  # auth:                  # optional HTTP Basic Auth for the UI only
  #   username: "admin"
  #   password: "${SWAGGER_PASSWORD}"

With enabled: true, the Swagger UI is at /swagger-ui and the raw spec at /swagger-ui/openapi.json. The spec documents every route, request/response schema, and the authentication header — treat it as the source of truth for direct HTTP integration. API routes always require the configured API key or JWT regardless of this setting; swagger.auth only gates access to the UI itself.


Next

On this page