Configuration
Every MPC Agent config knob — identity, license, intra-cluster transport, storage, API auth, rate limits, policy webhooks, performance & limits.
Configuration
The agent reads a YAML config file at startup. Pass its path with --config; when omitted it defaults to config.yml in the working directory. Every field below maps to a top-level key in that file. Env-var interpolation via ${VAR_NAME} is supported in any string value, so secrets stay out of the config file and in your secret store.
agent start --config /etc/mpc-agent/config.ymlA starter config you can drop in and customize:
node:
id: "node-1"
display_name: "MPC Node 1"
identity:
private_key_hex: "${NODE_PRIVATE_KEY}"
allow_dynamic_nodes: false
trusted_peers: []
license:
certificate: "${MPC_LICENSE_CERT}"
rabbitmq:
url: "amqp://mpc:mpc@broker:5672/%2F"
database:
url: "sqlite:///app/data/agent.db?mode=rwc"
api:
host: "0.0.0.0"
port: 3000
auth:
auth_type: api_key
api_keys:
- "${API_KEY_1}"
rate_limit:
enabled: true
max_per_minute: 300
security:
session_timeout_ms: 300000
session_expiry_window_ms: 30000
max_concurrent_sessions: 200
ack_timeout_ms: 60000
session_retention_ms: 86400000
session_sweep_interval_ms: 300000
log:
level: info
format: jsonnode
node:
id: "node-1" # REQUIRED — stable identifier, used in cluster routing and logs.
display_name: "MPC Node 1" # Optional, surfaces in /node/info responses.
index: # Optional. Auto-derived from peer list on each request.The id is the agent's stable identity within the cluster. It must be unique across every node in the same logical cluster (the cluster sharing one license, one intra-cluster transport, one ceremony session-id namespace). Changing it after the first DKG breaks every wallet that node was a party to — treat it as immutable.
index is normally auto-detected from each ceremony request's peer list (matched by node_id). Set explicitly only if you're debugging an asymmetric mis-assignment.
identity
identity:
private_key_hex: "${NODE_PRIVATE_KEY}" # REQUIRED. 32-byte Ed25519 seed, hex-encoded.
allow_dynamic_nodes: false # If true, accept any node presenting a valid signature.
trusted_peers: # Hex Ed25519 pubkeys; ignored when allow_dynamic_nodes=true.
- "<node-1-public-key-hex>" # include THIS node's own key (see note below)
- "<node-2-public-key-hex>"
- "<node-3-public-key-hex>"Generate the seed with agent gen-secret. It prints two labelled lines — Private key : <hex> and Public key : <hex>. Use the private hex for private_key_hex and the public hex when pinning this node in other nodes' trusted_peers. Each node MUST have its own seed — sharing seeds across nodes breaks cluster identity.
allow_dynamic_nodes: true accepts any node that presents a valid Ed25519 signature — useful for permissive dev environments where you spin nodes up and down freely. In production, set false and pin the public keys of every node you trust under trusted_peers.
:::warning
When you set trusted_peers to a non-empty list, it must include this node's own public key alongside the other nodes'. Each node sees its own ceremony messages as well as its peers', so a list that omits the node's own key causes it to reject its own messages and ceremonies stall. The two safe options are: list every node including this one, or leave trusted_peers: [] empty (which disables pinning entirely — every node presenting a valid signature is accepted, equivalent to allow_dynamic_nodes: true for trust purposes).
:::
license
license:
certificate: "${MPC_LICENSE_CERT}"The certificate is an issuer-signed bundle that pins:
- The licensee identity (your organization).
- The allowed curves (
Secp256k1,Ed25519, etc.). - The allowed ceremony kinds (DKG, presigning, sign, reshare, sign-pack export).
The agent verifies the cert at startup and refuses every ceremony if it can't. The cert is public-facing material — fine to put in a config file or env var.
rabbitmq
rabbitmq:
url: "amqp://mpc:mpc@broker:5672/%2F"
exchange: "mpc.ceremony" # Optional. Defaults to "mpc.ceremony".
# Optional throughput tuning. Defaults are fine for production workloads
# up to ~10k ceremonies/day per node; raise the pool sizes for higher
# volume. See the "Scaling tiers" section below for concrete recipes.
prefetch: 100 # Per-consumer prefetch. Caps unacked-in-flight on the broker side.
publish_channels: 4 # AMQP channel pool for parallel outbound publishes.
workers: 32 # Worker pool size for inbound envelope dispatch.Ceremony envelopes between cluster nodes flow over a message broker. The agent supports any AMQP 0.9.1 broker (any production AMQP broker works — managed alternatives like AWS MQ or CloudAMQP work without changes).
publish_channels controls how many AMQP channels the agent opens for outbound publishes. Each channel is independent at the AMQP layer — confirms on one don't block sends on another, so N channels give roughly N-way parallelism on publish throughput. Default 4 is enough for typical production loads; raise to 16-32 for high-volume deployments. the broker's per-connection channel cap (default 2047) is the hard ceiling.
workers is the inbound dispatch pool size. Each worker pulls envelopes from a shared queue and runs the per-envelope handler (commitment / share / partial-signature processing). Increasing this raises the agent's ability to handle concurrent in-flight ceremonies; the practical limit is the host's CPU + memory budget. The earlier hard cap of min(prefetch, 32) is gone — set as high as your hardware supports.
prefetch tunes the broker side: how many unacknowledged messages the broker will deliver to this consumer at once. Keep it ≥ workers so the worker pool isn't starved waiting for the broker.
Production sizing. A single broker instance handles tens of thousands of ceremonies per day without tuning. For higher volume, run a clustered broker (3 replicas is the standard pattern) or use a managed broker. The agent reconnects on backoff if the broker goes down — transient outages don't kill the process.
Security. The broker is intra-cluster only — do NOT expose it to the public internet. Treat the connection string as a secret. Inside the cluster, every envelope is end-to-end X25519-sealed before it ever hits the broker, but firewall the broker port anyway as defense-in-depth.
database
database:
url: "sqlite:///app/data/agent.db?mode=rwc" # REQUIRED for persistence.
max_connections: 10 # Optional. Default 10.The database block is what turns persistence on. With it set, the agent persists ceremony state, key shares, presignatures, and audit logs, and runs schema migrations at startup. Omit it entirely and the agent runs stateless — nothing is persisted and agent.keyShares.list() returns an error, so DKG output can't be looked up. A persistent database is required for any real deployment.
Three URL schemes are accepted:
| Scheme | Example |
|---|---|
| SQLite | sqlite:///app/data/agent.db?mode=rwc |
| PostgreSQL | postgres://user:pass@host:5432/dbname |
| MySQL | mysql://user:pass@host:3306/dbname |
For SQLite, the directory in the path (/app/data above) must already exist and be writable by the agent — mode=rwc creates the database file but not its parent directory. The published image ships a writable data directory at /app/data; point the SQLite URL there (and mount your volume at /app/data) and it works without extra setup. Sensitive fields (key shares, presignatures) are encrypted at rest with a key derived from the node identity — no separate key management is required.
Backup. Snapshot the database (or the SQLite file's directory) regularly. Restoring on a new host requires the same identity.private_key_hex and the same license.certificate — both are bound into the at-rest encryption, so restoring one node's data onto a different node's identity fails to decrypt.
storage
storage:
backend: db # db (default) | file | s3Controls where encrypted key-share blobs live. The default db backend stores them inline in the database above — most deployments need nothing here. Set backend: file (with file.path) to keep blobs as one encrypted file per share on a local volume, or backend: s3 for an S3-compatible object store. All backends still require the database block for metadata; storage is not a substitute for database.
api
api:
host: "0.0.0.0"
port: 3000
tls: # Optional. Omit when terminating at a load balancer.
source: file # file | secrets_manager
cert_path: "/certs/server.pem"
key_path: "/certs/server.key"
# Or, AWS Secrets Manager (secret must be JSON: {"cert_pem":"...","key_pem":"..."}):
# source: secrets_manager
# secret_id: "arn:aws:secretsmanager:us-east-1:123:secret:mpc-agent-tls"
# region: "us-east-1"
auth:
auth_type: api_key # api_key | jwt
api_keys: # When auth_type: api_key
- "${API_KEY_1}"
# When auth_type: jwt:
# auth_type: jwt
# jwt_secret: "${JWT_SECRET}"
# (Tokens must contain a "sub" claim and be signed with the secret.)
rate_limit:
enabled: true
max_per_minute: 300 # Per API key.The agent's HTTPS interface. Every request is authenticated (no anonymous access). Pick auth_type: api_key for the simplest setup; configure multiple keys to rotate without downtime.
For production, terminate TLS upstream (ACM + ALB on EKS is the canonical pattern) and leave api.tls unconfigured. The internal HTTP between the LB and the agent is fine because it never leaves your VPC. If you must terminate TLS on the agent itself, the tls.source: file path is standard; the secrets_manager source is a convenience for AWS deployments where cert rotation runs through Secrets Manager.
rate_limit.max_per_minute is per API key. The default 300/min is sized for individual-device clients; raise it for backend integrations that batch thousands of operations.
security
security:
session_timeout_ms: 300000 # 5 min — how long a ceremony stays active before timing out.
session_expiry_window_ms: 30000 # 30s grace period after timeout before the session is GC'd.
max_concurrent_sessions: 200 # Per-node concurrent session ceiling.
ack_timeout_ms: 60000 # 60s to collect all acks before failing the ceremony.
session_retention_ms: 86400000 # 24h — how long terminal sessions stay in the DB. Set 0 to disable retention.
session_sweep_interval_ms: 300000 # 5 min — how often the retention sweeper runs.These knobs control ceremony liveness, GC, and DB growth. Defaults are tuned for embedded-wallet usage (5-minute ceremonies, 200 concurrent users per node, 24h audit trail).
For high-throughput backends, raise max_concurrent_sessions (it gates concurrency at the session-state-machine layer with near-zero memory overhead per session). For audit-heavy environments, raise session_retention_ms and let the DB grow.
session_sweep_interval_ms: 0 disables the sweeper, leaving terminal rows in the DB forever. Useful when you have an external archival pipeline that pulls terminal sessions to long-term storage; otherwise leave the default.
policy_webhooks (optional)
policy_webhooks:
pre_dkg:
url: "https://policy.example.com/mpc/pre-dkg"
secret: "${POLICY_WEBHOOK_SECRET}"
pre_sign:
url: "https://policy.example.com/mpc/pre-sign"
secret: "${POLICY_WEBHOOK_SECRET}"
on_event:
url: "https://policy.example.com/mpc/on-event"
secret: "${POLICY_WEBHOOK_SECRET}"
timeout_ms: 5000
fail_open: falseEvery ceremony creation can fire a webhook to your infrastructure BEFORE the ceremony proceeds. Your endpoint returns { "allow": bool, "reason": "..." } and the agent enforces.
Request envelope:
POST <url>
Content-Type: application/json
x-mpc-webhook-timestamp: <unix-ms when the agent signed>
x-mpc-webhook-nonce: <hex of 16 random bytes — anti-replay>
x-mpc-webhook-signature: <hex of HMAC-SHA256(secret, timestamp + "." + nonce + "." + body)>
{ "ceremony_kind": "sign", "key_id": "...", "curve": "Secp256k1", ... }
Verify the signature on your end — it pins the request to the timestamp + nonce + body and rejects replays. The fail_open flag controls behavior when the webhook is unreachable / times out:
fail_open: false(default, recommended): ceremonies fail closed when the policy endpoint is unreachable. Conservative — prefers "no signature" over "potentially unauthorized signature."fail_open: true: ceremonies proceed when the policy endpoint is unreachable. Use only in dev / non-production deployments.
on_event is fire-and-forget — phase transitions to Complete or Failed fire this hook without blocking the ceremony.
permissions (optional)
permissions:
rules: [] # Empty = open policy (allow all curves + operations the license allows).
# Example: only allow secp256k1 DKG + sign:
# rules:
# - allow:
# curves: ["Secp256k1"]
# operations: ["dkg", "sign"]A second layer of allow-list on top of the license. Use to lock down a node further than the license allows — e.g., a node deployed in a region that should only sign for one curve.
log
log:
level: info # trace | debug | info | warn | error
format: json # json | pretty
# Optional distributed-tracing span export (OTLP gRPC). When unset
# (the default), no exporter is initialised and the agent runs with
# stderr logs only.
otlp_endpoint: "http://otel-collector:4317"
otlp_service_name: "mpc-agent-prod" # default: "mpc-agent"JSON format is recommended for production (parseable by log aggregators); pretty is human-friendly for dev. Per-module verbosity can be tuned via the standard level setting; for finer control reach for environment-level log filters supported by your runtime.
Distributed tracing (OTLP)
When log.otlp_endpoint is set, the agent exports every span over OTLP gRPC to the configured collector. Pair with Jaeger, Tempo, Honeycomb, Datadog, or any OTLP-compatible backend.
Two operational benefits the metric scrape doesn't give:
- Drill-down from a metric spike. When
ceremony_duration_secondsp99 climbs, traces show which phase (DKG commit-collection, presigning sub-share exchange, sign partial-collection) ran long for a specific session id. - Per-handler latency attribution. Each ceremony handler emits a span; sub-spans on broker publish, DB write, and crypto operations reveal exactly where the wall-clock time went.
Minimal local setup with Jaeger via Docker:
# docker-compose.yml fragment — add alongside the agent service
services:
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686" # Jaeger UI
- "4317:4317" # OTLP gRPC (agent → collector)
environment:
COLLECTOR_OTLP_ENABLED: "true"Then in the agent's config:
log:
otlp_endpoint: "http://jaeger:4317"
otlp_service_name: "mpc-agent-dev"Open http://localhost:16686 and search for service mpc-agent-dev. Every incoming request shows up as a root span with nested handler / broker / DB spans, and inter-node ceremony envelopes carry W3C trace context so a multi-node ceremony collapses into a single trace across services.
Production collector pattern. Run an OpenTelemetry Collector sidecar (or DaemonSet on K8s) that batches + ships spans to your trace backend. The agent talks OTLP gRPC to the local collector; the collector handles backend-specific export, retry, and TLS. This keeps the agent's transport surface narrow: it only knows OTLP, not the dozen backend-specific protocols.
Cost note. OTLP export is gated by log.level (set to info to skip debug/trace spans). At Production-Large + Enterprise XL tiers, watch the collector's queue depth — sustained back-pressure means the collector itself needs scaling, not the agent.
Failure mode. If the collector is unreachable at agent startup, OTLP is silently disabled (a single eprintln! warns) and the agent continues with stderr logs only. The agent never fails to start because of a misconfigured collector — that would turn an observability tool into a deployment dependency, which is the wrong trade.
Performance and limits
Per-ceremony cost model
A few constants set the baseline for every sizing decision:
| Metric | Value (per agent node) | Notes |
|---|---|---|
Pre-signed sign() latency | < 20 ms p50 | When the device-side presignature pool is warm, sign skips the presigning round-trip and only collects partials. |
Cold sign() latency | 150-400 ms p50 | Includes inline presigning. Network-bound by the slowest cluster node. |
| DKG / reshare latency | 300-800 ms p50 | Two round-trips through the intra-cluster transport; bound by the slowest participating node. |
| Storage growth | ~2 KB / completed ceremony | Audit trail + session metadata. With session_retention_ms: 86400000 (24h) and 1k ceremonies/day, the DB grows by ~2 MB/day. |
| Cluster scale-out | linear with node count for read; threshold-bound for ceremonies | Listing / fetching scales with replicas; ceremony throughput is set by the slowest participating node since DKG / presigning / reshare all need threshold-many parties at once. |
Scaling tiers
The defaults ship with the Production-Small tier in mind (embedded-wallet workloads, low-thousands of ceremonies per day). For higher volume, tune the table below. Numbers are per-node unless flagged "cluster"; multiply for cluster-level throughput.
| Tier | Target volume | Per-node hardware | Key config knobs | Sustained throughput |
|---|---|---|---|---|
| Dev / CI | ad-hoc | 1 vCPU, 512 MB RAM, 5 GB disk | defaults | ~50 ceremonies/min |
| Production-Small (defaults) | up to 10k ceremonies/day, ~500 concurrent | 2 vCPU, 1 GB RAM, 10 GB disk | max_concurrent_sessions: 500 (default)workers: 32 (default)publish_channels: 4 (default)prefetch: 100 (default) | ~5 ceremonies/s steady-state |
| Production-Medium | up to 100k ceremonies/day, ~5k concurrent | 4 vCPU, 4 GB RAM, 25 GB disk | max_concurrent_sessions: 5000workers: 64publish_channels: 8prefetch: 200database.max_connections: 25 | ~30 ceremonies/s steady-state |
| Production-Large | up to 1M ceremonies/day, ~50k concurrent | 8 vCPU, 16 GB RAM, 100 GB SSD | max_concurrent_sessions: 50000workers: 128publish_channels: 16prefetch: 500database.max_connections: 50session_retention_ms: 21600000 (6h) | ~200 ceremonies/s steady-state |
| Enterprise XL | up to 10M ceremonies/day, ~100k concurrent | 16 vCPU, 64 GB RAM, NVMe SSD; postgres backend recommended over SQLite | max_concurrent_sessions: 100000workers: 256publish_channels: 32prefetch: 1000storage.backend: postgres (when supported)session_retention_ms: 3600000 (1h) + external archival | ~1000 ceremonies/s steady-state |
Validated under load
The tier numbers above are the 24/7-sustained floor with headroom. To validate the agent holds its shape under burst load far above that floor, a single Production-Small-tier node (2 vCPU / 1 GB RAM, default config) was driven through the SDK's load-test harness against a 3-node local cluster.
Burst sweep — N parallel create() → sign() round-trips fired all at once against one node, then repeated at growing N:
| Concurrent burst | Wall time | Throughput | Lock-wait p50 | Lock-wait p99 | Lock-wait max |
|---|---|---|---|---|---|
| 20 | 0.16 s | 375 /s | < 1 µs | < 1 µs | 2.5 µs |
| 50 | 0.22 s | 676 /s | < 1 µs | 20 µs | 90 µs |
| 100 | 0.34 s | 888 /s | < 1 µs | < 1 µs | 40 µs |
| 200 | 0.82 s | 734 /s | < 1 µs | < 1 µs | 90 µs |
| 500 | 1.74 s | 861 /s | < 1 µs | < 1 µs | 310 µs |
| 1000 | 3.67 s | 816 /s | < 1 µs | < 1 µs | 420 µs |
Each row is a fresh burst of full DKG + presigning + signing — i.e. 3 × N ceremonies completing in the wall-time column. Throughput peaks around 100 concurrent (CPU-bound on dev hardware), stays flat through 1000 concurrent, and never surfaces a lock-wait p99 above 10 microseconds — meaning the per-session Mutex is not a bottleneck even at 50× the documented Production-Small ceiling.
End-to-end scenario suite — run in parallel against the same 3-node cluster, each verifying its signatures end-to-end (every signature checked against the signer's published public key):
| Scenario | What it exercises |
|---|---|
20 parallel create() | Concurrent DKG kickoffs, signer-ID uniqueness, public-key derivation under contention |
50 parallel sign() across 10 signers | The per-session lock and the worker pool under fan-in load (5 signs per signer) |
| 100 sequential signs with auto-refilling presignature pool | Pool depletion → background refill → continued signing without depletion errors |
5 parallel reshare() on the same keyId | Cross-replica reshare contention: exactly one ceremony wins, the rest fail cleanly |
10 parallel sign() on a 2-of-N multi-agent signer | Multi-agent inbound envelope routing, threshold partial-signature aggregation under burst |
Cluster aggregates after the full run (single node-1):
| Counter | Value |
|---|---|
| DKG ceremonies started / completed | 2,517 / 2,517 |
| Presigning ceremonies started / completed | 2,862 / 2,862 |
| Signing ceremonies started / completed | 2,809 / 2,809 |
| Reshare ceremonies started / completed | 20 / 20 |
| Ceremonies aborted (any rule) | 0 |
| Envelopes dropped (any reason) | 0 |
| Rate-limit rejections | 0 |
| HTTP 5xx responses | 0 |
| Lock-wait samples observed | 13,607 |
| Lock-wait p99 across all samples | 1.25 µs |
What this means for sizing. A correctly-configured node tolerates short bursts ~150× its sustained-throughput tier. The documented per-tier Sustained throughput column is the number to plan capacity against (it assumes load is roughly even across the day); burst absorption above that is built in. The metric to watch in production is mpc_agent_active_sessions against the max_concurrent_sessions cap — that's the saturating dimension, not raw CPU.
Production-Medium reference config
A drop-in config.yml snippet for the 100k/day tier:
rabbitmq:
url: "amqp://mpc:mpc@broker:5672/%2F"
prefetch: 200
publish_channels: 8
workers: 64
security:
max_concurrent_sessions: 5000
session_timeout_ms: 300000
session_retention_ms: 43200000 # 12h — halves audit DB growth vs default
session_sweep_interval_ms: 60000 # 1m — keep terminal-row count low
api:
rate_limit:
enabled: true
max_per_minute: 5000 # for backend integrations doing batch ops
database:
max_connections: 25Production-Large reference config
For the 1M/day tier, the bottleneck shifts from the agent's worker pool to the broker. Run a 3-replica AMQP broker cluster (or use a managed broker — AWS MQ Multi-AZ, CloudAMQP "Production" plan) and raise the agent's pool sizes:
rabbitmq:
url: "amqp://mpc:mpc@broker-1:5672,broker-2:5672,broker-3:5672/%2F"
prefetch: 500
publish_channels: 16
workers: 128
security:
max_concurrent_sessions: 50000
session_timeout_ms: 300000
session_retention_ms: 21600000 # 6h
session_sweep_interval_ms: 30000 # 30s
api:
rate_limit:
enabled: true
max_per_minute: 50000
database:
max_connections: 50Observability — knowing which knob to turn
The agent exposes Prometheus metrics on GET /metrics (auth-exempt, same as /health and /ready). Scrape every 15-30 s and wire to Grafana / Datadog / your alerting stack. Five metrics map directly to the scaling-tier table above:
| Metric | Type | What it tells you | When to act |
|---|---|---|---|
mpc_agent_active_sessions | gauge | In-flight ceremony count | Climbing toward security.max_concurrent_sessions → raise the cap or scale out. Sustained >80% utilisation = you're one burst away from rejecting requests with HTTP 429. |
mpc_agent_ceremony_completed_total{kind,curve} | counter | Throughput per ceremony kind | rate(mpc_agent_ceremony_completed_total[5m]) is your sustained ceremonies/second. Cross-check against the "Sustained throughput" column in the scaling-tier table. |
mpc_agent_ceremony_duration_seconds{kind,curve} | histogram | Per-ceremony wall-clock latency | p99 climbing alongside throughput indicates a saturated dimension — usually broker (publish_duration_seconds will also climb) or worker pool. |
mpc_agent_publish_duration_seconds | histogram | AMQP basic_publish + confirm round-trip | p99 > 50 ms = the publish pool is undersized for the load. Bump rabbitmq.publish_channels from 4 → 8 → 16. |
mpc_agent_envelopes_received_total{kind} | counter | Inbound envelope volume by message kind | Cross-check against envelopes_dropped_total per reason to spot rate-limit drops, untrusted-peer drops, or replay attempts. |
mpc_agent_worker_pool_size | gauge | Static worker count (from rabbitmq.workers) | Constant — useful only as the denominator in the saturation panel below. |
mpc_agent_worker_queue_depth | gauge | Envelopes queued between the AMQP consumer dispatch and the worker pool | Persistent non-zero → worker pool saturated. Bump rabbitmq.workers (default 32; double per tier). Spike-then-drain during bursts is normal; sustained queue depth >0 for >1 minute is the action signal. |
mpc_agent_http_requests_total{method,route,status_class} | counter | Per-route HTTP request volume, split by 2xx / 4xx / 5xx | rate(...{status_class="4xx"}[5m]) climbing = caller is misusing the API (auth failures, validation errors). rate(...{status_class="5xx"}[5m]) climbing = the cluster itself is unhealthy — pair with ceremony_aborted_total for attribution. |
mpc_agent_http_request_duration_seconds{method,route} | histogram | Per-route end-to-end HTTP latency | p99 on the /presignatures/:id/sign route is the most user-visible signal — climbing here means agent saturation, not network. |
mpc_agent_db_pool_size / mpc_agent_db_pool_idle | gauges | Database connection pool size + idle count | db_pool_idle / db_pool_size near 0 sustained → pool is fully busy, raise database.max_connections. The docs' scaling tiers bump this from 10 (default) to 25 (Medium), 50 (Large). |
mpc_agent_session_lock_wait_seconds | histogram | Time waiting to acquire the per-session Mutex before processing an envelope | Sub-microsecond at low load (p99 typically < 10µs). Climbing p99 = a hot ceremony has many concurrent inbound envelopes serialising on its session lock. There's no agent-side fix without splitting the session state machine; the practical mitigation is caller-side back-pressure (don't fire 100 parallel signs on the same LoadedKeyShare handle). |
The full registry — including aborts (mpc_agent_ceremony_aborted_total{kind,rule}), drops (mpc_agent_envelopes_dropped_total{reason}), and rate-limit rejections (mpc_agent_rate_limited_total) — is emitted by the same /metrics endpoint and discoverable by scraping # HELP lines.
A Grafana panel that lights up the scaling-tier match:
# Throughput at this node (compare to "Sustained throughput" column)
sum by (kind) (rate(mpc_agent_ceremony_completed_total[5m]))
# Session-cap utilisation (alert when > 0.8)
mpc_agent_active_sessions / on() group_left() <max_concurrent_sessions_const>
# Publish-pool saturation indicator (alert when p99 > 50 ms)
histogram_quantile(0.99, rate(mpc_agent_publish_duration_seconds_bucket[5m]))
# Ceremony p99 latency by kind
histogram_quantile(0.99, sum by (le, kind) (rate(mpc_agent_ceremony_duration_seconds_bucket[5m])))
# Worker-pool saturation (alert when sustained > 0.5)
mpc_agent_worker_queue_depth / mpc_agent_worker_pool_size
# HTTP error rate by route (alert when 5xx rate > 0)
sum by (route) (rate(mpc_agent_http_requests_total{status_class="5xx"}[5m]))
# Sign-path p99 latency (the most user-visible signal)
histogram_quantile(0.99,
rate(mpc_agent_http_request_duration_seconds_bucket{route="/presignatures/:id/sign"}[5m])
)
# Database pool saturation (alert when sustained < 0.2)
mpc_agent_db_pool_idle / mpc_agent_db_pool_size
Tuning notes
- Channel pool vs worker pool.
publish_channelsandworkerstune different dimensions: the publish pool sets the outbound concurrency (how many AMQPbasic_publishround-trips can be in flight at once), the worker pool sets the inbound concurrency (how many incoming envelopes the agent processes in parallel). Both grew bottleneck-bound in earlier builds — the publish side because every send serialised on a single channel mutex, the worker side because of an artificialmin(prefetch, 32)cap. Both caps are gone; raise the knobs as the host can support. - Worker pool sizing. Start at 2-4× host vCPU. The work each worker does is mostly I/O-bound (AMQP round-trips, DB writes) so the pool can exceed vCPU count without thrashing.
session_retention_msis the audit DB growth knob. At 1M ceremonies/day with the default 24h retention, terminal-session rows hold ~2 GB; dropping to 6h holds ~500 MB. Pair short retention with an external archival pipeline (the sweeper exposes the deleted rows via the audit log before purging) if you need long-term forensics.- SQLite ceiling. SQLite tops out around ~1-5k writes/second per database file because of the WAL serialisation. At Production-Large+ volumes plan for a postgres / managed-storage backend (config field exists; native postgres backend lands when needed).
- Sub-millisecond per-publish wins compound. At 1M ceremonies/day with 2 inbound + 2 outbound envelopes per ceremony, the cluster moves ~50 envelopes/second average — but bursts hit 5-10× that. Channel pool + worker pool sizing matters more than peak vCPU.
Configuration validation
The agent validates the config at startup and refuses to boot on any inconsistency. Common errors:
| Error | Cause |
|---|---|
license: invalid signature | The certificate isn't signed by the issuer key this build was compiled against. Check that you're using the right build for the right environment (dev cert + dev binary, prod cert + prod binary). |
identity: private_key_hex must be 32 bytes | Wrong key format. Use agent gen-secret to generate. |
intra-cluster transport: connection refused | The broker isn't reachable. Check the URL, port, and the broker's container state. The agent retries connect on a backoff so transient broker outages don't kill the process. |
api: port already in use | Another process is bound to api.port. Change the port or stop the conflicting process. |
The full validation runs as a dedicated subcommand:
agent validate-config --config /etc/mpc-agent/config.ymlPrints a config summary and exits 0 on success; exits 1 with the validation error on failure. Wire it into your deploy pipeline so a broken config never reaches a running node.
OpenAPI / Swagger UI
The agent serves an interactive OpenAPI spec and Swagger UI for its REST API — useful when you're driving the agent over raw HTTP rather than through the SDK. It is disabled by default; enable it explicitly:
swagger:
enabled: true # default false
path: /swagger-ui # UI path; raw spec served at <path>/openapi.json
# auth: # optional HTTP Basic Auth for the UI only
# username: "admin"
# password: "${SWAGGER_PASSWORD}"With enabled: true, the Swagger UI is at /swagger-ui and the raw spec at /swagger-ui/openapi.json. The spec documents every route, request/response schema, and the authentication header — treat it as the source of truth for direct HTTP integration. API routes always require the configured API key or JWT regardless of this setting; swagger.auth only gates access to the UI itself.
Next
- Deployment → — Docker compose for a local cluster, Kubernetes manifests for production.
- How it works → — what happens inside the cluster during a ceremony.
- Integration patterns → — connect the cluster to your devices or backend.