Documentation

Admin Ops

Operational metrics and alerts exposed via admin ops endpoints (live data, no mocks).

Overview

Ops endpoints return live container, queue, host, and deploy info. Queue metrics cover runner job types plus worker queues (payouts, virtual_account_issue, webhook_notifications, pos_settlement). processed_per_sec is a 5m rate; failures_last_hour covers the last 60m. Host metrics come from Prometheus + node-exporter and use WALLET_CORE_OPS_PROMETHEUS_URL when set; CPU/mem thresholds default to 85%. deploy-info reads deploy_info.json. Containers can include Kamal accessories when include_accessories=true.

Containers

GET /api/v1/admin/ops/containers

GET/api/v1/admin/ops/containers

Returns core service containers (web/runner/worker). Set include_accessories=true to include Kamal accessories like redis, prometheus, grafana, rabbitmq, and outpost. Use include_stats=true to attach cpu_pct/mem_pct from Docker stats.

Headers

ParameterTypeRequiredDescription
AuthorizationstringYesBearer <token>

Query Parameters

ParameterTypeRequiredDescription
envstringNoFilter by environmentValues: staging, production
include_accessoriesbooleanNoInclude Kamal accessory containers for the selected env
include_statsbooleanNoInclude cpu_pct and mem_pct per container (Docker stats)

Response

json
{
  "example": [
    {
      "id": "abc123",
      "name": "wallet-core-staging-web-latest",
      "role": "web",
      "status": "running",
      "health": "healthy",
      "version": "deploy-20260110-130322-prod",
      "commit_sha": "abc123",
      "restarts": 0,
      "host": "37.27.91.235-56703216100c",
      "created_at": "2026-01-13T09:40:00Z",
      "started_at": "2026-01-13T09:40:10Z",
      "image": "ghcr.io/schoolable/wallet-core:latest",
      "ports": [
        "0.0.0.0:8080"
      ],
      "cpu_pct": 12.4,
      "mem_pct": 34.1
    },
    {
      "id": "def456",
      "name": "wallet-core-staging-redis",
      "role": "redis",
      "status": "running",
      "health": "none",
      "version": "7-alpine",
      "commit_sha": "",
      "restarts": 0,
      "host": "wallet-core-staging-redis",
      "created_at": "2026-01-13T09:10:00Z",
      "started_at": "2026-01-13T09:10:05Z",
      "image": "redis:7-alpine",
      "ports": [
        "0.0.0.0:6379"
      ]
    }
  ]
}

Error Responses

StatusCodeDescription
401unauthorizedMissing or invalid Authorization header

Container actions

GET /api/v1/admin/ops/containers/:name, /logs, /health-check, and /restart support the same include_accessories query param to target accessory containers.

Example

bash
curl -H "Authorization: Bearer $TOKEN" "https://<host>/api/v1/admin/ops/containers?env=staging&include_accessories=true&include_stats=true"

Queue Metrics

GET /api/v1/admin/ops/queues

GET/api/v1/admin/ops/queues

Returns queue depths for runner and worker queues (includes POS settlement), processed_per_sec (5m window), failures_last_hour (60m), and recent job errors.

Headers

ParameterTypeRequiredDescription
AuthorizationstringYesBearer <token>

Response

json
{
  "example": {
    "queues": [
      {
        "name": "payouts",
        "depth": 4,
        "processed_per_sec": 1.2,
        "failures_last_hour": 0
      },
      {
        "name": "pos_settlement",
        "depth": 12,
        "processed_per_sec": 0.6,
        "failures_last_hour": 1
      },
      {
        "name": "webhook.aella.process",
        "depth": 2,
        "processed_per_sec": 0.4,
        "failures_last_hour": 0
      }
    ],
    "errors": [
      {
        "queue": "pos_settlement",
        "job_id": "job_123",
        "error": "pq: serialization failure",
        "timestamp": "2026-01-13T09:45:12Z",
        "count": 2
      },
      {
        "queue": "payouts",
        "job_id": "job_456",
        "error": "timeout contacting provider",
        "timestamp": "2026-01-13T09:41:10Z",
        "count": 1
      }
    ]
  }
}

Error Responses

StatusCodeDescription
401unauthorizedMissing or invalid Authorization header

Example

bash
curl -H "Authorization: Bearer $TOKEN" https://<host>/api/v1/admin/ops/queues

Breaking change

No. Responses are richer (fields populated); consumers should tolerate additional fields.

Host Health

GET /api/v1/admin/ops/hosts

GET/api/v1/admin/ops/hosts

Reports CPU/mem utilization from Prometheus (node-exporter) with threshold flags (defaults 85%). Set WALLET_CORE_OPS_PROMETHEUS_URL if Prometheus is not at http://prometheus:9090.

Headers

ParameterTypeRequiredDescription
AuthorizationstringYesBearer <token>

Response

json
{
  "example": {
    "thresholds": {
      "cpu": 0.85,
      "mem": 0.85
    },
    "hosts": [
      {
        "name": "staging-host-1",
        "cpu_util": 0.72,
        "mem_util": 0.64,
        "alert": false
      },
      {
        "name": "staging-host-2",
        "cpu_util": 0.91,
        "mem_util": 0.88,
        "alert": true
      }
    ],
    "updated_at": "2026-01-13T09:46:00Z"
  }
}

Error Responses

StatusCodeDescription
401unauthorizedMissing or invalid Authorization header
503prometheus_unavailablePrometheus not reachable or missing node-exporter metrics

Example

bash
curl -H "Authorization: Bearer $TOKEN" https://<host>/api/v1/admin/ops/hosts

Breaking change

No. Adds threshold flags; consumers should tolerate additional fields.

Deploy Info

GET /api/v1/admin/ops/deploy-info

GET/api/v1/admin/ops/deploy-info

Reads deploy_info.json written by deploy pipeline (current/previous tag, commit SHA, deployed_by, deployed_at).

Headers

ParameterTypeRequiredDescription
AuthorizationstringYesBearer <token>

Response

json
{
  "example": {
    "current_tag": "deploy-20260110-130322-prod",
    "previous_tag": "deploy-20260109-101500-prod",
    "commit_sha": "abc123",
    "deployed_by": "ops-user",
    "deployed_at": "2026-01-10T13:15:00Z"
  }
}

Error Responses

StatusCodeDescription
401unauthorizedMissing or invalid Authorization header
404deploy_info_missingdeploy_info.json not found on host

Example

bash
curl -H "Authorization: Bearer $TOKEN" https://<host>/api/v1/admin/ops/deploy-info

Breaking change

No. Newly populated fields; ensure deploy_info.json exists on host.