Skip to content
Odysseus

An agent upgrade did not take

Symptom

An upgrade was approved for a node and the node is degraded, rolled back, or reporting its previous version. GET /api/v1/nodes/{name}/upgrade/history shows the attempt.

Diagnose

  1. What did the agent say?

    sudo journalctl -u odysseus-agent --since "30 minutes ago"

    Expect the upgrade to be announced, the new image to be pulled, and the service to restart. Image pull errors, permission errors and start-up failures each name themselves.

  2. Can the node pull the new image? The agent image is pulled by the node itself, from the registry the node is configured for — not through the control plane.

  3. Did it roll back on its own? A failed upgrade returns the previous version. A node that is healthy on the old version has already recovered; the question is why the new one did not start.

  4. Check the upgrade history rather than the node’s current state. A node that recovered looks identical to one that was never upgraded.

Resolve

  • Pull failed: restore the node’s access to the registry, then approve the upgrade again from the dashboard.
  • Started and crashed: the log names the reason. Do not re-approve until it is addressed — a second identical attempt produces a second identical rollback.
  • Stuck degraded after a rollback: restart the agent service on the node. It re-registers; it does not need enrolling again.

Prevent

Upgrade one node and let it settle before approving the rest. Tenant-owned nodes are upgraded by approval rather than remotely, and that is the point at which one node’s failure is cheap.