All documentation pages ↓

Troubleshooting

Where to look

• The job page: setup, takeover, publish, zone and settings changes, promote and the health check's demotions each have a log at /jobs/<n>, linked from the node's page (Recent jobs) and the zone's History tab (Publishes).
• The Jobs page (/jobs, linked from Nodes, the Dashboard's Recent activity and every job's page) lists recent jobs on every node, newest first, with how long each took. Filter by kind, status (running, ok, failed) or node; Older goes further back. A job that ended ok with ⚠ lines in its log is marked warnings. Jobs the health check starts show liteblox as their user, command-line ones cli.
• LiteBlox's own log: docker compose logs liteblox (scheduled backups, errors).
• On a node:

  systemctl status liteblox-agent
  journalctl -u liteblox-agent -n 50
  journalctl -u named -n 50
  sudo named-checkconf -px          # the configuration named runs, key secrets masked
  journalctl -u kea-dhcp4-server -u kea-dhcp-ddns-server -n 50    # on a DHCP node
  

A node shows "error" or "cannot reach agent"

• Is the agent running? systemctl status liteblox-agent on the node. Repair / upgrade agent reinstalls and restarts it.
• Can the LiteBlox host reach TCP 8530 on the node? Check the node's firewall. From the LiteBlox host: nc -vz NODE 8530.
• Is the node's clock right? Certificates are refused if it's far off (timedatectl).
• Did the node's address change? See A node moved to a new address.

The health check asks every node once a minute, so the status corrects itself within a minute of the node coming back.

Setup fails

Take over is blocked

The preview lists what stops it, and nothing changes on the node:

• "the generated config is not equivalent to the node's" with a diff: LiteBlox's version would behave differently. Usually a statement in an unusual form; the diff shows which. Change it on the server (or remove it), then open the preview again.
• views, secondary or stub zones, dynamic zones LiteBlox can't import, options with paths: see What blocks a takeover.
• an ACL, key or zone LiteBlox already has with other contents: make the two agree (on the server, or in LiteBlox) first.

• "the node's config changed since the preview": open the preview again.

• "LiteBlox already manages the whole BIND config on this node" on a node this LiteBlox doesn't manage: another LiteBlox generated it, usually one whose data was lost. Re-adopt it instead (Disaster recovery).

If the takeover itself fails (did not serve as before after the takeover: a zone doesn't answer with the same serial), the node is rolled back to its own files and the job log says why.

Re-adopt is refused

Re-adopting rebuilds LiteBlox's data from nodes another LiteBlox managed, and changes nothing on them. So it refuses rather than guess:

• a diff under a node: rendering that node's config back from the rebuilt data doesn't give its files. Someone changed a generated file by hand, or an older LiteBlox version wrote it. Put the file back as it was (a node backup has it), or untick the node. There's no Overwrite here: the old LiteBlox's data is what's being recovered.
• "is not one of the nodes being re-adopted": add the missing node (the primary, a secondary, or the other half of the failover pair) under its old name, repair it, and preview again.
• "Re-adopting only rebuilds into an empty LiteBlox": this LiteBlox already has zones, keys or managed nodes. Re-adopt into a fresh install.

A zone shows "not writable"

The agent can't write that zone file or its directory. Re-run setup with Make /etc/bind and the zone files group-writable ticked, or give group bind write access yourself.

Publish is blocked or fails

If the serial check fails although the node reloaded, the node may answer DNS on a different address than LiteBlox asks. Set its DNS address on the node's page.

A secondary is behind

A secondary transfers each zone from the primary after the primary notifies it. If it doesn't catch up:

• Can it reach the primary on TCP 53, and is the primary's address right? The primary is listed by its DNS address (on its page) in the secondary's primaries.
• journalctl -u named on the secondary shows the transfer attempts; on the primary, the transfers it served.
• Does the primary's Answer queries from (its Server settings, or the zone's) leave the secondary out? LiteBlox lets its own key through a restricted list, but check the log.
• sudo rndc refresh ZONE on the secondary makes it check right away.

Dynamic zones

DHCP (Kea)

A node is flagged

Backups are "stale"

No successful backup in 48 hours. Open the node's Backups page: failed attempts are listed with their error, usually the agent being unreachable. Back up now retries.

The SSH host key changed

LiteBlox pins each node's SSH host key and refuses to connect when it changes, which protects the setup password. It changes legitimately when a server is reinstalled or replaced, or its SSH keys are regenerated. To pin the new key:

1. On the server, note its fingerprint: ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub.
2. On the node's page, choose Change address or host key, leave the address as it is, and Read host key.
3. LiteBlox shows the key it received next to the pinned one. Confirm only if it matches what the server showed.

From the command line: liteblox node edit NAME.

The agent itself doesn't use SSH, so a node that's already set up keeps working until it needs repair or upgrade. A reinstalled server needs Repair / upgrade agent anyway, since the agent is gone with the old system; then take it over again.

A node moved to a new address

The agent's certificate names the address LiteBlox reaches it by, and its config names its port. After changing either on the server:

1. On the node's page, choose Change address or host key, enter the new address (and ports), and confirm the host key as above.
2. The node's page then says the certificate no longer matches. Run Repair / upgrade agent, which issues a certificate for the new address.
3. If the node is the primary or a secondary, run Make … the primary again on the Nodes page (with the current primary), so the others use the new address for transfers and NOTIFY.

From the command line: liteblox node edit NAME --host NEW-ADDRESS, then liteblox node bootstrap NAME --ssh-user USER.

LiteBlox doesn't touch DNS data for you: update the node's own A/AAAA and NS records, and any Also notify or Allow transfers to lists that name the old address.

Removing a node

Remove from LiteBlox, in the Danger zone at the bottom of the node's page (or liteblox node remove NAME), makes LiteBlox forget it: its zones, publish results and backup list. It doesn't contact the server, so it works for a dead one too. Its backup files stay in ./backups/NAME/.

• A zone that no other node serves stays in LiteBlox, with its history, but can't be published until a node serves it again.
• If it was the primary, promote another node first.
• If you're replacing a dead server under the same name, don't remove it: re-pin the host key and run Repair / upgrade agent instead, so its backups stay listed.

Nothing on the server changes: BIND keeps serving what it has. To stop it serving LiteBlox's zones, use Stop serving on a node in each zone's Settings → Danger zone before removing the node, while it's reachable. To hand the server back to its own configuration afterwards:

ls /etc/bind/pre-liteblox/                              # the takeover's folder, e.g. 20261028T182537Z
sudo cp -a /etc/bind/pre-liteblox/STAMP/. /etc/bind/
sudo rm -rf /etc/bind/liteblox
sudo named-checkconf && sudo systemctl restart named

and to uninstall the agent:

sudo systemctl disable --now liteblox-agent
sudo rm -rf /usr/local/lib/liteblox-agent /etc/liteblox-agent /var/lib/liteblox-agent \
    /etc/systemd/system/liteblox-agent.service
sudo systemctl daemon-reload
sudo userdel liteblox-agent

A node's clock is off

The health check compares each node's clock with LiteBlox's every minute. The Nodes page and the node's page show:

On the node, timedatectl shows the state. sudo timedatectl set-ntp true turns NTP on (Debian: apt install systemd-timesyncd first), or let LiteBlox run chrony there (Time (NTP)). The flag clears at the next check after the clock is right. The LiteBlox container takes its time from its host, so check the host too if every node shows the same offset.

Time sync (chrony)

Behind a proxy, LiteBlox still says http

Add node warns about plain text, /api/docs and the token page show http:// URLs, or the audit log shows one address for everyone: LiteBlox doesn't trust the proxy's X-Forwarded-* headers. Set FORWARDED_ALLOW_IPS to the address the proxy's requests come from. For a proxy in a container on the same docker host, that's the docker network's gateway, not the host's address (details). Then docker compose up -d --force-recreate.

Locked out of LiteBlox

docker compose exec liteblox liteblox user passwd admin
docker compose exec liteblox liteblox user add rescue --role admin