Troubleshooting

Systematic diagnostic procedure to validate node accessibility, network connectivity, and blockchain consistency, plus known non-defect issues and their fixes.

Use these commands from the AnyLog CLI or via REST to diagnose connectivity and confirm your node and network are operating correctly. If you recognize a specific error message, jump straight to Known Issues instead of running the full procedure.

Diagnostic Procedure

Step Command Confirms
1 get processes Services are running
2 test node Local TCP/REST accessibility
3 test network Peer connectivity
4 blockchain test Local blockchain is intact
5 get metadata version Metadata is in sync with peers
6 test cluster Cluster config — operator nodes only
7 get streaming Data is arriving at the node

get processes

Shows which AnyLog services are currently running on the node. This is the quickest first check — run it any time a node is misbehaving or a service is suspected to be down.

get processes

Services are listed with their status. Any service shown as Not declared or absent is not running. Common services to verify:

Process         Status        Details
----------------|-------------|--------------------------------------------------------------------
TCP             | Running     | Listening on: 10.0.1.5:32148, Threads Pool: 6
REST            | Running     | Listening on: 10.0.1.5:32149, Threads Pool: 6, Timeout: 20, SSL: False
MCP             | Not declared|
Operator        | Running     | Cluster Member: True, Using Master: 10.0.0.1:32048, Threads Pool: 3
Blockchain Sync | Running     | Sync every 120 seconds with master using: 10.0.0.1:32048
Scheduler       | Running     | Schedulers IDs in use: [0 (system)] [1 (user)]
Blobs Archiver  | Running     | Flags: dbms = False, folder = True, compress = True, reuse_blobs = True
MQTT            | Running     |
MSG Client Pool | Not declared|
MSG Broker      | Running     | Listening on: 10.0.1.5:32150, Threads Pool: 6
SMTP            | Not declared|
Streamer        | Running     | Default streaming thresholds are 60 seconds and 10,240 bytes
UNS Streamer    | Not declared|
Query Pool      | Running     | Threads Pool: 3
Kafka Consumer  | Not declared|
gRPC            | Not declared|
PLC Client      | Not declared|
Pull Processes  | Not declared|
Video Processes | Not declared|
Publisher       | Not declared|
Distributor     | Not declared|
Consumer        | Not declared|

Restoring a service that didn’t start

Compare the list against what the node’s configuration is supposed to run. A service the deployment should have started, but that shows as Not declared, usually failed during startup — for example because a policy wasn’t synced yet or a port was still held by a previous instance. Check get error log for the reason, then use either approach:

  • Restart the AnyLog agent container. The deployment scripts run again from the top and start every configured service. This is the simplest option and the right one when several services are missing. 
    docker restart <container-name>
  • Start (or stop) the single service manually from the AnyLog CLI, or send the same command through REST. This avoids a restart and leaves the other services untouched — useful on a production node that is otherwise healthy. Use the same command and parameters the deployment scripts use for that service, then run get processes again to confirm its status changed to Running.

test node

Validates that the node is self-accessible — checks that the TCP service, REST service, and local blockchain copy are all reachable and consistent.

test node

A passing result confirms:

  • TCP port is bound and accepting connections
  • REST port is bound and responding
  • Local blockchain file is readable and valid

Run this first when a node appears unreachable from the network — if test node fails, the issue is local to the node itself rather than a network path problem.

Test TCP
[************************************************************]

Test REST
[************************************************************]

Test                                        Status
--------------------------------------------|----------------------------------------------------
Metadata Version                            | a3f9c2e1d84b7a6f2c1e0d95b83a417
Metadata Test                               | Pass
TCP test using 10.0.1.5:32148               | [From Node 10.0.1.5:32148] NODE-01@10.0.1.5:32148 running
REST test using http://10.0.1.5:32149       | NODE-01@10.0.1.5:32148 running

test network

Validates that the node can communicate with the rest of the network — sends a test message to all peer nodes and reports which ones respond.

test network

A failure here (while test node passes) points to a network path issue between nodes rather than a local configuration problem.

Common causes of test network failure:

  • Nodes on the same DNS network — bind IP addresses to shared local IPs rather than the external router-assigned IP
  • Nodes on different DNS networks — confirm that AnyLog ports are accessible externally; this can be a machine-level firewall issue or a DNS/modem port-forwarding issue

Expected output — a + means the node replied; no + means it failed to respond:

Address               Node Type  Node Name                    Status
----------------------|----------|----------------------------|------
10.0.0.1:32048        | master   | anylog-master               |  +
10.0.1.1:32148        | operator | wind-turbine1               |  +
10.0.1.2:32148        | operator | wind-turbine2               |  +
10.0.1.3:32148        | operator | wind-turbine3               |  +
10.0.1.4:32148        | operator | wind-turbine5               |  +
10.0.2.1:32148        | operator | water-plant-operator1       |  +
10.0.2.2:32148        | operator | power-plant-operator1       |  +
10.0.2.3:32148        | operator | waste-water-plant-operator1 |  +
10.0.2.4:32148        | operator | water-plant-operator1-bkup1 |  +
10.0.2.5:32148        | operator | power-plant-operator1-bkup1 |  +
10.0.3.1:32348        | query    | anylog-ucsc-query           |      ← no + means failed to return a reply
10.0.3.2:32348        | query    | anylog-remote-query         |  +
10.0.1.6:32148        | operator | RIG-TX-001                  |  +
10.0.1.7:32148        | operator | RIG-TX-007                  |  +
10.0.1.8:32148        | operator | RIG-GOM-023                 |  +
10.0.3.3:32348        | query    | mark-query                  |  +
10.0.3.4:32348        | query    | bachelor-query              |  +
10.0.1.5:32148        | operator | RIG-ND-012                  |  +
10.0.1.9:32148        | operator | vessel-operator-dlt         |  +

During test network you may see blockchain sync warnings such as Policy #NNNN is not recognized or Terminating blockchain sync process — unrecognized file format. These indicate a blockchain consistency issue between nodes — proceed to the blockchain validation steps below.

If no error appears on screen, check the error log:

get error log

Once the issue is resolved, clear the log to confirm no new errors appear:

reset error log

Blockchain validation

The blockchain is AnyLog’s metadata layer — it holds node policies, cluster assignments, and network topology. If nodes can reach each other (TCP/REST pass) but queries fail or nodes aren’t routing data correctly, a blockchain mismatch is the likely cause.

blockchain test

When: test network passes but nodes still aren’t finding each other or routing queries correctly.

Why: Confirms the local copy of the blockchain file is intact. A corrupted or missing local copy means the node has no metadata to work from.

blockchain test

get metadata version

When: Nodes are communicating but behaving inconsistently — one node sees a policy another doesn’t.

Why: Every node should be running the same metadata version. A mismatch means one or more nodes have a stale local copy and need to re-sync from the master.

get metadata version

Run on the problem node, then on the master or a known-good node and compare the returned ID — they must match.

run client (IP:Port) get metadata version

test cluster

When: Operator nodes are running and network connectivity is confirmed, but data is not being shared or replicated within the cluster.

Why: test cluster alone is a top-level check — use the subcommands below to isolate exactly where the cluster divergence is. A node can be reachable on the network but still misconfigured at the cluster level — wrong cluster ID, missing operator assignment, database/partition mismatch, or stale policy that hasn’t synced.

Run all four on each operator node in the cluster, in order:

test cluster setup

Validates the node’s configuration supports HA — confirms the operator is correctly assigned to a cluster and the local setup is consistent with what the blockchain policy describes.

test cluster setup

Start here. If this fails, the remaining tests are not meaningful — fix the configuration first.

test cluster databases

Compares the databases defined on each member of the cluster — all operators in the same cluster should have the same logical databases.

test cluster databases

A mismatch here means one or more operators are missing a database that peers have, or have an extra one.

test cluster partitions

Compares the partition scheme defined on each member — operators in the same cluster must partition data identically or queries will return inconsistent results.

test cluster partitions
test cluster data

Compares the actual TSD (time-series data) tables across cluster members to confirm data is being replicated consistently. Optionally scope to a recent window:

test cluster data
test cluster data where start_date = -7d

A divergence here (after setup, databases, and partitions all pass) means replication is failing at the data level — check the Operator and Streamer services are running on all members (get processes) and that the cluster policy on the blockchain is consistent across nodes.

get streaming

Shows the data the node has received through its streaming services, per table. Run it last: once steps 1–6 pass, this answers whether data is actually reaching the node.

get streaming

If the node is healthy but get streaming shows no data (or the counts aren’t increasing between runs), the problem is upstream of AnyLog — see Node is healthy but no new data is arriving.

Known Issues

Node is healthy but no new data is arriving

A typical case is after a machine restart or power outage at the edge: the AnyLog node comes back up on its own, but data stops flowing into the network. Usually the node is fine and the device or software feeding it didn’t recover.

Example: PowerPlant

Data from the PowerPlant site stopped reaching the network. The fix was:

  1. Connect to the site’s network (VPN) and attach to the AnyLog node.
  2. Run the diagnostic procedure:
    • test node — passed
    • test network — passed
    • get processes — every expected service was Running
    • get streaming — no data
  3. Because AnyLog itself was healthy, check the upstream data source. At PowerPlant this is a Dynics edge device (a third-party product, not AnyLog), managed through its own web UI on the same machine.
  4. The Dynics license was no longer configured, so it had stopped publishing. Re-apply the license.
  5. Restart the data flows in Dynics (at PowerPlant: start v6 with the Sabetha PV in the flows).
  6. Run get streaming again to confirm data is arriving.

No action was needed on the AnyLog side — it had restarted automatically without errors.

General approach: if test node, test network and get processes all pass but get streaming stays empty, don’t restart AnyLog. Check the data source instead: is it running, licensed, configured to publish to this node’s broker or REST port, and are its flows started?

Attempting to run a query and getting an error message

Error Type 1: Failed to load table metadata from blockchain

AL query +> run client () sql my_db "select count(*) from t1;"
Failed to load table metadata from blockchain
AL query +> get error log
ID     Count Thread     Time                     Type  Text                                                                                                 
------|-----|----------|------------------------|-----|----------------------------------------------------------------------------------------------------|
702271|    1|MainThread|Mon May 26 20:52:20 2025|Error|Failed to retrieve table definition from metadata policy for table: 'my_db.t1'                      |
702272|    1|MainThread|Mon May 26 20:52:20 2025|Error|Failed to get metadata info from Table Policy with DBMS: 'my_db' and Table: 't1'                    |
702273|    1|MainThread|Mon May 26 20:52:20 2025|Error|(Failed: Failed to load table metadata from blockchain) run client () sql my_db "select count(*) fro|
      |     |          |                        |     |m t1;"                                                                                              |

In this situation, the metadata information is not yet accessible, usually due to buffer/blockchain sync timing between the nodes. This issue resolves itself once the data is processed and the node is synced against the blockchain.

Error Type 2: Local query database not available: 'system_query'

AL query +> run client () sql my_db "select count(*) from t1;" 
AL query +> DBMS not open
AL query +> get error log 
ID     Count Thread     Time                     Type  Text                                                                     
------|-----|----------|------------------------|-----|--------------------------------------------------------------------------|
698413|    1|MainThread|Mon May 26 20:45:56 2025|Error|Local query database not available: 'system_query'                        |
698414|    1|MainThread|Mon May 26 20:45:56 2025|Error|(Failed: DBMS not open) run client () sql my_db "select count(*) from t1;"|

In this situation, the user is missing the system_query logical database, which is used to aggregate results from operator node(s). See SQL Storage for connection details.

connect dbms system_query where type=sqlite and memory=true

Why is my replicated data inconsistent?

When users deploy multiple operators, they sometimes notice data coming in from only 1 out of 2 operators, and inconsistently at that — data appears and then disappears. This is most commonly because the operators share a cluster name, but HA is not actually enabled. Two ways to resolve this:

Option 1: Restart the operator nodes with HA enabled, or run the following commands directly on the operator nodes:

run data distributor
run data consumer where start_date=!start_data

To verify: start_date=!start_data — the variable name doesn’t match the parameter name. This exact same mismatch appears in the FAQ’s deployment config example too, which suggests it’s a real (if confusingly named) variable from the actual deployment-scripts repo rather than a one-off typo — worth confirming either way before relying on it.

Option 2: Restart the second operator with a new cluster name — this approach works for EdgeLake.

  1. Attach to the node 
    docker attach --detach-keys=ctrl-d edgelake-operator2
  2. Drop the operator policy for the node 
    operator_id = blockchain drop policy where name = edgelake-operator2 bring [*][id]
run client (!ledger_conn) blockchain drop policy where id=!operator_id
  3. Detach from container — ctrl-d
  4. Stop the operator — docker stop edgelake-operator2
  5. Remove the blockchain volume — docker volume rm docker-compose_edgelake-operator2-anylog-blockchain
  6. In the (basic) configuration file for operator 2, update the cluster name.
  7. Start operator 2: 
    make up ANYLOG_TYPE=operator2