Building reliability into the time mesh

Good timing is only part of owning a time mesh. You also want a clock to recover when a reference disappears, explain why it stopped trusting a source, and leave useful clues if its software crashes. Those details determine how much attention an installation needs.

For T096 and N30, stability testing helps us find problems, understand them and check the fixes. The Nordic-based T096 and ESP32-S3-based N30 have different constraints, but both benefit from tests, watchdogs and useful diagnostics. A repeatable lab lets us carry those lessons from one board and firmware build to the next.

Testing through interruptions

It is satisfying to switch on a clock and see it synchronize. Leaving it running asks more of the firmware. References come and go, radios get busy, and devices start in an unexpected order. A receiver that works after boot may struggle when the same source returns hours later.

Longer tests give slow resource leaks and unusual sequences of events time to appear. Repeated loss and recovery tests let us check whether a clock can regain service without someone stepping in. Both matter when the goal is an installation you can leave alone.

We also need to understand what an interruption means. A source may still be reachable after its time becomes unusable. A receiver may be receiving packets but waiting for permission to use them. A clock may continue estimating time while its uncertainty grows. Recording these states helps explain what the clock is doing and when it can safely resume.

Startup, restarts and missing observations belong in the results alongside the best timing measurements. They show how much of a run delivered useful service and how long recovery took.

Helping a stalled clock recover

A watchdog watches for software that has stopped making progress. If the software cannot recover, a hardware watchdog can reset the device. Software watchdogs can supervise individual tasks, helping identify a stuck worker even while the rest of the system keeps running. Zephyr’s task-watchdog guide explains how these layers work together.

Choosing what counts as progress takes care. A quiet radio may simply have nothing to receive. A receiver may have been switched off deliberately. Neither should be treated like a blocked task. Equally, activity elsewhere in the system should not hide a stalled receiver.

On a receiver with both BLE and LoRa, each input needs its own health checks. LoRa may still be supplying time even when the BLE receiver has stalled. Monitoring them separately lets us detect the BLE fault and start the appropriate recovery.

The goal is fewer interruptions that need a manual restart. After a restart, the clock must authenticate its source again, collect fresh timing observations and pass its timing checks before resuming service. Recovery cannot make an expired source trustworthy again.

Understanding what went wrong

“The clock lost sync” is a starting point for an investigation. Useful diagnostics tell us which source it followed, why it changed state and what happened just before the interruption. Device identity, firmware version and boot identity let us connect those events to the right run.

On N30, recording source changes alongside timing observations from the same run helps explain behavior that a final screen reading cannot. It becomes easier to distinguish a lost reference from a transport problem or a local software fault. Missing timing corrections, overwritten records and incomplete responses must remain visible so gaps in the recording do not turn into misleading conclusions.

Recording also consumes processor time, memory and storage. Limited buffers and recording sessions, with work kept outside critical timing paths, help control that cost. We test full buffers and shutdown too, so we can recognize when a recording is incomplete.

Keeping clues after a crash

A crash can stop normal logging before the useful message gets out. A core dump preserves processor state and selected memory for later inspection. With the exact firmware executable and debug information, captured addresses can be traced to functions and source lines.

Espressif provides core-dump support for ESP32-S3 systems; capture depends on the firmware configuration. On T096, compact crash records preserve fault registers and known stack regions within the device’s tighter limits. Either approach can turn an unexplained restart into a more focused investigation.

The record must match the build that crashed. Power loss or corruption may leave it incomplete, and a stack can show where damage surfaced without revealing where it began. Even a partial clue can help us choose the next experiment.

Reproducing a failure, checking a fix

The PToB testing framework brings these tools together. Offline tests run production code with controlled clocks, radio events and operating-system behavior. We can repeat an awkward sequence, cross a timeout deliberately and check the response without waiting for the same accident to happen again. Keeping that case as a regression test helps catch its return in later changes.

Hardware runs add actual radio behavior, scheduling and interaction between devices. The lab records which devices and builds took part, preserves raw captures and identifies incomplete runs. A checked installation and a known recovery procedure make the next run easier to repeat.

With a mature lab, an investigation is easier to resume and a fix can be checked against the failure that prompted it. Longer runs and field experience still have to show how well each release holds up. The framework helps us work toward a mesh that needs less attention, with clearer answers when something goes wrong.