{"format":"ptob-page","version":1,"canonical":"https://ptob.dev/blog/firmware-stability-time-mesh/","title":"Firmware stability testing for a reliable time mesh — PToB Blog","description":"How T096 and N30 stability testing, watchdogs, diagnostics, crash records and a repeatable lab framework help build dependable firmware and a reliable time mesh.","language":"en","robots":"index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1","dates":{"published":"2026-11-06T00:00:00.000Z","modified":"2026-11-06T00:00:00.000Z"},"authors":["Jonathan Yantis"],"mainEntity":"https://ptob.dev/blog/firmware-stability-time-mesh/#article","representations":{"application/json":"https://ptob.dev/blog/firmware-stability-time-mesh/?format=json","text/markdown":"https://ptob.dev/blog/firmware-stability-time-mesh/?format=markdown"},"documentation":"https://ptob.dev/agent-formats.json","social":{"openGraph":{"title":"Firmware stability testing for a reliable time mesh — PToB Blog","description":"How T096 and N30 stability testing, watchdogs, diagnostics, crash records and a repeatable lab framework help build dependable firmware and a reliable time mesh.","url":"https://ptob.dev/blog/firmware-stability-time-mesh/","type":"article","siteName":"PToB","locale":"en_US","image":{"url":"https://ptob.dev/blog/firmware-stability-time-mesh-v2.png","secureUrl":"https://ptob.dev/blog/firmware-stability-time-mesh-v2.png","type":"image/png","width":"4800","height":"3000","alt":"Engineering diagram connecting T096 and N30 clocks with watchdog recovery, diagnostics, crash records and a laboratory test and verification loop."}},"twitter":{"card":"summary_large_image","title":"Firmware stability testing for a reliable time mesh — PToB Blog","description":"How T096 and N30 stability testing, watchdogs, diagnostics, crash records and a repeatable lab framework help build dependable firmware and a reliable time mesh.","image":{"url":"https://ptob.dev/blog/firmware-stability-time-mesh-v2.png","alt":"Engineering diagram connecting T096 and N30 clocks with watchdog recovery, diagnostics, crash records and a laboratory test and verification loop."}}},"citations":[],"provenance":{"sourceUrl":"https://ptob.dev/blog/firmware-stability-time-mesh/","sourceMediaType":"text/html","sourceSha256":"e112934c7bd8ed2463f388781a8ccfc91fb3a9f5cadf5bd8647cbdbdb81ac22d","rightsUrl":"https://github.com/yantisj2/PToB/blob/main/LICENSING.md"},"contentUsage":{"aiTrain":true,"search":true,"aiInput":true,"reason":"summary-or-introduction"},"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"WebSite","@id":"https://ptob.dev/#website","url":"https://ptob.dev/","name":"PToB","alternateName":["Precision Time over BLE","Precision Time over Bluetooth"],"description":"Bluetooth and USB software clocks: W12 60.3 ns relative-PPS RMS over 522/600 observations, separate 64 s sampled holdover, and CoreSE diagnostics.","inLanguage":["en","de"],"publisher":{"@id":"https://ptob.dev/#jonathan-yantis"}},{"@type":"Person","@id":"https://ptob.dev/#jonathan-yantis","name":"Jonathan Yantis","url":"https://github.com/yantisj2/PToB/blob/main/CREDITS.md"},{"@type":"Project","@id":"https://ptob.dev/#project","name":"PToB","alternateName":"Precision Time over BLE","url":"https://ptob.dev/","description":"An independent protocol project researching precision time transfer across Bluetooth-enabled microcontrollers and USB clients.","sameAs":"https://github.com/yantisj2/PToB","member":{"@id":"https://ptob.dev/#jonathan-yantis"},"knowsAbout":{"@id":"https://ptob.dev/#protocol"}},{"@type":"Thing","@id":"https://ptob.dev/#protocol","name":"Precision Time over BLE","alternateName":"PToB","description":"An experimental PTP-inspired protocol proposal for wireless clock synchronization over Bluetooth Low Energy."},{"@type":"WebPage","@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#webpage","url":"https://ptob.dev/blog/firmware-stability-time-mesh/","name":"Firmware stability testing for a reliable time mesh — PToB Blog","description":"How T096 and N30 stability testing, watchdogs, diagnostics, crash records and a repeatable lab framework help build dependable firmware and a reliable time mesh.","inLanguage":"en","keywords":["Firmware stability","T096","N30","Watchdogs","Crash diagnostics","Time mesh"],"isPartOf":{"@id":"https://ptob.dev/#website"},"author":{"@id":"https://ptob.dev/#jonathan-yantis"},"about":{"@id":"https://ptob.dev/#protocol"},"dateModified":"2026-11-06T00:00:00.000Z","image":"https://ptob.dev/blog/firmware-stability-time-mesh-v2.png","breadcrumb":{"@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#breadcrumbs"},"mainEntity":{"@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#article"}},{"@type":"BreadcrumbList","@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#breadcrumbs","itemListElement":[{"@type":"ListItem","position":1,"name":"PToB","item":"https://ptob.dev/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://ptob.dev/blog/"},{"@type":"ListItem","position":3,"name":"Building reliability into the time mesh","item":"https://ptob.dev/blog/firmware-stability-time-mesh/"}]},{"@type":"Blog","@id":"https://ptob.dev/blog/#blog","url":"https://ptob.dev/blog/","name":"PToB Blog","description":"Engineering notes on shared time, inexpensive hardware and the experiments behind PToB.","inLanguage":"en","publisher":{"@id":"https://ptob.dev/#jonathan-yantis"},"isPartOf":{"@id":"https://ptob.dev/#website"},"blogPost":[{"@id":"https://ptob.dev/blog/esp32-wireless-softpll/#article"},{"@id":"https://ptob.dev/blog/esp32-clock-fundamentals/#article"},{"@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#article"},{"@id":"https://ptob.dev/blog/lab-sensor-timing-economics/#article"},{"@id":"https://ptob.dev/blog/starlink-local-stratum-1-ntp/#article"},{"@id":"https://ptob.dev/blog/raspberry-pi-ptp-ntp-ble/#article"},{"@id":"https://ptob.dev/blog/gnss-resilient-time-mesh/#article"},{"@id":"https://ptob.dev/blog/passive-lora-channel-hopping/#article"},{"@id":"https://ptob.dev/blog/esp32-sub-microsecond-holdover/#article"},{"@id":"https://ptob.dev/blog/local-time-utc-mesh/#article"},{"@id":"https://ptob.dev/blog/grandmaster-failover-multi-master-ensemble/#article"},{"@id":"https://ptob.dev/blog/ble-clocks-indoor-navigation/#article"},{"@id":"https://ptob.dev/blog/passive-ble-esp32/#article"},{"@id":"https://ptob.dev/blog/why-utc/#article"},{"@id":"https://ptob.dev/blog/why-ptob/#article"}]},{"@type":"BlogPosting","@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#article","url":"https://ptob.dev/blog/firmware-stability-time-mesh/","headline":"Building reliability into the time mesh","description":"How T096 and N30 stability testing, watchdogs, diagnostics, crash records and a repeatable lab framework help build dependable firmware and a reliable time mesh.","mainEntityOfPage":{"@id":"https://ptob.dev/blog/firmware-stability-time-mesh/#webpage"},"isPartOf":{"@id":"https://ptob.dev/blog/#blog"},"author":{"@id":"https://ptob.dev/#jonathan-yantis"},"publisher":{"@id":"https://ptob.dev/#jonathan-yantis"},"inLanguage":"en","dateCreated":"2026-11-06T00:00:00.000Z","dateModified":"2026-11-06T00:00:00.000Z","datePublished":"2026-11-06T00:00:00.000Z","image":"https://ptob.dev/blog/firmware-stability-time-mesh-v2.png","articleSection":"Firmware reliability and testing","keywords":["Firmware stability","T096","N30","Watchdogs","Crash diagnostics","Time mesh"],"wordCount":908,"citation":[{"@type":"CreativeWork","name":"PToB · Firmware and testing source","url":"https://github.com/yantisj2/PToB"},{"@type":"CreativeWork","name":"Zephyr · Task watchdog","url":"https://docs.zephyrproject.org/latest/services/task_wdt/index.html"},{"@type":"CreativeWork","name":"Espressif · ESP32-S3 core dumps","url":"https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-guides/core_dump.html"}],"about":{"@id":"https://ptob.dev/#protocol"}}]}],"content":{"mediaType":"text/markdown","headings":[{"level":1,"text":"Building reliability into the time mesh","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#main"},{"level":2,"text":"Testing through interruptions","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#testing-through-interruptions"},{"level":2,"text":"Helping a stalled clock recover","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#helping-a-stalled-clock-recover"},{"level":2,"text":"Understanding what went wrong","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#understanding-what-went-wrong"},{"level":2,"text":"Keeping clues after a crash","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#keeping-clues-after-a-crash"},{"level":2,"text":"Reproducing a failure, checking a fix","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#reproducing-a-failure-checking-a-fix"},{"level":2,"text":"Explore the details","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#related-title"},{"level":3,"text":"Hardware and firmware guides","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#main"},{"level":3,"text":"Planning a time mesh","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#main"},{"level":3,"text":"Building a GNSS-resilient time mesh","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#main"},{"level":2,"text":"Sources and further reading","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#sources-title"},{"level":2,"text":"Cite this article","url":"https://ptob.dev/blog/firmware-stability-time-mesh/#cite-title"}],"links":[{"text":"Jonathan Yantis","url":"https://github.com/yantisj2/PToB/blob/main/CREDITS.md"},{"text":"Zephyr’s task-watchdog guide","url":"https://docs.zephyrproject.org/latest/services/task_wdt/index.html"},{"text":"Espressif provides core-dump support for ESP32-S3 systems","url":"https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-guides/core_dump.html"},{"text":"PToB testing framework","url":"https://github.com/yantisj2/PToB"},{"text":"Hardware and firmware guides The clock platforms, supported roles and timing paths behind PToB.","url":"https://ptob.dev/hardware/"},{"text":"Planning a time mesh Sources, relay clocks and clients working together in an installation.","url":"https://ptob.dev/deployment/"},{"text":"Building a GNSS-resilient time mesh Reference diversity, holdover and the importance of visible source limits.","url":"https://ptob.dev/blog/gnss-resilient-time-mesh/"},{"text":"PToB · Firmware and testing source","url":"https://github.com/yantisj2/PToB"},{"text":"Zephyr · Task watchdog","url":"https://docs.zephyrproject.org/latest/services/task_wdt/index.html"},{"text":"Espressif · ESP32-S3 core dumps","url":"https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-guides/core_dump.html"},{"text":"https://ptob.dev/blog/firmware-stability-time-mesh/","url":"https://ptob.dev/blog/firmware-stability-time-mesh/"},{"text":"Back to the blog","url":"https://ptob.dev/blog/"},{"text":"Inspect the project evidence","url":"https://github.com/yantisj2/PToB/blob/main/evidence/README.md"}],"text":"# Building reliability into the time mesh\n\nStability tests, watchdogs and useful fault records help us find failures, improve recovery and check each firmware release under repeatable conditions.\n\n[Jonathan Yantis](https://github.com/yantisj2/PToB/blob/main/CREDITS.md) 6 November 2026  5 min read\n\n![Engineering diagram connecting T096 and N30 clocks with watchdog recovery, diagnostics, crash records and a laboratory test and verification loop.](https://ptob.dev/blog/firmware-stability-time-mesh-v2.svg)\n\nA concept engineering diagram of the reliability framework, from clock supervision and fault records to repeatable tests. Connections show functional relationships rather than hardware wiring.\n\nIn this post\n\nGood timing is only part of owning a time mesh. You also want a clock to recover when a reference disappears, explain why it stopped trusting a source, and leave useful clues if its software crashes. Those details determine how much attention an installation needs.\n\nFor T096 and N30, stability testing helps us find problems, understand them and check the fixes. The Nordic-based T096 and ESP32-S3-based N30 have different constraints, but both benefit from tests, watchdogs and useful diagnostics. A repeatable lab lets us carry those lessons from one board and firmware build to the next.\n\n## Testing through interruptions\n\nIt is satisfying to switch on a clock and see it synchronize. Leaving it running asks more of the firmware. References come and go, radios get busy, and devices start in an unexpected order. A receiver that works after boot may struggle when the same source returns hours later.\n\nLonger tests give slow resource leaks and unusual sequences of events time to appear. Repeated loss and recovery tests let us check whether a clock can regain service without someone stepping in. Both matter when the goal is an installation you can leave alone.\n\nWe also need to understand what an interruption means. A source may still be reachable after its time becomes unusable. A receiver may be receiving packets but waiting for permission to use them. A clock may continue estimating time while its uncertainty grows. Recording these states helps explain what the clock is doing and when it can safely resume.\n\nStartup, restarts and missing observations belong in the results alongside the best timing measurements. They show how much of a run delivered useful service and how long recovery took.\n\n## Helping a stalled clock recover\n\nA watchdog watches for software that has stopped making progress. If the software cannot recover, a hardware watchdog can reset the device. Software watchdogs can supervise individual tasks, helping identify a stuck worker even while the rest of the system keeps running. [Zephyr’s task-watchdog guide](https://docs.zephyrproject.org/latest/services/task_wdt/index.html) explains how these layers work together.\n\nChoosing what counts as progress takes care. A quiet radio may simply have nothing to receive. A receiver may have been switched off deliberately. Neither should be treated like a blocked task. Equally, activity elsewhere in the system should not hide a stalled receiver.\n\nOn a receiver with both BLE and LoRa, each input needs its own health checks. LoRa may still be supplying time even when the BLE receiver has stalled. Monitoring them separately lets us detect the BLE fault and start the appropriate recovery.\n\nThe goal is fewer interruptions that need a manual restart. After a restart, the clock must authenticate its source again, collect fresh timing observations and pass its timing checks before resuming service. Recovery cannot make an expired source trustworthy again.\n\n## Understanding what went wrong\n\n“The clock lost sync” is a starting point for an investigation. Useful diagnostics tell us which source it followed, why it changed state and what happened just before the interruption. Device identity, firmware version and boot identity let us connect those events to the right run.\n\nOn N30, recording source changes alongside timing observations from the same run helps explain behavior that a final screen reading cannot. It becomes easier to distinguish a lost reference from a transport problem or a local software fault. Missing timing corrections, overwritten records and incomplete responses must remain visible so gaps in the recording do not turn into misleading conclusions.\n\nRecording also consumes processor time, memory and storage. Limited buffers and recording sessions, with work kept outside critical timing paths, help control that cost. We test full buffers and shutdown too, so we can recognize when a recording is incomplete.\n\n## Keeping clues after a crash\n\nA crash can stop normal logging before the useful message gets out. A core dump preserves processor state and selected memory for later inspection. With the exact firmware executable and debug information, captured addresses can be traced to functions and source lines.\n\n[Espressif provides core-dump support for ESP32-S3 systems](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-guides/core_dump.html); capture depends on the firmware configuration. On T096, compact crash records preserve fault registers and known stack regions within the device’s tighter limits. Either approach can turn an unexplained restart into a more focused investigation.\n\nThe record must match the build that crashed. Power loss or corruption may leave it incomplete, and a stack can show where damage surfaced without revealing where it began. Even a partial clue can help us choose the next experiment.\n\n## Reproducing a failure, checking a fix\n\nThe [PToB testing framework](https://github.com/yantisj2/PToB) brings these tools together. Offline tests run production code with controlled clocks, radio events and operating-system behavior. We can repeat an awkward sequence, cross a timeout deliberately and check the response without waiting for the same accident to happen again. Keeping that case as a regression test helps catch its return in later changes.\n\nHardware runs add actual radio behavior, scheduling and interaction between devices. The lab records which devices and builds took part, preserves raw captures and identifies incomplete runs. A checked installation and a known recovery procedure make the next run easier to repeat.\n\nWith a mature lab, an investigation is easier to resume and a fix can be checked against the failure that prompted it. Longer runs and field experience still have to show how well each release holds up. The framework helps us work toward a mesh that needs less attention, with clearer answers when something goes wrong.\n\nEvery timestamp needs a reference.\n**Which event did you capture?**\n\n## Explore the details\n\n[Hardware and firmware guides The clock platforms, supported roles and timing paths behind PToB.](https://ptob.dev/hardware/)[Planning a time mesh Sources, relay clocks and clients working together in an installation.](https://ptob.dev/deployment/)[Building a GNSS-resilient time mesh Reference diversity, holdover and the importance of visible source limits.](https://ptob.dev/blog/gnss-resilient-time-mesh/)\n\n## Sources and further reading\n\nPrimary references for the hardware and methods discussed.\n\n1. [PToB · Firmware and testing source](https://github.com/yantisj2/PToB)\n\n   Target firmware, production-code host tests, diagnostic tools and build guides.\n2. [Zephyr · Task watchdog](https://docs.zephyrproject.org/latest/services/task_wdt/index.html)\n\n   Task supervision and hardware watchdog fallback in a real-time operating system.\n3. [Espressif · ESP32-S3 core dumps](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-guides/core_dump.html)\n\n   Platform support for preserving task state and investigating fatal errors; enabled capture depends on the firmware configuration.\n\n## Cite this article\n\nJonathan Yantis. Building reliability into the time mesh PToB Blog. Published 6 November 2026.\n\nArticle permalink: [https://ptob.dev/blog/firmware-stability-time-mesh/](https://ptob.dev/blog/firmware-stability-time-mesh/)\n\n[Back to the blog](https://ptob.dev/blog/)[Inspect the project evidence](https://github.com/yantisj2/PToB/blob/main/evidence/README.md)"}}