Hardware vs Sim86

Will your firmware behave the same way in Sim86 as it does on a board? We ran a firmware test on a physical ESP32-C3 and in Sim86 to compare their behavior.

Sim86 produced the same signals, in the same order, as the board. Some happened a little earlier or later.

We also measured CPU cycles—the processor’s clock ticks—with the same instrumented firmware on both sides. We ran this test five times on hardware and five times in Sim86. Hardware repeated its cycle counts exactly, and Sim86 repeated its own counts exactly. The small differences were between the board and Sim86, not between repeated runs of either. Here’s how they compare.

The results

We tested three ways firmware exchanges data: I2C, SPI, and CAN protocol logic. Both sides of the conversation run on one chip. This test doesn’t use a physical CAN transceiver or a fleet of devices.

A signal transition is a change from low to high or back again. CPU cycles count processor clock ticks during each part of the test.

What we checked	Hardware	Sim86	Result
Data signal	208 transitions	Same 208 transitions, in the same order	Matches
Clock signal	480 transitions	Same 480 transitions, in the same order	Matches
Data timing	Recorded edge times	Up to 0.167 ms earlier or later	Differs
Clock timing	Recorded edge times	Up to 0.020 ms earlier or later	Differs
I2C execution	1,082,154 CPU cycles	1,085,296 CPU cycles	0.29% more cycles
SPI execution	1,273,780 CPU cycles	1,272,574 CPU cycles	0.095% fewer cycles
Two CAN frames	1,003,694 / 1,179,694 cycles	1,003,685 / 1,179,686 cycles	9 / 8 fewer cycles

For timing, we lined up the first signal change in both recordings. The signal and CPU-cycle measurements use separate firmware builds. These results come from a local Sim86 build. The full setup is below.

What does this mean?

In this test, Sim86 produced the same sequence of signals as the board. Some signals arrived a little earlier or later. The largest timing difference was about 0.17 milliseconds.

Think of two people playing the same notes in the same order, with one slightly ahead of the other. The sequence matches; the timing isn’t identical.

Why is there a difference?

We found that Sim86 moved through one pause faster than the board. That explains where most of the timing gap appeared. We’re still investigating what caused it.

Some parts of the test deliberately wait before continuing. Those waits can make the overall times look very close, even if individual steps take different amounts of time. So these results don’t mean every processor step takes exactly as long as it does on hardware.

Does it matter for my product?

It depends on what your firmware needs to do. If you’re checking the order of signals, these results show a match. If your device must respond within a strict time limit, that limit needs its own test.

This is one test on one controller. It doesn’t yet tell us how your firmware handles a failed update, a lost connection, or a power cut. We can help you define a comparison around the situations that matter to your product.

Try it with your firmware

Book a call to discuss your board and the behavior you need to verify. We add controllers on demand, free of charge for our users when you ship us a development board. Contact us about the controller or component you need.

Explore the evidence

Download the benchmark files for the firmware, raw captures, results, and comparison scripts. The details below explain exactly what we measured and how to check it yourself.

Signal measurements and test setup

A signal transition is an edge: a pin changing from low to high or high to low. We line up the first falling edge on D4, then compare the time of each corresponding edge. This removes the boot-time difference.

Offsets below are simulator minus hardware. A negative value means the simulated edge happened earlier. These are offsets from the aligned start, not the error in each individual bit period.

Timing measurement	D4 data	D5 clock
Median edge offset	−7.317 µs	−4.088 µs
Minimum edge offset	−166.737 µs	−14.202 µs
Maximum edge offset	+19.571 µs	+19.596 µs
Largest absolute edge offset	166.737 µs	19.596 µs

The logic analyzer samples at 12 MHz, approximately once every 83.3 ns. The measured timing differences are larger than that sampling interval. One capture per environment does not measure run-to-run variation or temperature sensitivity. Matching signal sequences alone does not demonstrate timing accuracy.

Setup and provenance

Item	Setup
Physical board	XIAO ESP32-C3, D4/GPIO6 and D5/GPIO7
Capture	FX2 logic analyzer, 12 MHz, common ground
Hardware acquisition	September 29, 2026; archived capture
Simulator replay	October 15, 2026; local native simulator build, 1,000 simulated ms
Firmware	Merged flash image; binary matches the repository artifact retained with the original bench capture
Alignment	First falling edge on D4 is time zero in each trace

The original acquisition did not record a separate hash read back from the physical board. The manifest records that limitation, the artifact hashes, and the provenance of this replay. The local simulator executable is identified by its hash.

Where the timing differences accumulate

We re-analyzed the original GPIO captures, aligning the first falling D4 edge and looking for intervals longer than 1 ms with no transitions on either recorded pin. There are three such gaps in each capture. The last is 2,603.416 µs on hardware and 2,458.044 µs in Sim86: a difference of −145.372 µs.

The offset is −18.842 µs at the start of this gap and −164.214 µs at its end. The largest data-edge offset later reaches −166.737 µs. This localizes most of that cumulative difference to the quiet interval; it does not establish its cause. A quiet interval can include firmware work with no visible pin transitions.

The firmware source shows that can_frame uses deadlines calculated from the frame’s start, while wait_us starts a fresh relative wait each time. The CAN measurement excludes reporting and the wait between frames; the GPIO trace includes the gaps between visible transitions. Also, the GPIO and cycle experiments use different binaries. Their numbers cannot be treated as measurements of the same run.

Instruction costs, interrupt handling, memory access, and reporting overhead are possible contributors to investigate, not established explanations for the measured differences. The next diagnostic step is to measure those intervals separately on hardware and in the same simulator build.

The bundle includes explain-timing.py and its output. To reproduce:

python3 hardware-benchmark/explain-timing.py

CPU cycle counts and measurement method

Phase	Hardware	Sim86	Difference
I2C phase: CPU cycles	1,082,154	1,085,296	Differs — +3,142 cycles (+0.2903%)
SPI phase: CPU cycles	1,273,780	1,272,574	Differs — −1,206 cycles (−0.0947%)
CAN arbitration frame: CPU cycles	1,003,694	1,003,685	Differs — −9 cycles (−0.000897%)
CAN retry frame: CPU cycles	1,179,694	1,179,686	Differs — −8 cycles (−0.000678%)

On October 16, 2026, we flashed a newly instrumented version of the loopback firmware onto the connected ESP32-C3 and ran the same merged binary in a local Sim86 build. Flash verification confirmed that the board’s image matched the file. We collected five hardware runs with a USB reset before each, and five fresh simulator runs. All ten completed the workload.

The firmware reads esp_cpu_get_cycle_count() at the start and end of the I2C phase, SPI phase, and each of the two CAN frames. It subtracts the start count from the end count and reports the totals after the workload. CPU frequency is configured at 160 MHz, with power management disabled.

The I2C and SPI measurements include interrupts and intentional waits; CAN frame measurements run inside the workload’s existing critical section. Console reporting is outside the measured intervals. Counter-read overhead is retained on both sides. Startup is excluded; the hardware USB reset and simulator fresh boot differ.

This is an elapsed-cycle comparison for these four phases. It does not isolate individual instruction costs, interrupt latency, or cache effects. In particular, the workload uses cycle-counter waits to control bus timing. A small percentage difference here must not be presented as a general CPU accuracy percentage.

The instrumented firmware is a different binary from the original GPIO capture. Its results are a separate experiment, not extra measurements of the older run.

The benchmark bundle includes a cycles/ directory containing the instrumented firmware source, merged binary, all ten console captures, results, and build and measurement details. To recalculate the table after extracting the bundle:

python3 hardware-benchmark/cycles/compare-cycles.py hardware-benchmark/cycles

Bus edge cases included in the workload

These cases are present in the published firmware. The comparison checks the complete pin traces; it does not separately decode and score each protocol case. Included means exercised by the workload, not an independent pass for that behavior.

Edge case	Hardware	Sim86	What the comparison establishes
I2C clock stretching	Firmware holds the clock low while processing	Same workload	Included — overall signal sequence matches; stretch duration not scored separately
I2C repeated START	Switches from write to read without an intervening STOP	Same workload	Included — overall signal sequence matches; transaction not separately decoded
I2C missing device / NACK	Addresses an unassigned device address	Same workload	Included — overall signal sequence matches; recovery policy not evaluated
I2C final-byte NACK	Reader ends the read with a NACK	Same workload	Included — overall signal sequence matches; ACK/NACK values not separately scored
Irregular I2C clock periods	Fixed-seed variation in bit delays	Same workload	Included — timing differences reported above; not a timing-boundary sweep
SPI shared data-line turnaround	Command sender releases the line for the response	Same workload	Included — overall signal sequence matches; electrical contention not measured
CAN arbitration	Two software roles compete; one stops transmitting	Same workload	Included — single-chip GPIO logic, not a physical CAN bus test
CAN retry after arbitration loss	Losing role transmits in the next frame	Same workload	Included — overall signal sequence matches; retry latency not separately scored

Failure and recovery cases still to benchmark

For a deployment decision, we also want evidence for failure and recovery paths. The cases below are not measured in this published benchmark. “Not measured” is a gap in the evidence here, not a statement that Sim86 supports or cannot support the scenario. Each needs a defined setup and paired hardware/simulator runs.

Edge case	Hardware result	Sim86 result	What we would compare
Power loss during an OTA write	Not measured	Not measured	Image integrity, selected boot image, and recovery after restart
Reset during OTA activation	Not measured	Not measured	Boot selection and whether the device reaches a usable state
Corrupted or incomplete update	Not measured	Not measured	Rejection behavior and preservation of the previous working image
New firmware fails its first boot	Not measured	Not measured	Rollback decision, persistent state, and time to recovery
Network drops during an update	Not measured	Not measured	Retry, resume, timeout, and final image integrity
Devices running mixed firmware versions	Not measured	Not measured	Communication and behavior while only part of the fleet is updated
Duplicate, delayed, or reordered messages	Not measured	Not measured	State transitions, repeated actions, and timeout handling
Device disconnects and rejoins	Not measured	Not measured	Reconnection, state synchronization, and handling of stale data
I2C data or clock held low	Not measured	Not measured	Timeout, bus recovery, and whether other work continues
Peripheral stops responding mid-transfer	Not measured	Not measured	Error reporting, retries, and return to normal operation
Corrupted bus frame	Not measured	Not measured	Error detection, discarded data, and recovery behavior
Interrupt burst or competing interrupts	Not measured	Not measured	Missed events, execution order, and response latency
Watchdog expiry during a stalled task	Not measured	Not measured	Reset timing, reset cause, and state after reboot
Deadline under peak load	Not measured	Not measured	Completion times against an application-defined deadline
Reset while an output is active	Not measured	Not measured	Pin states during reset and the application’s required output state
Interrupted persistent-state write	Not measured	Not measured	Stored-data integrity and recovery on the next boot
Supply dip or brownout	Not measured	Not measured	Reset thresholds and recovery; requires a defined electrical model

These are candidate comparisons, not safety qualification results. Acceptance criteria must come from the specific device and its required behavior.

Reproduce the comparison

The Sim86 CLI is currently available to a closed group of users. Book a call with us to request access. You do not need CLI access to compare the downloadable traces below.

The complete benchmark bundle contains both captures, the comparison script, results, manifest, and firmware source project.

Download hardware.json, simulation.json, and compare.py into one directory. Python 3 is sufficient; the script has no third-party dependencies.

python3 compare.py hardware.json simulation.json

This checks edge counts and ordered levels, and reports timing offsets. It exits nonzero when the level sequences differ. Without a timing threshold, a pass means sequence agreement only. The complete output is available as result.json.

You can set an acceptance threshold explicitly:

python3 compare.py hardware.json simulation.json --max-offset-us 20

That command fails for this capture because D4 exceeds 20 µs. The threshold is an illustration, not a recommended tolerance for your application. Choose a tolerance from the behavior your firmware must satisfy, before inspecting results.

The published traces reproduce the calculation above. They do not include a rebuildable distribution of the simulator backend. For a fresh hardware capture, the firmware source and merged firmware image identify the workload. The bundle includes the ESP-IDF project under firmware-source. In an ESP-IDF shell, build it with idf.py set-target esp32c3 and idf.py build from that directory. A different compiler or ESP-IDF version can change timing; use the retained binary to keep the artifact fixed when repeating the bench run.

Design a benchmark for your own firmware

1. Keep the artifact fixed. Record the firmware hash, board revision, toolchain, clocks, and configuration. Flash that artifact to the physical board and load the corresponding image into the simulator.
2. Keep the circuit explicit. Record connections, supplies, pull-ups, loads, and measurement points. A carrier board or analyzer can change pin behavior.
3. Choose observations and tolerances. Separate protocol bytes, pin levels, event timing, and wall-clock execution speed. Declare which differences fail the test and how startup is aligned.
4. Capture both sides. Preserve the raw traces, simulator version or build identifier, seed, analyzer sample rate, and acquisition method. Repeat runs when the behavior varies; report the distribution rather than one favorable run.
5. Keep the failures. Record missing edges, unexpected states, unsupported peripherals, and timing differences alongside successful checks.

The comparison script accepts two channels named w0 and w1, each with an edges array of [timestamp_ns, level] pairs. Level 1 means low and 2 means high in this capture format. Export or convert both instruments to those units and conventions before comparing them; reject floating or unknown states instead of silently turning them into zeros.

More firmware testing guides

Use the automated firmware testing guide to turn a requirement into a repeatable CI check. For board setup, see the ESP32-C3 guide or the firmware simulation examples. For wireless tests, start with the RF Environment and define a separate comparison for the radio behavior you need to measure.