Embedded & ECU Software

RTOS Debugging Guide

A practical engineering guide to finding timing, scheduling, resource, and integration defects in firmware built around a Real-Time Operating System.

Engineering model

How RTOS fits together

RTOS debugging is the analysis of how software execution, deterministic scheduling, runtime services, and hardware interfaces interact over time. The engineer must correlate source-code behavior with task timing, shared state, resource limits, and device-driver activity on the target. Effective investigation depends on a clear execution model, repeatable observations, and separation of application defects from configuration or hardware-integration effects.

Core concepts

The parts of a practical RTOS setup

01

Deterministic scheduling

Scheduling determines when executable work is allowed to run and whether time-sensitive behavior receives sufficient execution opportunity. Debugging requires comparing intended timing with observed execution rather than judging behavior from source-code order alone.

02

Runtime resource ownership

An RTOS provides runtime services that coordinate execution and shared resources. Defects often arise when ownership, access order, or waiting behavior is inconsistent across firmware modules.

03

Target-specific execution

The processor platform, compiler language, and configuration influence timing, memory use, and peripheral behavior. The same firmware logic can therefore expose different symptoms on STM32 and ESP32 targets.

04

Peripheral interface behavior

A device driver connects firmware to a hardware peripheral through a defined interface. RTOS failures can be caused by the interaction between driver activity, execution timing, and the surrounding firmware module rather than by scheduling alone.

Build an execution model before changing code

Start by describing what should execute, when it should execute, and what data or hardware interface it depends on.

For each relevant firmware module, identify its trigger, expected execution interval, inputs, outputs, and relationships with other modules. Then identify where the RTOS controls execution and where a device driver or hardware prototype introduces external timing. This model provides a basis for distinguishing an execution-order defect from an incorrect requirement, configuration, or hardware assumption.

ObservationLikely area to examineReasoning
Work runs later than expectedScheduling and waiting behaviorThe code may be ready to execute but not receiving the required execution opportunity.
Output changes after unrelated activityShared state or access orderingAnother execution path may be changing data or altering the order of operations.
Failure appears only on the targetPlatform, resource use, or device driverTiming and hardware-interface behavior can differ from assumptions made during source-code analysis.
Failure disappears during observationTiming sensitivityThe investigation method may be changing execution timing or resource availability.

Separate scheduling defects from application defects

A useful investigation narrows the fault by testing whether the RTOS is producing the observed behavior or merely exposing an error elsewhere.

  1. 01

    Define the expected sequence

    Use the requirements and existing source code to write the expected order of relevant events, including inputs, outputs, and device-driver interactions.

  2. 02

    Mark execution boundaries

    Identify where the RTOS determines when work runs and where the firmware performs ordinary computation or accesses shared state.

  3. 03

    Compare expected and observed timing

    Check whether the symptom is caused by delayed execution, unexpected repetition, missing execution, or incorrect data after execution.

  4. 04

    Reduce unrelated activity

    Temporarily narrow the involved firmware modules and interfaces while preserving the conditions that reproduce the symptom.

  5. 05

    Form one testable hypothesis

    State whether the next test targets scheduling, resource use, source-code logic, configuration, or hardware integration.

  • A late result does not by itself prove that scheduling is incorrect; the executing code may be using excessive time or waiting on another condition.
  • A repeatable result under the same input is more useful than a symptom described only as intermittent.
  • A change that alters timing can hide a defect without correcting its cause.

Investigate timing and shared state

Many RTOS failures are interactions between execution timing and data that is accessed from more than one execution path.

Review every value that can be read or changed by multiple execution contexts. Establish which context owns the value, when updates become visible, and what happens if a reader observes the value between related updates. Also examine whether a waiting operation can delay the context that produces required data. The goal is to identify an explicit ownership and sequencing rule rather than adding arbitrary delays.

  • Check whether a value has one clear writer or requires coordinated access.
  • Check whether a multi-step update can be observed in a partially updated state.
  • Check whether the producer and consumer use compatible timing assumptions.
  • Check whether a wait can block progress needed by another firmware module.
  • Check whether diagnostic observation changes the timing being investigated.

Trace device-driver and platform interactions

When firmware depends on a peripheral, debug the interface contract and execution context together.

A device driver may be called from firmware execution or may report hardware activity back to the firmware. Determine which side initiates each interaction, what data is exchanged, and what timing the surrounding module expects. On STM32 or ESP32 hardware, also compare the assumed platform behavior with the actual target configuration. Keep platform-dependent observations separate from conclusions about the RTOS itself.

BoundaryQuestions to answerUseful conclusion
Firmware module to RTOSWhen is the module allowed to run, and what runtime service does it depend on?Whether the symptom is coupled to scheduling or waiting behavior.
Firmware module to device driverWhat interface is called, and what result or state is expected?Whether the module is interpreting the driver interaction correctly.
Device driver to hardware prototypeWhat target behavior is assumed, and what behavior is observed?Whether integration conditions can explain the symptom.
C or C++ source to target platformWhich implementation and resource assumptions are platform-dependent?Whether the defect is sensitive to language, build, or target characteristics.

Keep the investigation bounded: change one interface assumption or configuration at a time, then repeat the same test. This makes it easier to distinguish a platform effect from a source-code or RTOS effect.

Turn observations into a regression test

A debugging result is valuable when the failure condition and the expected behavior can be exercised again.

  1. 01

    Capture the failure condition

    Record the requirements involved, target platform, relevant firmware module, input conditions, and the timing or state that identifies the symptom.

  2. 02

    Create the smallest repeatable exercise

    Retain only the source-code paths, RTOS behavior, and device-driver interaction needed to reproduce the issue.

  3. 03

    Apply the correction

    Change the smallest justified area and document whether it affects scheduling, shared state, resource use, or integration.

  4. 04

    Repeat the original exercise

    Confirm that the expected behavior is restored under the original conditions and that the correction does not introduce a different timing or resource problem.

  5. 05

    Preserve the test with the firmware module

    Keep the test evidence with the affected firmware module so later integration and optimization work can distinguish a regression from a new symptom.

  • Use requirements to define expected behavior, not only the absence of the original symptom.
  • Test both the failing condition and nearby timing or input conditions when the defect is timing-sensitive.
  • Record unresolved uncertainty instead of treating a plausible correlation as a confirmed root cause.

Resource and configuration review

Some apparent RTOS defects are configuration or resource failures that become visible only under particular execution patterns.

Review the configured execution resources, runtime-service use, and platform assumptions against the behavior required by the firmware module. Look for excessive execution demand, insufficient room for nested activity, incompatible interface assumptions, and configuration values that do not match the requirements. Compare the review with observed behavior on the hardware prototype; source-code inspection alone may not reveal target-specific limits.

  • Compare configured behavior with the requirements and intended module timing.
  • Review resource use under the busiest relevant execution condition, not only during nominal operation.
  • Check whether C and C++ components make different assumptions about ownership, lifetime, or execution cost.
  • Re-test after configuration changes because timing changes can expose a separate defect.
  • Treat an improvement after configuration changes as evidence about a dependency, not automatic proof of the only cause.

Engineering pitfalls

Common mistakes

  1. Assuming source-code order is execution order

    RTOS scheduling can interleave firmware activity, so statements in separate execution paths do not necessarily run in the order an engineer expects from a linear reading.

  2. Using delays to hide timing defects

    A delay changes timing relationships and may suppress the symptom without establishing correct ownership, sequencing, or resource behavior.

  3. Blaming the RTOS before checking the interface

    A device driver, hardware prototype, configuration, or firmware module can produce the same symptom. The investigation should compare the complete boundary rather than assign cause from appearance.

  4. Changing several variables in one test

    Simultaneous changes to source code, configuration, and platform conditions prevent a reliable link between the change and the observed result.

  5. Ignoring observation effects

    Debugging activity can change timing and resource use, especially when the defect depends on execution order. A symptom that disappears under observation still requires an explanation.

Related resources

FAQ

RTOS questions

What should be checked first when an RTOS firmware module runs late?
First establish the expected execution timing and identify whether the module is ready to run, waiting on a runtime service, consuming excessive execution time, or dependent on another module or device driver. A late result alone does not identify the cause.
How can an engineer tell whether a failure is caused by scheduling or shared state?
Compare the timing of execution with the values observed at each boundary. If execution is delayed or omitted, scheduling or waiting is a candidate. If execution occurs but data changes unexpectedly or appears partially updated, shared-state ownership and sequencing deserve review.
Why can the same RTOS issue behave differently on STM32 and ESP32?
The target platform can change execution timing, resource availability, hardware-interface behavior, and configuration assumptions. The difference indicates a platform-sensitive dependency, but it does not by itself prove that the RTOS is defective.
Should a delay be used to fix an intermittent RTOS symptom?
Usually it should be treated as a diagnostic experiment rather than a fix. A delay can change scheduling relationships and hide the original condition while adding latency or creating another resource problem.
What makes an RTOS debugging result reusable?
A reusable result identifies the requirements, target platform, affected firmware module, input condition, relevant timing or state, correction, and expected behavior. Preserving a repeatable exercise allows later integration and optimization work to detect regressions.

Engineering support

Discuss an RTOS Project

Need focused help investigating an RTOS firmware module, device-driver boundary, or target-platform symptom? Provide the existing source code, requirements, or hardware prototype context for a bounded engineering investigation.

Discuss an RTOS ProjectBook a 20-min call