Re: [PATCH 3/3] Documentation: devel: troubleshooting: add new chapter

["Ulrich Ölmann" <u.oelmann@pengutronix.de>]

Hi Ahmad,

just some typos that I stumbled over.
Thanks for writing all those things down!

On Fri, Jul 04 2025 at 16:38 +0200, Ahmad Fatoum <a.fatoum@pengutronix.de> wrote:
> A consequence of running bare metal is that early failures are difficult
> to diagnose. Let's add a troubleshooting section to help users take
> the first step in diagnosing issues.
>
> Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> ---
>  Documentation/devel/devel.rst           |   2 +
>  Documentation/devel/troubleshooting.rst | 377 ++++++++++++++++++++++++
>  Documentation/devicetree/index.rst      |   2 +
>  3 files changed, 381 insertions(+)
>  create mode 100644 Documentation/devel/troubleshooting.rst
>
> diff --git a/Documentation/devel/devel.rst b/Documentation/devel/devel.rst
> index d985bff40d42..b90805263bbd 100644
> --- a/Documentation/devel/devel.rst
> +++ b/Documentation/devel/devel.rst
> @@ -8,7 +8,9 @@ Contents:
>  .. toctree::
>     :maxdepth: 2
>  
> +   architecture
>     porting
> +   troubleshooting
>     filesystems
>     background-execution
>     project-ideas
> diff --git a/Documentation/devel/troubleshooting.rst b/Documentation/devel/troubleshooting.rst
> new file mode 100644
> index 000000000000..67c4e3102be2
> --- /dev/null
> +++ b/Documentation/devel/troubleshooting.rst
> @@ -0,0 +1,377 @@
> +.. _troubleshooting:
> +
> +##########################
> +Boot Troubleshooting Guide
> +##########################
> +
> +Especially during development or bring-up, very early failure situations can leave
> +the system hanging before recovery is even possible.
> +
> +This guide helps diagnose and debug such issues across barebox' different boot stages.
> +
> +Boot Flow Overview
> +==================
> +
> +A barebox binary consists of two main stages:
> +
> +1. **PBL (Pre-Bootloader)**: This is a smaller barebones loader that does
> +   what's necessary to download the full barebox binary.
> +   At the very least, this is decompressing barebox proper and jumping
> +   to it while passing it a device tree.
> +   Depending on platform, it may also need to setup DRAM, install a secure
> +   monitory like TF-A or a secure operating system like OP-TEE and chainload
> +   barebox from a boot medium.
> +2. **barebox proper**: The main bootloader logic. This is always loaded
> +   by a prebootloader passing a device tree and including drivers for
> +   device initialization, environment setup, and booting the OS.
> +
> +If barebox hangs, it's essential to identify *where* in this process the
> +failure occurs. Here's how to debug different stages.
> +
> +Refer to the :ref:`barebox architecture <architecture>` for more background
> +information on the different stages and the images.
> +
> +Completely silent console

s/Completely silent console/Completely Silent Console/

> +=========================
> +
> +Even the barebox prebootloader is most often loaded by another
> +bootloader. This is commonly a mask BootROM hardwired into the
> +System-on-chip.

s/System-on-chip/System-on-Chip/

> +
> +**Common problems**:
> +
> +- Wrong bootloader image or format
> +- Bootloader installed to wrong location
> +- System hang before serial driver probe
> +- enabled, but misconfigured CONFIG_DEBUG_LL

s/enabled/Enabled/

> +
> +**What to try**:
> +
> +- Check for BootROM boot indicators:
> +
> +  Some BootROMs (e.g. AT91) write to a serial port when they start up
> +  or blink a GPIO (e.g. STM32MP) if they fail to boot the next stage
> +  bootloader.
> +
> +- Check that barebox is in the format and at the location that the
> +  previous stage bootloader expects. Compare with a previously working
> +  bootloader image, refer to the barebox documentation and/or the
> +  vendor documentation or ask around.
> +
> +- Enable ``CONFIG_DEBUG_LL``
> +
> +  This enables very early low-level UART debugging.
> +  It bypasses console frameworks and writes directly to UART registers.
> +  Many boards in barebox, print a ``>`` character, when ``CONFIG_DEBUG_LL``
> +  is enabled. If you see such a character after enabling ``DEBUG_LL``, it
> +  indicates that the barebox prebootloader has been found and control was
> +  successfully handed over to it. Note that on some SoCs, ``DEBUG_LL``
> +  requires co-operation from the board entry point, e.g., the pin muxing for
> +  the serial console needs to be done in software in some situations before
> +  the UART is accessible from the outside.
> +
> +  .. note::
> +     Make sure the correct UART index or address is selected under
> +     **Kernel low-level debugging por** in ``menuconfig``.
> +     Configuring the wrong UART might hang your system, because barebox would
> +     be tricked into accessing hardware that's not there or is powered off.
> +     The numbering/addresses of ports are described in the System-on-Chip
> +     datasheet or reference manual and may differ from labels on the hardware.
> +     Refer to the config symbol help text and ``/chosen/stdout-path`` in the
> +     device tree if unsure.
> +
> +- Enable ``CONFIG_PBL_CONSOLE`` and ``CONFIG_DEBUG_PBL``
> +
> +  For boards that don't have an early ``putc_ll('>');``, the first output
> +  being printed is often the debugging output from the uncompress entry
> +  point (``barebox_pbl_start()``). Enable these options to see if the
> +  CPU gets that far.
> +
> +  .. warning::
> +     CONFIG_DEBUG_PBL increases the size of the PBL, which can make it
> +     exceed a hard limit imposed by a previous stage bootloader.
> +     Best case, this will be caught by the build system, but might not
> +     if you are adding a new board and haven't told it yet.
> +
> +- Toggle a GPIO from the board entry point
> +
> +  A number of platforms (e.g. i.MX or STM32MP) have header-only GPIO helper
> +  functions that can be used to toggle a GPIO. These can be used for
> +  debugging early hangs by toggling an LED for example.
> +
> +- Trace BootROM activity
> +
> +  If you have no indication that the barebox prebootloader is being started,
> +  consider tracing what the BootROM is doing, e.g. via JTAG or a logic analyzer
> +  for the SD-Card.

s/SD-Card/SD card/  <-- this spelling seems to dominate within barebox.

> +
> +If you managed to get some serial output, move along to the next step.
> +
> +Hang after first stage PBL console output

s/Hang after first stage PBL console output/Hang after First Stage PBL Console Output/

> +=========================================
> +
> +The first stage prebootloader handles:
> +- Basic initialization (e.g., clocks, SDRAM)
> +- installation of secure firmware if applicable

s/installation/Installation/

> +- invocation of the second stage

s/invocation/Invocation/

> +
> +**Common problems**:
> +
> +- issues in board entry point

s/issues/Issues/

> +- Hang in firmware
> +
> +**What to try**:
> +
> +- Check where hang occurs
> +
> +  If you get just some early output, you'll need to pinpoint, where the issue
> +  occurs. if enabling ``CONFIG_PBL_CONSOLE`` along with a correctly configured

s/if enabling/If enabling/

> +  ``CONFIG_DEBUG_PBL`` doesn't help, try adding ``putc_ll('@')`` (or any other
> +  character) to find out, where the startup is stuck. ``putc_ll`` has the
> +  benefit of being usable everywhere, even before ``setup_c()`` is or
> +  ``relocate_to_current_adr()`` is called. Once these are called, you may
> +  also use ``puts_ll()`` or just normal ``printf`` if ``CONFIG_PBL_CONSOLE=y``.
> +
> +- Check if hang occurs in other loaded firmware
> +
> +  On platforms like i.MX8/9 and RK35xx, barebox will install ARM trusted
> +  firmware as secure monitor and possibly OP-TEE as secure OS.
> +  Hangs can happen if TF-A or OP-TEE is configured to access the wrong
> +  console (hang/abort on accessing peripheral with gated clock).
> +  If output ends with the banner of the firmware, jumping back to barebox
> +  may have failed. In that case, double check that the memory size
> +  configured for TF-A/OP-TEE is correct and that the entry addresses
> +  used in barebox and TF-A/OP-TEE are identical.
> +
> +Hang during chainloading

s/Hang during chainloading/Hang During Chainloading/

> +========================
> +
> +Once basic system initialization is done, barebox prebootloader
> +will load the second stage.
> +
> +**Common problems**:
> +
> +- wrong SDRAM setup

s/wrong/Wrong/

> +- corrupted barebox proper read from boot medium

s/corrupted/Corrupted/

> +
> +**What to try**:
> +
> +- Check computed addresses
> +
> +  If your last output is ``jumping to uncompressed image``, this suggests that
> +  the hang occured while trying to execute barebox proper. barebox prints
> +  the regions it uses for its stack, barebox itself and the initial RAM
> +  as debug output. Verify these with the actual size of RAM installed and
> +  check if values are sane.
> +
> +- Check that barebox was loaded correctly
> +
> +  You can enable ``CONFIG_COMPILE_TEST`` and ``CONFIG_PBL_VERIFY_PIGGY``
> +  to have the barebox build system compute a hash of barebox proper,
> +  which the prebootloader will compare against the hash it computes
> +  over the compresed data read from the boot medium.
> +
> +- Check SDRAM setup
> +
> +  SDRAM setup differs according to the RAM chip being used, the System-on-chip,

s/System-on-chip/System-on-Chip/

> +  the PCB traces between them as well as outside factors like temperature.
> +  When a System-on-Module is used, the hardware vendor will optimally provide
> +  a validated RAM setup to be used. If RAM layout is custom, the System-on-Chip
> +  vendor usually provides tools for calculating initial timings and tuning them
> +  at runtime.
> +
> +  Because writes can be posted, issues with wrongly set up SDRAM may only become
> +  apparent on first execution or read and not during mere writing.
> +
> +  Issues of writes silently misbehaving should be detectable by
> +  ``CONFIG_PBL_VERIFY_PIGGY``, which reads back the data to hash it.
> +
> +  If the prebootloader is already running from SDRAM, boot hangs due to completely
> +  wrong SDRAM setup are less likely, but running a memory test from within barebox
> +  proper is still recommended.
> +
> +- Check if an exception happened
> +
> +  barebox can print symbolized stack traces on exceptions, but support for that
> +  is only installed in barebox proper. Early exceptions are currently not enabled
> +  by default, but can be enabled manually with ``CONFIG_ARM_EXCEPTIONS_PBL``.
> +
> +Preinitcall Stage
> +=================
> +
> +The prebootloader ``barebox_pbl_start`` ends up calling ``barebox_non_pbl_start``
> +in barebox proper. This function does:
> +
> +- relocation and setting up the C environment
> +- setting up the malloc area and KASAN
> +- calling ``start_barebox``, which runs the registered initcalls
> +
> +**Common problems**:
> +
> +- None, this is quite straight-forward code
> +
> +**What to try**:
> +
> +- Check if the code is executed. This can be done with ``putc_ll``. ``printf``
> +  is not safe to use everywhere in this function, because the C environment
> +  may not be set up yet.
> +
> +initcall Stage

s/initcall/Initcall/

> +=================
> +
> +After decompression and jumping to barebox proper, barebox will walk through
> +the compiled in initcalls.
> +
> +**Symptoms**:
> +
> +- Hangs after PBL output but before typical barebox banners
> +
> +**What to try**:
> +
> +- Enable ``CONFIG_DEBUG_INITCALLS`` while ``CONFIG_DEBUG_LL`` is enabled
> +
> +  This shows output for each initcall level, helping pinpoint where execution stops.
> +  ``CONFIG_DEBUG_LL`` is useful here, because it allows showing output, even
> +  before the first serial driver is probed.
> +
> +Driver Probe Stage
> +==================
> +
> +Initcalls don't necessarily correspond to driver probes as a driver may be
> +registered before a device or the device probe is postponed until resources
> +become available.
> +
> +**Symptoms**:
> +
> +- Hangs during hardware initialization
> +
> +**What to try**:
> +
> +- Enable``CONFIG_DEBUG_PROBES``
> +
> +  This prints each driver probe attempt and can help isolate the problematic peripheral.
> +
> +- Disable drivers selectively to see if a shell can be reached.
> +
> +Interactive Console
> +===================
> +
> +If you see output only with ``CONFIG_DEBUG_LL``, but not otherwise, you may not
> +have any consoles enabled or you are looking at the wrong console.
> +
> +For testing, you can enable ``CONFIG_CONSOLE_ACTIVATE_ALL`` to have barebox
> +proper print out logs on all console devices that it registers.
> +
> +Once you have the correct console figured out, consider enabling the option
> +``CONFIG_CONSOLE_ACTIVATE_ALL_FALLBACK``. This will fall back to activating all
> +consoles, when no console was activated by normal means (e.g. via the environment
> +or the device tree ``/chosen/stdout`` property).
> +
> +Kernel hang

s/Kernel hang/Kernel Hang/

> +===========
> +
> +**Symptoms**:
> +
> +- Hang after a line like
> +  ``Loaded kernel to 0x40000000, devicetree at 0x41730000``
> +
> +With kernel hangs, it's important to find out, whether the hang happens in barebox
> +still or already while executing the kernel.
> +Without EFI loader support in barebox, there is no calling back from kernel to barebox,
> +so a kernel hanging is usually indicative of an issue within the kernel itself.
> +
> +It's often useful to copy the kernel image into ``/tmp`` instead of booting directly
> +to verify that the hang is not just a very slow network connection for example.
> +The ``-v`` option to :ref:`command_cp` is useful for that.
> +The file size copied may differ from the original if the mean of transport rounds
> +up to a specific block size. In that case, round up the size on the host system
> +and run a digest function like :ref:`command_md5sum` to check  that the image
> +was transferred successfully.
> +
> +If the image is transferred correctly, the :ref:`command_boot` verbosity is increased
> +by each extra ``-v`` option. At higher verbosity level, this will also print out
> +the device tree passed to the kernel. The :ref:`command_of_diff` command is useful
> +to :ref:`visualize only the fixups that were applied by barebox to the device tree<of_diff>`.
> +
> +If you are sure that the kernel is indeed being loaded, the ``earlycon`` kernel
> +feature can enable early debugging output before kernel serial drivers are loaded.
> +barebox can fixup an earlycon option if ``global.bootm.earlycon=1`` is specified.
> +
> +Spurious aborts/hangs

s|Spurious aborts/hangs|Spurious Aborts/Hangs|

> +=====================
> +
> +**Symptoms**:
> +
> +- Hangs/Panics/Aborts that happen in a non-deterministic fashion and whose

s|Hangs/Panics/Aborts|Hangs/panics/aborts|

> +  probability is greatly influenced by enabling/disabing barebox options
> +  and corresponding shifts in the barebox binary
> +
> +It's generally advisable to run a memory test to verify basic operation and to check
> +if the RAM size is sane. barebox provides two commands for this: :ref:`command_memtest`
> +and :ref:`command_memtester`. In addition, some silicon vendors like NXP provide their
> +own memory test blobs, which barebox can load to SRAM via :ref:`command_memcpy` and
> +execute using :ref:`command_go`. By having the memory test outside DRAM, a much more
> +thorough memory test is possible.
> +
> +With ``CONFIG_MMU=y``, the decompression of barebox proper in the prebootloader
> +and the runtime of barebox proper will execute with MMU enabled for improved performance.
> +
> +This increase in performance is due to caches and speculative execution.
> +barebox will mark memory mapped I/O devices and secure firmware as ineligible for
> +being accessed speculatively, but it can only do so if the memory size it's told
> +is correct and if secure memory is marked reserved in the device tree.
> +
> +The memory map as barebox sees it can be printed with the :ref:`command_iomem`
> +command. Everything outside ``ram`` region is mapped non executible and uncacheable

s/executible/executable/

Best regards
Ulrich


> +by default. Everything inside ``ram`` regions that doesn't have a ``[R]`` next
> +to it is cacheable by default. The :ref:`command_mmuinfo` command can be used
> +to show specific information about the MMU attributes for an address.
> +
> +Memory Corruption Issues
> +========================
> +
> +Some hangs might be caused by heap corruption, stack overflows, or use-after-free bugs.
> +
> +**What to try**:
> +
> +- Enable ``CONFIG_KASAN`` (Kernel Address Sanitizer)
> +
> +  This provides runtime memory checking in barebox proper and can detect
> +  invalid memory accesses.
> +
> +  .. warning::
> +     KASAN gratly increases memory usage and may itself cause hangs in
> +     constrained environments.
> +
> +
> +Summary of Debug Options
> +========================
> +
> ++-----------------------------+-------------------------------------------------------+
> +| Option                      | Description                                           |
> ++=============================+=======================================================+
> +| CONFIG_DEBUG_LL             | Early low-level UART output                           |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_PBL_CONSOLE          | Print statements from PBL                             |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_DEBUG_PBL            | Enable all debug output in the PBL                    |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_PBL_VERIFY_PIGGY     | Verify barebox proper in PBL before decompression     |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_ARM_EXCEPTIONS_PBL   | Enable exception handlers in PBL                      |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_DEBUG_INITCALLS      | Logs each initcall                                    |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_DEBUG_PROBES         | Logs each driver probe                                |
> ++-----------------------------+-------------------------------------------------------+
> +| CONFIG_KASAN                | Detects memory corruption                             |
> ++-----------------------------+-------------------------------------------------------+
> +
> +Final Tips
> +==========
> +
> +- If all else fails, a JTAG debugger to single-step through the code can
> +  be very useful. To help with this, ``CONFIG_PBL_BREAK`` triggers an
> +  exception at the start of execution of the individual barebox stages,
> +  which ``scripts/gdb/helper.py`` can use to correctly set the base
> +  address, so symbols are correctly located.
> diff --git a/Documentation/devicetree/index.rst b/Documentation/devicetree/index.rst
> index 94e8d04f63c3..4f25b6c6869b 100644
> --- a/Documentation/devicetree/index.rst
> +++ b/Documentation/devicetree/index.rst
> @@ -175,6 +175,8 @@ In the ``chosen``-node, barebox fixes up
>  These values can be read from the booted linux system in ``/proc/device-tree/``
>  or ``/sys/firmware/devicetree/base``.
>  
> +.. _of_diff:
> +
>  To see a dry run of what barebox would fixup, the ``of_diff`` command can be
>  used::
-- 
Pengutronix e.K.                           | Ulrich Ölmann               |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |