Improving debug / development support in our builds

[TerryE]

This is probably closer to a feature request rather than a bugrep, but more of a lessons-learnt from my developing the LFS (Lua Flash Store) as discussed in #2068, and the changes that I needed to make to support the debugging environment. These changes were not LFS-specific, but they were needed to support useful debugging of LFS.

The ./Makefile and app/Makefile magic already support the concept of debug and release flavours -- though this is always defaulted to debug. I think that there is a good justifcation for revisiting this and properly enabling these two flavours:

• release. Used to build standard images where you as a developer are only working in Lua and don't want access to image internals. This will build the current default module list. This will be the default and the normal option for Docker builds, and typically not modified, other than the normal user configuration changes.

• debug. Used to build debug images where you as a developer are working on one or more C modules / library functions and therefore want access to image internals, and to use the GDB stub. You are going to be cycling through bugfix builds, so minimising flash times is essential, so this build should be be default an absolute cutdown module and option set, and a small SPIFFS at a fixed offset for rapid SPIFFS reimaging. You are also probably going to be connecting a second USB serial chip to GND/TDX1 for extra diagnostic output. Developers will be typically working in Linux (possibly in a VM), and this is core which will be extended / customised to their needs.

What I propose is to make this a separate PR covering the changes that I needed for LFS development:

• Makefile. Changes to enable remote debug (-ggdb compile flag) and -O2 instead of -Os. We've talked about this last one and this should be standard for dev and master as there is a roughly %3 increase in the image size, but a huge drop in unaligned flash exceptions without the unaligned string constants. A no-brainer, IMO, and should also be the default for release builds.

• esp-gdbstub/gdbstub.c. Fix compile warning.

• include/user_config.h, include/user_modules.h. I have a minimal configuration for testing to reduce reflash times to the minimum. Also a fixed 32b SPIFFS at a fixed 0x100000 offset for the same reason. What I suggest is that we extend the use of the DEVELOP_VERSION define to group configuration options into core and non-core, so that the standard versions generate a minimal build if DEVELOP_VERSION is defined

• libc/c_stdio.h, libc/dbg_printf.c, libc/Makefile, driver/uart.c. Extra function dbg_set_uart which enables the dbg_printf to go UART1 instead of UART0. (If you output debug diagnostics to UART0 before the GDB stub is initialised, then this can prevent the successful connection of the GDB stub to debug_set_uart). Also export a previously internal uart1_write_char function.

• modules/gpio.c. Bugfix to allow gpio.c to compile if GPIO_INTERRUPT_ENABLE is not defined and DEVELOP_VERSION is.

• user/user_exceptions.c. Fix bug in exception chaining when in GDB stub debugging.

Yes, this patch would also impact on how we document all of this as discussed in #1862, and I am happy to take this one on as well. If I get general support then I will raise this as a separate PR to LFS one.

[TerryE]

I have now really got to grips with the make system and its subtleties, but also in the detailed debugging of all of the main build variants for #2068, I have sorted out a lot of the usability issues with the remote debugger, and I am now finding it a really powerful tool even though some annoying limitations remain. So as it stands:

• I fixed the bug which meant that the debugger and non-aligned exception handler didn't interoperate properly.
• I have changed the gdbstub initialisation to be lazy, that is you don't have to initialise the stub until you need to use it. And hence I have a version of lua_assert() which calls a break routine. Hence you can compile with asserts enabled and test interactively using the UART. If the assert fires, the break routine if not already initialised than now prints an information message to start up the xtensa debugger; it then starts up the remote clients, and issues a break to enter it. So you can be testing in screen or ESPlorer and when you get this message, you can leave this session and start up the host debugger which will them synchronise with the client. frame 1 is the failing routine and you can now examine data etc., and even continue execution robustly.
• I am extending the debugger UI by adding ESP8266 Lua .gdbinit with macros, e.g. dumpRAMstrt dumps out the RAM string table, prTV o prints out the TValue o in readably format, etc.
• I have added a hook into lua_main so that if DEVELOPMENT_USE_GDB and BREAK_ON_STARTUP_PIN are defined, then it will check to see if the corresponding pin is pulled low and start the debugger. This allows you to have a development build which doesn't start the debugger at boot, so you can test using the UART, then if you do find an issue, you can modify init.lua to trigger it and then reboot with the pin pulled low to step through the sequence, and track down the root cause.

I couldn't have done the LFS patch without all this so I will be including these features in the patch. But the scope of debugging is quite different hence this issue.

I've also picked up a couple of benign bugs in the elua base code (e.g. enabling lua_assert() cause an assert failure if you try to print a function variable pointing to a lightweight CFunction. This leads me to suspect that eLua has never been tested with lua_asserts enabled.

All of this experience would really help most C library developers in the development of their code, so another C developers FAQ is definitely needed.

[TerryE]

Just as a footnote, this allows some really powerful ways of using the debugger:

• You can have a development build which runs normally -- so long as nothing fails, and in which case you can "talk" to the Lua interactive prompt using the UART.
• You can configure lua_assert() so that if an assert fail, then the stub tells you to come of of the interactive session and star the xtensa debugger. This will then sync with the remote client and you have full debugger diagnostics available to work out the root cause of the issue.
• You can also configure an I/O pin so that if you pull it low for restart, then the firmware will enter the debugger before starting the LUa RTS, allowing you to set a hard/ware watch / break point and continue.
• OK, having done this means that you've lost the UART for interactive work, but you can still hook your interactive prompt into a telnet session.
• In this mode, any exceptions will throw to into the debugger for you to diagnose the root cause of the exception, rather than simply rebooting the ESP.

If you are a developer of C libraries, once you start using this, then you won't go back to the old techniques.

[TerryE]

Having got into power debugging, and resolved most of the issues, what we are really left with is a documentation issue. In terms of programmatic functionality, this issue is effectively done.