Implement low power mode api

[devyte]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: All
• Core Version: latest git today
• Development Env: All
• Operating System: All

Problem Description

PR #6989 provides a low power usage reference example, and it has been merged. Per @d-a-v 's comment, it should be investigated how to unify the current experimental pseudo modes with the reference code in the example, and how to provide a consistent api to the users. Once that api is available, the example in #6989 should be migrated to make use of that api without changing the current behavior of the tests in that example sketch.

MCVE Sketch

See the example merged in #6989.

[devyte]

CC @Tech-TX

[devyte]

Related to discussion in #6642

[Tech-TX]

(laughs) That's not exactly an MCVE, but it'll do. Timed Light Sleep (timer or GPIO wake) and Forced Light Sleep (GPIO wake) would benefit from API wrappers; the rest of the modes are already encapsulated. Both of those Light Sleep modes need the WiFi off, so it'd be more friendly if it were an ESP API and not a WiFi API. Similarly, an ESP API for Forced Modem Sleep would be nice if you're not using WiFi (the last section in the README).

Additionally I'm still researching the low-power boot to see if I can get that working again. I saw something last night in another thread that doesn't solve it exactly, but it certainly helps. During most of the boot the current is below 37mA. The RTC write does something weird to the modem and current only peaks at 53mA then drops to 30mA, but I can't drop it to the 15-18mA 'Forced Modem Sleep' range after that's been done (yet). It's zeroing the target in the RTC for Deep/Light Sleep wakeup to mostly disable the modem. On the plus side, the current only hits ~53mA for 650us which is filterable by an electrolytic cap.

Unfortunately it's not entirely consistent. 1/5 to 1/8 times it does this instead:

I've duplicated the write to RTC inside preinit() and it's occasionally not working the same. The first write in RF_PRE_INIT() seems to always work.

[Tech-TX]

Done! 😴 I beat igrr's results by ~20mA, and it works today. It barely peaks at 54mA, should be able to run with a 50mA PSU if you're not using WiFi. An electrolytic will smooth that ~50mA peak out. Except for the brief peak, the maximum current during boot is ~35mA, and is lower on successive resets after power on. This fixes the issue originally raised here:
#2111

#include <user_interface.h>

#define _R (uint32 *)PERIPHS_RTC_BASEADDR
#define valRTC 0xFE000000  // normal value during CONT (modem turned on)

RF_PRE_INIT() {
  *(_R + 4) = 0;  // value of RTC_COUNTER for wakeup from light/deep-sleep (sleep the modem)
  system_phy_set_powerup_option(2);  // stop the RFCAL at boot
  wifi_set_opmode_current(NULL_MODE);  // set Wi-Fi to unconfigured, don't save to flash
  wifi_fpm_set_sleep_type(MODEM_SLEEP_T);  // set the sleep type to Forced Modem Sleep
  // moving the two SDK commands up here saves another 670us @ 8 mA of boot current :-)
}

void preinit() {
  *(_R + 4) = 0;  // sleep the modem again after another initialize attempt before preinit()
  wifi_fpm_open();  // enable Forced Modem Sleep, not in RF_PRE_INIT or it doesn't sleep
  wifi_fpm_do_sleep(0xFFFFFFF);  // enter Modem Sleep mode, in RF_PRE_INIT causes WDT
  // won't go into Forced Modem Sleep until it sees the delay() in setup()
}

void setup() {
   *(_R + 4) = valRTC;  // Force Modem Sleep engaged, set the modem back to 'normal' briefly
  delay(1);  // and sleep the modem at 18mA nominal substrate current
}

void loop() {
  yield();
}

zoomed out to show the power on:

I'll just leave this here for the ultra-low-power folks. There's no clean way to implement this in an API as it's scattered across 3 different routines. The 4 SDK calls in preinit() that do modem sleep are essentially the same as WiFi.forceSleepBegin() without loading the WiFi library. The maximum instantaneous power is slightly higher with other modules, and the operating current will be higher than the 18mA I'm getting if you have a USB chip, voltage regulator or power LED.

[Tech-TX]

@d-a-v I'll play with the Light Sleep modes (the only two currently not encapsulated) to make sure the optional items can go before the base sleep functions. The Timed Light Sleep needs more setup than the simpler Forced Light Sleep (wakeup with GPIO only). The optional item for Forced Light Sleep are whether you wanted a callback, which GPIO to use for the interrupt, and whether it's LOLEVEL or HILEVEL. I'll see if it'll work out of the order I have them in currently.

Timed Light Sleep was twitchy to get running, and if they don't use a callback the timed sleep is weird (first sleep(x millis), then delay(x millis)) so I'd make the callback required, even if it's empty. Although the SDK call passes micros for the sleep time, it has to be followed by delay(time in micros + 1 milli) so there's no reason to send it micros. Timed Light Sleep can also be woken by an optional GPIO interrupt (also LOLEVEL or HILEVEL). If that's too ugly to add as optional passed parameters then it might work having them set the GPIO interrupt before the API is called. I'll test that and add a message with the results. Give me a few days to verify the variations thoroughly.

If I had the first clue how to encapsulate those two in an ESP class I'd do it, but I'm not up on C++.

Your pseudo modes work great, no need for anything else on them. I can edit the .rst file so that they're documented, if you'd like.

[Tech-TX]

It appears there's WAY more involved in bringing it out of Light Sleep than I thought. What I'd been fighting is that the top two timers on the os_timer list are from inside the SDK, and just copying the values back to the list doesn't re-attach them. I haven't used the os_timers before, so I didn't know that wouldn't work. The top/fastest timer (check_timeouts_timer) is the one that controls everything else on the list. Without that timer, the os_timer list falls apart. Google to the rescue, once I had the right search terms. 😉

It turns out the NodeMCU crowd did it 2 years ago, nodemcu/nodemcu-firmware#1231 (2000 additions, 80 deletions for that PR...) The ugliest part of it is saving and restoring the os_timer list. They had to walk the list and disarm everything, save it, then re-attach all of the timers again after sleep. Here's the short description of their version of Timed Light Sleep: https://nodemcu.readthedocs.io/en/master/modules/wifi/#wifisuspend
and the doc for their version of Forced Light Sleep: https://nodemcu.readthedocs.io/en/master/modules/node/#nodesleep
(same low-power modes, merely different API names than Espressif used for Light Sleep). That PR is a long read.

Looks like they were at SDK 2.2.0 when this went through, and they had support for all of the different interrupt types for the Forced Modem Sleep (wakeup with GPIO), not just HILEVEL/LOLEVEL. I'll dig inside and see how they did that. They're currently at SDK 3.0 as of September.

You don't have to disarm anything to get either of the Forced Light Sleep modes to work, and for Timed Light Sleep you only need to point the timer_list to the last timer on the list. The SDK functions that put it to sleep disable the os_timers after that, but we DO need to save them so we can re-attach them afterwards. The first SDK call wifi_fpm_set_sleep_type(LIGHT_SLEEP_T) disables nearly all os_timers, and the rest are disabled after one of the later Sleep calls (I couldn't find which one). With Devyte's simple copy of the timer struct something weird was going on, and only the top 2 timers were recoverable (two SDK timers). The others on the list was being clobbered (by the SDK?) in the copy. They were there immediately after saving a copy but were gone after sleep.

Any API encapsulation of Forced Light Sleep (either mode) will need to save and restore/restart the os_timers.

Here's the code I was using to save and then view the timer_list:

extern os_timer_t *timer_list;  // get the list of os_timers
os_timer_t * orig_list = timer_list;  // make a copy of the list
 while (timer_list != 0) {  // point at the last os_timer in the list so the SDK can halt them
  Serial.printf("timer_address = %p\n", timer_list);
  Serial.printf("timer_expire  = %u\n", timer_list->timer_expire);
  Serial.printf("timer_period  = %u\n", timer_list->timer_period);
  Serial.printf("timer_func    = %p\n", timer_list->timer_func);
  Serial.printf("timer_next    = %p\n", timer_list->timer_next);
  Serial.printf("=============\n");
  timer_list = timer_list->timer_next;
}

and to restore the list after sleep just

timer_list = orig_list;

but unfortunately the copy only contains the top two (SDK) timers after sleep.

[ArnieO]

@Tech-TX Please forgive me in advance if this is not the right place to comment on LowPowerDemo (#6989 ).

I've been searching for a solid way to master sleep modes (in particular Auto and Forced Modem Sleep), and found your great work today.
I'm about to debug an application where Auto Modem Sleep behaves differently on two Wifi networks, and want to use LowPowerDemo to investigate the issue.

I ran into the following issue:
When uploading this to a virgin ESP-01, I did not succeed connecting to Wifi.
Arduino IDE 1.8.13. I tried several compiler options, including erasing both sketch and Wifi settings.
I also tried using another simpler sketch to connect, then recompiling LowPowerDemo with compiler option that only erase sketch (not Wifi settings) - but still no connection.
The only way I found to work was to comment out lines 408 and 421 (avoiding the "if"-test).
So now I can run the code. However, this code is at an advancement level where I fail to see if this influences on the "Wifi connect time" results - which is important for my investigation.

Do you have an idea why I fail to connect to Wifi with your original code?

Side comment regarding Line 414:
WiFi.setOutputPower(10); // reduce RF output power, increase if it won't connect
I fail to see where in the code RF output power is increased, as this seems to be the only call to WiFi.setOutputPower(x).
Using Wifi connection code that adjusts RF output power sounds potentially useful, so it would be great if you could clarify this point.

[vks007]

@Tech-TX , Thanks for all the pointers. I am trying to write code to go into light sleep with the ability of waking up either via GPIO interrupt or on expiry of a fixed time. So I have to look for a way to suspend and then re-enable the timers after sleep. If I use just the pointer method suggested by you I get back 3 out of 4 timers (instead of 2). These probably are internal timers.
I think suspending and restoring the timers is a bit more complicated than this. I looked at the code nodemcu team have written here for the same requirement and there's a lot going on. There is special handling of the timers with callback too. I am not an advanced user to be able to adapt that code to the need here. maybe somebody else can help.

[ericbeaudry]

@Tech-TX , That's great work there on managing to reduce power at boot! So good that I'm actually struggling to establish a connection once in that state. Would you mind clarify how to safely restore the modem when it's needed.
ESP.forceSleepWake(); isn't cutting it and neither are :

wifi_fpm_close;
wifi_set_opmode(STATION_MODE);
wifi_station_connect();

I must be missing sometime because once in that mode, I cannot reconnect to any network. My guess is I'm unable to wake it up.

[vks007]

@ericbeaudry , you should start with the low power example which comes with the IDE. Load that and see if everything is fine, then remove the parts you don't want form the program and adapt it to your needs.

[kevinstadler]

@Tech-TX thanks so much for all this work and documentation! I just spent several hours trying to track down a weird Heisenbug-like issue where I wasn't able to send my Wemos D1 Mini into timed light sleep, so I thought I'd chip in here regarding this:

Timed Light Sleep was twitchy to get running, and if they don't use a callback the timed sleep is weird (first sleep(x millis), then delay(x millis)) so I'd make the callback required, even if it's empty.

So on my D1 Mini not only is the callback required for timed light sleep to work, the callback also needs to take up some minimum amount of computation/IO time, otherwise the following delay() is waited out in full. The minimum amount of computation I was able to pin down was a Serial.println() followed by a Serial.flush(), either one by themselves won't do. The LowPowerDemo example has Serial output in the callback so it took me quite some triangulation to figure out why my minimal timed light sleep code wasn't working. To summarise, on the D1 Mini it seems that all of the following are necessary to successfully enter and then immediately resume from timed light sleep:

• timer_list = nullptr
• a registered wakeup callback...
• ... which contains at least a Serial.println() and a Serial.flush()
• a delay(sleeptimeMicroseconds / 1000 + 1) right after wifi_fpm_do_sleep(sleeptimeMicroseconds)

You probably have a better idea why this might be the case, here's a full working example for testing, with its output under different conditions below:

#include "user_interface.h" // for keeping time across light sleep based on the RTC

#define WAKE_UP_PIN 0 // D3

long RTCmillis() {
  return (system_get_rtc_time() * (system_rtc_clock_cali_proc() >> 12)) / 1000;
}

void printMillis() {
  Serial.print(F("millis() = "));
  Serial.println(millis());
  Serial.print(F("RTCmillis() = "));
  Serial.println(RTCmillis());
}

void wakeupCallback() {
  printMillis();
  // this flush is crucial, possibly because it is a blocking command that
  // allows the CPU to break out of the delay()? see output below
  Serial.flush();
}

void setup() {
  pinMode(WAKE_UP_PIN, INPUT_PULLUP);
  Serial.begin(115200);
  Serial.println();

  printMillis();
  extern os_timer_t *timer_list;
  timer_list = nullptr;
  wifi_fpm_set_sleep_type(LIGHT_SLEEP_T);
  wifi_fpm_open();
  gpio_pin_wakeup_enable(GPIO_ID_PIN(WAKE_UP_PIN), GPIO_PIN_INTR_LOLEVEL);
  wifi_fpm_set_wakeup_cb(wakeupCallback);
  wifi_fpm_do_sleep(5E6);
  delay(5e3 + 1);
  printMillis();
}

void loop() {
}

So the code prints the CPU millis() as well as the RTC-based time 3 times: once before entering sleep, once inside the callback, and then a third time after the main code continues to execute. Here's what happens based on the content of the callback:

Output of the code above when using the callback with both Serial.println() and Serial.flush()

When I wait for the 5s timer to expire

millis() = 58
RTCmillis() = 49
millis() = 866
RTCmillis() = 4267
millis() = 869
RTCmillis() = 4270

So that's all good, the callback and the post-delay() code follow right after one another, RTCmillis() has quite a bit of drift away from the 5 seconds that actually elapsed, but otherwise everything works as expected.

When I interrupt the timer by pulling the pin low after 3 seconds

millis() = 58
RTCmillis() = 80
millis() = 310
RTCmillis() = 2863
millis() = 313
RTCmillis() = 2866

Again that's all fine, not sure why the CPU's millis() are so much less when the wake-up is through external interrupt but at least the main code continues right after the callback.

Output of the code above, without Serial.flush() in the callback

When I wait for the 5s timer to expire

millis() = 58
RTCmillis() = 57
millis() = 866
RTCmillis() = 4150
millis() = 5061
RTCmillis() = 9141

So the callback is invoked at exactly the same time, but then the delay() holds up the main code for another 4.2 seconds, exactly until 5 seconds have elapsed since going to sleep according to CPU time (which is actually closer to 10 seconds in real time, as can be seen from the RTC).

When I interrupt the timer by pulling the pin low after 3 seconds

millis() = 57
RTCmillis() = 79
millis() = 416
RTCmillis() = 2952
millis() = 5060
RTCmillis() = 8169

Same problem, even after an early external interrupt wake, the delay() blocks the main code for another 5 seconds (real time) before continuing.

[kevinstadler]

The optional item for Forced Light Sleep are whether you wanted a callback, which GPIO to use for the interrupt, and whether it's LOLEVEL or HILEVEL.

It should maybe also be added that it's possible to have more than one GPIO interrupt rule (even with different target states) active at the same time, e.g.:

gpio_pin_wakeup_enable(D1, GPIO_PIN_INTR_HILEVEL);
gpio_pin_wakeup_enable(D2, GPIO_PIN_INTR_LOLEVEL);
wifi_fpm_do_sleep(0xFFFFFFF);

will wake up from light sleep when either D1 is high or D2 is low. Any previously enabled interrupts can be cleared using
gpio_pin_wakeup_disable() (no arguments, clears all at once).

[d-a-v]

Who is going to make a PR with these findings ?
Is the above snippet still valid with latest pushs and particularly the wifi-off-at-boot-time one (#7902) ?