Adding a coroutining example and explaining how it can be really useful

[TerryE]

Now that I am really getting to grips with the NodeMCU Lua execution model in my bones I've realised that we can use Lua coroutining for a whole class of long running applications and still stay within the SDK strictures.

In essence the dilemma of the SDK is that tasks are recommended to be no more than 15 mSec in length and so the standard way of tackling this is by creating CB event chains, but another simple way is to use coroutining instead. For example if I do a simple recursive walk of _G, ROM and the Registry then this will crash-out with a WDT error unless I smatter the code with tmr.wdclr() that could in turn cause the network stack to fail. So here is how to do it using coroutining:

do
  local function batch_tree_walk(taskYield, list)  
    local s, n, nCBs = {}, 0, 0
     
    local function list_entry (name, v) -- upval: taskYield, nCBs
      print(name, v)
      n = n + 1
      if n % 20 == 0 then nCBs = taskYield(nCBs) end
      if type(v):sub(-5) ~= 'table' or s[v] then return end
      if name == 'Reg.stdout' then return end  -- needed for telnet !!
      s[v]=true
      for k,tv in pairs(v) do
        list_entry(name..'.'..k, tv)
      end
      s[v] = nil
     end
     
     for k,v in pairs(list) do
       list_entry(k, v)
    end
    print ('Total lines, print batches = ', n, nCBs)
  end

  local co = coroutine.create(batch_tree_walk)

  local function taskYield(nCBs)  -- upval: co
    node.task.post(function () -- upval: co, nCBs 
      coroutine.resume(co, nCBs)
    end)   
    return coroutine.yield() + 1
  end 

  coroutine.resume(co, taskYield, {_G = _G, Reg = debug.getregistry(), ROM= ROM})
end

For my minimal test build this prints out just under 400 lines with each task printing out 20; and it works fine over telnet as well.

So do you understand how this works, and should we add it as an example / FAQ discussion point?

At the end of this the Lua GC collects the lot (since the coroutine state is in an upvalue that gets descoped). You can also have other Lua and SDK CBs transparently slotting in between the slices.

Really neat!

PS: and a bonus point for anyone who can tell me what the array s is doing here.

[nwf]

s is preventing cycles: it's the set of items currently being processed by the recursive list_entry calls. :) (Would it be worth making it a weak table? If the elements would otherwise get GC'd during traversal, one presumably doesn't wish for s to be the reason they aren't. You'd have to switch from keeping v live across the loop in list_entry to actually using s like a stack as well as a map, but these seem like easy changes?)

I'm curious what 'Reg.stdout' is all about?

I think an example of coroutine-ing long-running computations is a great idea, though I will confess I've never personally had an event handler run for so long that I feared bumping up against the watchdog.

On the point of "working over telnet", though, I wonder if there's an argument to be made that pipes should apply back-pressure to their producers, rather than relying on the producers to rate-limit themselves (so as not to exhaust memory). The fifo module has a sneaky cheat in it in that one can stick functions into the queue as well, allowing the producer to wait until sends happen and memory has been released (or made GC-able). Is it possible to detect in Lua if code is being called from within a coroutine, and if so, could it be worth making pipe call coroutine.yield() and node.task.post() as in this example?

[TerryE]

s is preventing cycles:

Spot on. 👍 This exploits the feature that Lua keys can be just about any type of valuable and in this case the table itself. You do need this sort of check because _G is in _G, etc. You could only get GC of referenced entries if weak tables are used and of course you could GC a table that is in s anyway.

I'm curious what 'Reg.stdout' is all about?

stdout in the registry is the stdout pipe used with my new #2836 changes. If you asre using telnet etc., then printed fields get sent to the stdout pipe which is emptied by a net socket-based reader. stdout[1] is the CB reader function and stdout[2]... are the UData buffers which are being filled by printing the tree walk. These support tostring() for debugging so print(stdout[2]) will print the contents of the 1st UData slot, etc. And this fills the pipe faster than it can be emptied, so this is a fatal PANIC if printing is being spooled to stdout!

BTW build #2836 and load onto an ESP. You will find that you can just bulk paste into the UART or telnet and it just works. No data drop.

I will confess I've never personally had an event handler run for so long that I feared bumping up against the watchdog.

Ditto usually, but every so often if I have a "batch-like" something to do (this tree-walk is a case in point) then converting it to a CB chain can be a total PITA. What coroutining allows is another class of usecases where the procedural logic can be moved into a coroutine.

there's an argument to be made that pipes should apply back-pressure to their producers

That's the advantage of having 3 priorities that you can task at. IIRC CBs run at priority 1. In general the less real time, the longer the task then the lower the priority should be, so the UART ISR posts to the input task which empties the UART into the stdin pipe at high priority. The interpreter loop runs at low priority and only processes one line at a time. This means that the stdin pipe does pick up the slack, but this also means that it is almost impossible to overflow the input. So when I was testing this cohelper.lua module for example, I just use miniterm all of the time. I temporarily top and tail the module with file.putcontents('cohelper.lua', [==[ and ]==]) so I can cut and past it into the terminal session up update the file. Far faster than Esplorer upload.

[HHHartmann]

I recently made a framework which allows to write code like

for i = 1 to 10
  tcp.send("Hello World")
end

waiting for the sent callback before returning control.
or

  gpio.write(pin, gpio.HIGH)
  fw.wait(interval)
  gpio.write(pin, gpio.LOW)
  fw.wait(interval)

where the wait is non blocking

New Methods wrapping a callback and returning its call parameters can be created easily, e.g. wrapping the http module.
See https://github.com/HHHartmann/nodemcu-syncro/ if you like.

[TerryE]

@HHHartmann Gregor, if you leave the reference to such implementations here in an issue, they will get forgotten when the issue closes. Perhaps we should consider adding a link to the FAQ or even the cohelper README.

[TerryE]

This example has been merged so I am closing this issue.