Handling A LOT of data from serial

[linfan68]

Our system is using ESP32 serial port heavily (we do a lot of processing on a lot of data)

I just noticed an issue that when the system is busy, some data get lot on serial port. Here is my code of processing serial data:

  private onReadable(count: number) {
    const newBuf = this._serial.read(count) as ArrayBuffer
    trace('this._serial.read: ' + Uint8ArrayToString(new Uint8Array(newBuf), 'hex') + '\n')
    if (newBuf.byteLength + this._rxBufferLen > RX_LIMIT) {
      trace('RX buffer overflow\n')
      return
    }

    if (this._rxTimeout) {
      Timer.clear(this._rxTimeout)
      this._rxTimeout = undefined
    }
    this._rxBuffer.set(new Uint8Array(newBuf), this._rxBufferLen)
    this._rxBufferLen += newBuf.byteLength

    this._rxTimeout = Timer.set(() => {
      const startMs = new Date().getTime()
      trace(`>>>>>> RX timeout, len=${this._rxBufferLen}\n`)
      this._receiveHandler(this._rxBuffer.slice(0, this._rxBufferLen))
      this._rxBuffer = new Uint8Array(RX_LIMIT)
      this._rxBufferLen = 0
      this._rxTimeout = undefined
      trace(`<<<<< RX timeout done ${new Date().getTime() - startMs}ms\n`)
    }, this._params.uartInit.txrx.rxTimeoutMs)
  }

I'm trying to accumulate data from port and process them when timeout happens.
But I noticed the log output looks like this:

this._serial.read: 3739abababab060 // More data...
this._serial.read: 2c412a
>>>>>> RX timeout, len=100
// Lots of processing 
<<<<< RX timeout done 255ms

Basically, the processing takes ~300-500ms. And to my understanding, during that time the onReadable() callback cannot be called, since it's a JS function.
After reviewing the code for /modules/io/serial/esp32/serial.c, I was unable to locate any buffer in the native code. Therefore, it is likely that the data is dependent on the hardware's FIFO. In such a scenario, it is possible that the data may be lost.

So, if I understand correctly, it may be necessary to modify the serial.c code to include a software buffer?

[phoddie]

You are correct that the serial implementation uses the hardware FIFO and that can overflow if not read often enough. Depending on the baud rate, your 300-500 ms processing has the potential for overflows.

One solution would be to add a software buffer to the serial implementation. That shouldn't be too difficult. Another approach would be to move the serial I/O to a worker. The worker runs in parallel, and can accumulate the data before passing it to the main task for processing. That has the advantage of keeping everything in JavaScript/TypeScript and being portable. I recall you mentioning that your hardware has lots of RAM, so a few KB for the worker won't be a concern.

FWIW – your onReadable implementation could be more efficient:

• You can read directly into an existing buffer to eliminates data copying and reduce GC activity - something like this._serial.read(this._rxbuffer.subarray(this._rxBufferLen, this._rxBufferLen + read))
• instead of recreating the _rxTimeout timer on each callback, you can reschedule it using Timer.schedule (docs). That will minimize the number of instances created and GC activity.

Note that the trace statements aren't free. They go out over serial too, and so they take some time, especially if the amount of data being written is large. The line trace('this._serial.read: ' + Uint8ArrayToString(new Uint8Array(newBuf), 'hex') + '\n') looks like it could be writing more data than is received. I understand for debugging that the tracing can be useful, but to understand the performance of the code (and in production!) you probably want to disable that.

[linfan68]

Got it! I also read the document on IO system design and your explanation makes perfect sense. The entire system has made appropriate compromises in the right places.

[phoddie]

Glad it all makes sense to you. I appreciate that you understand that API design is always a set of tradeoffs - providing the required functionality, ease of use by developer, efficiency, maintainability, security, etc.

[linfan68]

@phoddie quick question regarding using worker:

from the document, it said there is a way to use worker class (instead of module? I guess...)

class MyWorker extends Worker {
	onmessage(message) {
		trace(message, "\n");
	}
}
...
let aWorker = new MyWorker("simpleworker");

I'm still receiving an error message from the runtime stating that it cannot locate the 'simpleworker' module. Where I should place this class?

[phoddie]

The argument to the Worker constructor is a module specifier. The documentation explains that this way:

To launch a worker, create an instance of the Worker class, passing the name of the module to invoke when the worker starts to run.

In the SDK's worker example, the "simpleworker" is a module declared in in the manifest as usual:

moddable/examples/base/worker/manifest.json

Lines 24 to 29 in 787bf7c

	"modules": {
	"*": [
	"./main",
	"./simpleworker"
	]
	}

You can programmatically check if the module is available. Maybe that is useful for debugging here. Add $(MODDABLE)/modules/base/modules/manifest.json to your project. Then add there two lines to your main.js:

import Modules from "modules"

trace(Modules.host, "\n");

In the SDK's worker example, that outputs: time,timer,Resource,worker,modules,main,simpleworker,mc/config.

A couple things to note about the worker module:

• Like main.js, it usually isn't preloaded. It needs to run in the context of the worker which is created at runtime, and so doesn't exist yet at preload time. (Like main.js, it can be preloaded to by exporting a default function to perform the initialization when loaded on the device)
• It doesn't need to have any exports. It is more of a script that initializes self.onmessage to receive messages. Of course, it can contain classes and import other modules.

[linfan68]

So I should define MyWorker class inside simplewpker.js file, correct?

[phoddie]

So I should define MyWorker class inside simplewpker.js file, correct?

Short answer: yes.

A little more precisely: you need to define it in a place that is visible to simpleworker.js. One way to do that is to define it in simpleworker.js. Another approach, perhaps preferred from a memory perspective, is to put the MyWorker class in its own module. Then, you can import that module from simpleworker.js.