"Not enough memory" issues.

[autrilla]

Hello! I have the following code:

local DEVICE_TYPE = "A1"
local SERIAL_NUMBER = "AAAAAAA"

local SSID = ""
local PASSWORD = ""
local API_TOKEN = ""
local DEVICE_ID = ""

local status = ""

print("Starting up...")

--Tries to retrieve locals from the file system.
function read_locals_from_fs()
    if (file.open("settings", "r") == true) then
        line = file.readline()
        while (line ~= nil) do
            print(".")
            setting = settings_split(line)
            if (setting[0] == "SSID") then
                SSID = setting[1]
            elseif (setting[0] == "PASSWORD") then
                PASSWORD = setting[1]
            elseif (setting[0] == "API_TOKEN") then
                API_TOKEN = setting[1]
            elseif (setting[0] == "DEVICE_ID") then
                DEVICE_ID = setting[1]
            else
                print("Unknown settings key.")
            end
        end
    else
        return nil
    end
end

--Splits string with format key=value into key and value.
function settings_split(str)
    local t={} ; local i=0
    for str in string.gmatch(inputstr, "[^=]+") do
        t[i] = str
        i = i + 1
    end
    return t
end

--Writes locals to file system
function write_locals_to_fs()
    file.open("settings", "w+")
    file.writeline("SSID="..SSID)
    file.writeline("PASSWORD="..PASSWORD)
    file.writeline("API_TOKEN="..API_TOKEN)
    file.writeline("DEVICE_ID="..DEVICE_ID)
    file.close()
end

function get_settings_from_request(payload)
    local t = {}
    for set in payload:gmatch( "[^&]+" ) do
            print(".")
        t[set:match( "^(.-)=" )] = set:match( "=(.+)$" ) 
    end 
    return t 
end

function connect_to_wifi()
    print("Connecting to WiFi...")
    wifi.setmode(wifi.STATION)
    wifi.sta.config(SSID, PASSWORD)
    wifi.sta.connect()
    tmr.alarm(1, 1000, 1, function()
        if wifi.sta.getip() == "0.0.0.0" or wifi.sta.getip() == nil then
            print("Not yet connected, waiting...")
        else
            tmr.stop(1)
            print("Connected. IP: ", wifi.sta.getip())
        end
        end)
end

read_locals_from_fs();
print("Loaded config:");
print("SSID: ", SSID);
print("Password: ", PASSWORD);
print("Id: ", DEVICE_ID);
print("Api token: ", API_TOKEN);
print("End loaded config.");
print(node.heap())

I send it to the esp8266 with nodemcu 0.96 (I tested with 0.95 too), try to run it and it says it is out of memory. Running print(node.heap()) before running it outputs around 20kB. Is something wrong with my lua? Note that this isn't the whole code I want to use, I had to remove parts and slowly add them until they didn't run anymore. It doesn't seem to matter what I add, I tried with different functions and it still complains about the same thing.

[TerryE]

This is a "how to" Q and not a "I have found a bug" issue. Please use the correct forum for asking this sort of Q: www.esp8266.com 😄

[autrilla]

@TerryE It is not a how to question, it is a "nodemcu can't handle a 100 line program, it complains about not having memory for it" question. The program works, I'm not asking how to make it work. It doesn't work on nodemcu when you try to run all of its parts together, because as I said, it runs out of memory for some reason. So I suspect there is a memory leak somewhere or I'm doing something terribly wrong. I did try replacing all string concats on print method calls for multiple parameters and the issue persists. There is no specific method that makes it complain when I add it and it's really surprising to me it can't handle this program, as there isn't much that actually gets executed.

[dnc40085]

I tried to run it, first with no settings file (ran fine, returned blank values), then with the settings file containing:

SSID=myssid
PASSWORD=mypassword
API_TOKEN=over9000
DEVICE_ID=10646099

Ran it again, and it returned this error test.lua:40: bad argument #1 to 'gmatch' (string expected, got nil) and this was in the globals table line : string <<< SSID=myssid(the "<<<" should be ignored)
never an out of memory error.

[pastukhov]

String value must be quoted.

[autrilla]

@dnc40085 What version of nodemcu are you using? Could you tell me how much heap you have available when you boot up? Could you please try adding this method and see if it runs for you?

--Creates AP so that an Android device can give the ESP 
--the SSID and password of the network it should connect to
--access the internet, and also the API key it should use
function create_ap()
    wifi.setmode(wifi.SOFTAP); cfg = {};
    cfg.ssid = DEVICE_TYPE.."-"..SERIAL_NUMBER;
    cfg.pwd="(FAVEGA)";
    print("Starting up AP with SSID "..cfg.ssid.." and password "..cfg.pwd);
    wifi.ap.config(cfg);srv = net.createServer(net.TCP);status = "Waiting for settings.";
    srv:listen(80, function(conn)
        conn:on("receive", function(conn, payload)
            print("------Received------");
            print(payload);
            print("--------------------");
            if (string.find(payload, "GET / ")) then
                conn:send("<h1>Status: "..status.."</h1>");
                print("Answering");
                elseif (string.find(payload, "POST / ")) then
                    for k, v in pairs(get_settings_from_request(payload:match(".+[\r\n](.-)$"))) do
            print(".")
                        print(k.."="..v);

                        if (k == "SSID") then
                            SSID = v
                            print("Set SSID to "..SSID)
                        elseif (k == "PASSWORD") then
                            PASSWORD = v
                            print("Set password to "..PASSWORD)
                        elseif (k == "API_TOKEN") then
                            API_TOKEN = v
                            print("Set API token to "..API_TOKEN)
                        elseif (k == "DEVICE_ID") then
                            DEVICE_ID = v
                            print("Set device id to "..DEVICE_ID)
                        end
                    end
                    write_locals_to_fs()
                    conn:send("HTTP/1.1 200 OK\n<h1>OK</h1>");
                end
            end)
        conn:on("sent", function(conn) conn:close() end);
    end);
end

@pastukhov what do you mean? Where is an unquoted string?

[TerryE]

@autrilla

I'm not asking how to make it work. It doesn't work on nodemcu when you try to run all of its parts together, because as I said, it runs out of memory for some reason. So I suspect there is a memory leak somewhere or I'm doing something terribly wrong.

The ESP8266 has very limited RAM and SPIFFS resources. Unfortunately these tend to get less with each version because we keep adding new features which leave even less of this resource for your app. Yes you can write an application where individual parts work successfully but the total won't fit in RAM. This is a limitation of the chipset and the programming technique used, and not a bug in itself.

I could explain how to do this but I am not minded to on an issue list about bugs in the firmware. However in short, have you played with lua -i on your dev PC? you type in Lua code and each chunk once syntactically complete is executed by the Lua RTS. The nodeMCU firmware handles callback events just the same way. It has to do this because this is an architectural constraint that the ESP SDK imposes. Lua garbage collects any dead variables and code chunks between these event executions and the only context passed one one event to the next is through globals and upvalues hidden in the Lua registry. So my recommendation is:

• Avoid upvalues for context passed between event callbacks, as its very difficulat to get a handle on memory leaks created by these. Only use globals for this usecase.
• Nil globals once they are no longer needed so that they can be properly GCed.
• Allocate resources and create closures on a just-in-time basis.
• The cost of require or dofile is relatively small, so break your program into overlays one per event and use a small stub function as a callback to load each.

If you structure your Lua this way then you will find that you can run surprisingly large Lua applications within the ESP8266 memory constraints.

[dnc40085]

What version of nodemcu are you using? Could you tell me how much heap you have available when you boot up? Could you please try adding this method and see if it runs for you?

I'm using my own fork of the current dev096 branch that excludes the(MQTT, COAP, U8G, WS2812, CJSON) modules plus a few other changes(Increased firmware burn speed, Starting UART baud rate set to 115200 instead of 9600, no other major changes). What version are you using?

In StationAP mode while disconnected with no init.lua, the heap is 20848, while connected to an AP it's 20344.

When I add that code to the previous code it says not enough memory when I dofile.
I kinda figured it would though since I've run into this very issue before and then I discovered flashMod which made life much simpler. here's an example.

@pastukhov what do you mean? Where is an unquoted string?

I think he's referring to the settings file.

TerryE is right, this thread really does need to be continued on esp8266.com since the NodeMCU issues section is for bugs having to do with the firmware.

[vowstar]

Hello, we have some way to save memory,
http://www.esp8266.com/wiki/doku.php?id=nodemcu-unofficial-faq
But if you want more memory, new esp chip is coming.

[MarsTechHAN]

Also the bugs inside the program may cause this kind of issue...
And also... The number of lines not equal to the compexity of program~
At the end... A good program habbit can prevent you from those kind of issue...
(Like, use local, use index& concat to repleace "a"..“b”......)

I have write some codes... which use about 400 hundred lines in total, it works well : ）

[roccomuso]

@vowstar when new chip will be released? have you something to suggest?

[TerryE]

See #719

[mahersafadi]

I have faced the same problem I Solved it.
Part 1: Lua has 2 types of functions local functions and global ones, the local functions does not take any heap although they contain too many lines, only global functions take heap size. So Make all of your functions are locals just one function is global which you should use it outside the file
Part 2: compile your code to generate ".lc" file and use it using require instead of dofile it is better for heap
Part 3: do not set your code in init.lua just put in the require and call for the global function you made because dofile("secript.lua") takes more twice than dofile(script.lc) and require is the best
I follow these steps and my code now is more than 1000 line contains http server, mqtt, timers and files.... and it is working well with Only 23 kB of heap

[TerryE]

@mahersafadi, if you use the current master or dev then you will see that running lua source files are only about 10% than lc files now, because of my packed lineinfo patch. The other trick is to make your diffferent code phases ephemeral as I describe in my Unofficial FAQ.

[MacDada]

I get the same error ("not enough memory"). Could you please take a look at this commit and see what could be wrong? Before this change everything works.

MacDada/RemoteTemperatureMonitor@a54116c

The log:

> Soft restart by user command
node.restart()
> ���:��S�9��8�H�d��(=H��jq���

NodeMCU 0.9.6 build 20150704  powered by Lua 5.1.4



=======

Welcome to RemoteTemperatureMonitor!

=======



init.lua loaded.
Preparing to safetly load "init2.lua".


> 
Communication with MCU..Got answer! Communication with MCU established.
AutoDetect firmware...

Can't autodetect firmware, because proper answer not received (may be unknown firmware). 
Please, reset module or continue.
init2.lua" will be loaded in 3

"init2.lua" will be loaded in 2

"init2.lua" will be loaded in 1

error loading module 'ds18b20' from file 'ds18b20.lua':
	not enough memory

[TerryE]

Could you please take a look at this commit and see what could be wrong?

@MacDada, you are using an extremely old and no longer supported build. This is the wrong place to ask about the consequences of doing this.

[MacDada]

you are using an extremely old and no longer supported build

@TerryE You were right. I've upgraded the flash and now my code works. Thanks!

[HermanLacuna]

Hi MacDada

I see I have the same build that you used. Could you maybe describe how you upgraded the flash and what files you have used to upgrade it?

[dnc40085]

@HermanLacuna There is a section in the NodeMCU documentation called Flashing the firmware that should help you upgrade to the latest firmware.
The latest firmware can be obtained from nodemcu-build.com

[merotech]

I have same issue of out of memory.
I found out that all strings usage are ram biggest consumer.
Reduce as possible strings and you will don't have out of memory errors.

[TerryE]

Reduce as possible strings and you will don't have out of memory errors

Wrong answer to the problem. Strings are garbage collectable objects. If you load functions when you need them and correctly dereference them when you've done with them, then any strings that they use will be GCed as well. Read my FAQ in the documentation for more discussion on this.

Alternatively wait a week or so and #2292 will a PR in dev so you can in practice have as many functions and string in flash and not in RAM that you want.