Template processor corrupts output

[DeeEmm]

Outline of problem

I am using ESPAsyncWebserver to serve webpages that contain sensor data.

I have a single monolithic minified file that contains the HTML + CSS + Javascript components of the web page. This single file structure is a necessity determined by space constraints.

My setup is working but I have decided to move from websockets to templating to address some unrelated interrupt issues.

Enabling the template function results in the % characters being stripped from the output. This has already been reported in bug #644 and a fix proposed with PR #737. However this issue is still present and remains unaddressed. The PR was never merged and the bug auto closed due to inactivity.

Things I have tried

Several solutions were proposed within the discussions around bug #644 , but none of them are working in my case...

1- Delimiting the % character with %%. Whilst I disagree the delimiting the % within the page by using %% is a valid solution, I did try it and found that it does not actually work as it simply creates invalid CSS. (no surprise there really).

2- Changing the delimiter. I tried changing the delimiter by defining an alternate TEMPLATE_PLACEHOLDER character that is unused elsewhere in my code '$', but this still strips out the % characters in addition to the alternate character. (kinda weird TBH)

3- PR #737 - I manually added the PR code to test this out, however it also fails to address the issue and the output is still corrupted. It also does not work even in conjunction with solutions 1 and 2.

Even if solution 3 did work it's not really viable as my project is in the public domain and so I do not want to have to support customised libraries on top of supporting the project itself. My preference is to use vanilla libraries to avoid complexity and the avoidable support issues that would arise from running a custom library.

Solution needed

So I am kind of stuck needing to get this to work. I cannot use websockets and I don't really want to have to resort to writing a templating engine as this is basically doubling up on what should already work.

What's the status of this project? Is a fix for this issue even likely given that the previous issue's fix was not integrated into the main code?

I expected that changing the delimiter would be an easy fix but the templating is not functioning as expected.

Just in case it's a dumb issue of my own creation, here's a sample of my code...

index.html

    <!-- Status Modal -->
    <div id="statusModal" class="modal">
      <div class="modal-content">
        <span class="closeStatusModal">&times;</span>
          <h4>Storage</h4>
          <hr>
          <div class="config-row"><label class="config-label">Memory Size:  </label><span class="config-value" id="xSPIFFS_MEM_SIZE">%SPIFFS_MEM_SIZE%</span> (bytes)</div>
          <div class="config-row"><label class="config-label">Memory Used:  </label><span class="config-value" id="xSPIFFS_MEM_USED">%SPIFFS_MEM_USED%</span> (bytes)</div>
      </div>        
    </div>

webserver.cpp

...
      if (SPIFFS.exists("/index.html")) {
        request->send(SPIFFS, "/index.html", "text/html", false, processTemplate);
        // request->send(SPIFFS, "/index.html", "text/html");
      } else {
        request->send(200, "text/html", index_html); 
      } });

...

String Webserver::processTemplate(const String &var)
{

  extern struct DeviceStatus status;

  if (var == "SPIFFS_MEM_SIZE") return F(status.spiffs_mem_size);
  if (var == "SPIFFS_MEM_USED") return F(status.spiffs_mem_used);

  return String();
}

The code is truncated for brevity

Any help / pointers / ideas welcomed.

TIA

/DM

[barbiani]

I have been using the esp home fork without any troubles.

…

On Tue, Nov 15, 2022, 23:33 Mick Percy ***@***.***> wrote: ##Outline of problem I am using ESPAsyncWebserver to serve webpages that contain sensor data. I have a single monolithic minified file that contains the HTML + CSS + Javascript components of the web page. This single file structure is a necessity determined by space constraints. My setup is working but I have decided to move from websockets to templating to address some unrelated interrupt issues. Enabling the template function results in the % characters being stripped from the output. This has already been reported in bug #644 <#644> and a fix proposed with PR #737 <#737>. However this issue is still present and remains unaddressed. The PR as never merged. Several solutions were proposed within #644 <#644> , but none of them are working in my case... 1- Delimiting the % character with %%. Whilst I disagree the delimiting the % within the page by using %% is a valid solution, I did try it and found that it does not actually work as it simply creates invalid CSS. 2- Changing the delimiter. I tried changing the delimiter by defining an alternate TEMPLATE_PLACEHOLDER character that is unused elsewhere in my code '$', but this still strips out the % characters in addition to the alternate character. 3- PR #737 <#737> - I manually added the PR code to test this out, however it also fails to address the issue and the output is still corrupted. It also does not work even in conjunction with solutions 1 and 2. Even if this did work it's not really a manageable solution as my project code is in the public domain so I do not want to have to support customised libraries on top of supporting the project itself. My preference is to use vanilla libraries to avoid complexity So I am kind of stuck needing to get this to work. I cannot use websockets and I don't really want to have to resort to writing a templating engine as this is basically doubling up on what should already work. What's the status of this project? Is a fix for this issue even likely given that the previous issue's fix was not integrated into the main code? Just in case it's a dumb issue of my own creation, here's a sample of my code... index.html <!-- Status Modal --> <div id="statusModal" class="modal"> <div class="modal-content"> <span class="closeStatusModal">&times;</span> <h4>Storage</h4> <hr> <div class="config-row"><label class="config-label">Memory Size: </label><span class="config-value" id="xSPIFFS_MEM_SIZE">%SPIFFS_MEM_SIZE%</span> (bytes)</div> <div class="config-row"><label class="config-label">Memory Used: </label><span class="config-value" id="xSPIFFS_MEM_USED">%SPIFFS_MEM_USED%</span> (bytes)</div> </div> </div> webserver.cpp ... if (SPIFFS.exists("/index.html")) { request->send(SPIFFS, "/index.html", "text/html", false, processTemplate); // request->send(SPIFFS, "/index.html", "text/html"); } else { request->send(200, "text/html", index_html); } }); ... String Webserver::processTemplate(const String &var) { extern struct DeviceStatus status; if (var == "SPIFFS_MEM_SIZE") return F(status.spiffs_mem_size); if (var == "SPIFFS_MEM_USED") return F(status.spiffs_mem_used); return String(); } The code is truncated for brevity Any help / pointers / ideas welcomed. TIA /DM — Reply to this email directly, view it on GitHub <#1234>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYLTJAFA47XOSZFWQOCEPTWIRBXJANCNFSM6AAAAAASBVLFIM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[DeeEmm]

I have been using the esp home fork without any troubles.

with templating and % symbols in the body?

[DeeEmm]

Trawling through the interwebs to try and find a non-invasive solution for this issue has turned up more unresolved issues...

Issue #300 is also unresolved and relates to the same problem

Similarly issue #333 also unresolved and lacking a valid workaround - again specifically realting to % in the output

So that's at least four times that this issue has been raised and not addressed.

None of the solutions proposed are workable. They either create other problems (invalid CSS) or require core hacks (unacceptable).

@me-no-dev is there any chance we can get some traction on this issue? I'm not alone with this problem.

[DeeEmm]

I would like to propose a solution.

Declaring TEMPLATE_PLACEHOLDER as a compile time preprocessor essentially means that it is not accessible to external code unless that code is included, which it cannot be without modifying the library.

The obvious solution is therefore to make TEMPLATE_PLACEHOLDER publicly accessible.

I think that the easiest way to do this is to breakout TEMPLATE_PLACEHOLDER as a public method that defaults to the current '%' character if not called.

That way anyone wanting to use a custom delimiter can call the method, but it will not break existing installations if it is not called.

I think that this is a reasonable compromise between fixing the template engine properly with multi character open and closing template tags a la '<& ... &>' or keeping it simple whilst providing a usable solution.

My current workaround has been to replace all instances of % in my css files with alternate css methods that do not use %. (It is still possible to create fluid layouts without needing % dimensions) but it is essentially just a dirty hack and ultimately not a sustainable solution for a public project. The project will at some point break as someone will invariably use % dimensions, despite documenting the presence of the bug in this library.

Are there any devs that can make this change?

If I fork the code and make the change myself will my PR get merged? or just ignored like the previous solution?

There's currently 67 open pull requests, some dating back to 2017, most now out of date and most with conflicts.

A majority of the issues raised are unanswered

The last commit was back in March

There's zero activity on Gitter

It's kinda beginning to look like this project is dead. A real shame given its popularity.

/DM

[jonathanvanpeteghem]

Sadly it seems the camera lib of the ESP32 has priority ...
I asume this async webserver lib would be more important, How many IOT boards are there with wifi on board?
Are there good alternatives?

I check the githup page of this project every day, to see if something changed. All my esp32 projects are in the freezer, because the Async webserver is not reliable.

[DeeEmm]

Not sure that an alternate board is necessary. There are other ESP32 libraries that do similar / same but perhaps not as extensively or as well documented as this one.

Moving to another library is definitely one possibility, I also have I2C issues which seem to be related to this library, but I need to get to the bottom of that particular problem before deciding what to do.

The thing with libraries such as this is that they all 'mostly' work, but none are 100%. Given that there are almost infinite applications and uses, there will always be fringe cases where there are problems. Devs have to draw the line somewhere as to what they have the time to support, else they would not have time for new projects / family / golf / whatever.

So I think what happens is that when the project is mature and stable, that's as good as it gets. After all, libraries such as this one are just a starting point, they are never meant to be the solution to all problems. For my case, the reality is that it is a minor change to address, so it can be argued that I should just do that change myself, however for me that's not the best outcome. So it comes down to this - either it's my issue to solve, or the devs issue to solve. I think I know which way it will go :)

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.