Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

weird timezone issues in influxdb #2273

Closed
srLinux opened this issue Mar 31, 2023 · 8 comments · Fixed by #2278
Closed

weird timezone issues in influxdb #2273

srLinux opened this issue Mar 31, 2023 · 8 comments · Fixed by #2278
Labels
bug Something isn't working

Comments

@srLinux
Copy link

srLinux commented Mar 31, 2023

The Problem

hi,
influxdb client always should send the UTC time. It causes weird issues like "in the future" measurements. Take a look over here: https://community.grafana.com/t/weird-timezone-problems/72815

ksnip_20230331-200845

After i changed the timezone on the webpanel to UTC0 the problem went away. It is maybe a influx/grafana related problem but the recommendation is to set the timezone to UTC in on the measurement sending clients.

Version

rolling 15.1.1

Logfile

-

Expected Behavior

No response

Screenshots

No response

Additional Context

No response

@srLinux srLinux added the bug Something isn't working label Mar 31, 2023
@wetneb
Copy link
Contributor

wetneb commented Apr 1, 2023

Seems to be a similar problem to #2150.

I would still be interested to hear from @jomjol why this curious arithmetic had to be introduced in the first place:

struct tm tm;
strptime(_timestamp.c_str(), PREVALUE_TIME_FORMAT_OUTPUT, &tm);
time_t t = mktime(&tm); // Time in Localtime (looks like timezone is not used by strptime)
struct tm * ptm;
ptm = gmtime ( &t );
time_t utc = mktime(ptm);
utc = 2*t - utc; // Take care of timezone (looks difficult, but is easy: t = t + (t - utc), weil t-utc = timezone)

I agree that sending the time in UTC seems much cleaner.

@jomjol
Copy link
Owner

jomjol commented Apr 1, 2023

I agree in general. Does anybody know, how I get the UTC time in the ESP with a local time zone set?

wetneb added a commit to wetneb/AI-on-the-edge-device that referenced this issue Apr 1, 2023
@wetneb
Copy link
Contributor

wetneb commented Apr 1, 2023

I think you can just use the same code I had written in the first place: #2278

@srLinux
Copy link
Author

srLinux commented Apr 1, 2023

Seems to be a similar problem to #2150.

I would still be interested to hear from @jomjol why this curious arithmetic had to be introduced in the first place:

struct tm tm;
strptime(_timestamp.c_str(), PREVALUE_TIME_FORMAT_OUTPUT, &tm);
time_t t = mktime(&tm); // Time in Localtime (looks like timezone is not used by strptime)
struct tm * ptm;
ptm = gmtime ( &t );
time_t utc = mktime(ptm);
utc = 2*t - utc; // Take care of timezone (looks difficult, but is easy: t = t + (t - utc), weil t-utc = timezone)

I agree that sending the time in UTC seems much cleaner.

sorry, i don't used the search function this time :-)

jomjol pushed a commit that referenced this issue Apr 1, 2023
@jomjol
Copy link
Owner

jomjol commented Apr 1, 2023

I cannot read the time in this functioni due to the following reason:
I want to send the time, when the picture was taken. This is handover as a parameter:
void InfluxDB_V2_Publish(std::string _key, std::string _content, std::string _timestamp)

Unfortunatly the string _timestamp is in local time. Therefore I need to convert it back into UTC. I could not find a function for this.

@jomjol
Copy link
Owner

jomjol commented Apr 1, 2023

I have made a branch to work on this problem: https://github.com/jomjol/AI-on-the-edge-device/tree/debug-influxdb

Please submit updates and changes there.

@jomjol
Copy link
Owner

jomjol commented Apr 1, 2023

Can you test my latest commit here: https://github.com/jomjol/AI-on-the-edge-device/actions/runs/4585284979

Should solve the problem.

@garciaargos
Copy link

Can you test my latest commit here: https://github.com/jomjol/AI-on-the-edge-device/actions/runs/4585284979

Should solve the problem.

I just installed it and this works for me so far, Berlin timezone. Thanks!

@jomjol jomjol closed this as completed in 63ac38a Apr 2, 2023
caco3 added a commit that referenced this issue Apr 23, 2023
* Testcase for #2145 and debug-log (#2151)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Fix timezone config parser (#2169)

* make sure to parse the whole config line

* fix crash on empty timezone parameter

---------

Co-authored-by: CaCO3 <[email protected]>

* Enhance ROI pages (#2161)

* Check if the ROIs are equidistant. Only if not, untick the checkbox

* renaming

* Check if the ROIs have same y, dy and dx. If so, tick the sync checkbox

* only allow editing space when box is checked

* fix sync check

* show inner frame on all ROIs

* cleanup

* Check if the ROIs have same dy and dx. If so, tick the sync checkbox

* checkbox position

* renaming

* renaming

* show inner frame and cross hairs on all ROIs

* update ROIs on ticking checkboxes

* show timezone hint

* fix deleting last ROI

* cleanup

---------

Co-authored-by: CaCO3 <[email protected]>

* restart timeout on progress, catch error (#2170)

* restart timeout on progress, catch error

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* BugFix #2167

* Release 15.1 preparations (#2171)

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update changelog

* Fix links to PR

* Formating

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>

* fix typo

* Replace relative documentation links with absolute ones pointing to the external documentation (#2180)

Co-authored-by: CaCO3 <[email protected]>

* Sort model files in configuration combobox (#2189)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Sort model files in combobox

* reboot task - increase stack size (#2201)

Avoid stack overflow

* Update interface_influxdb.cpp

* Update Changelog.md

* Show PSRAM usage (#2206)

* centralize PSRAM usage (application code only)

* update logging

* update logging

* fix use after free

* initialize buffer

* free rgb_image before ussing it for new allocation

* use wrapper function

* switch log level to debug

* .

* undo adding free() calls

* .

* add names to all CImage instances

* .

* .

* .

* revert changes of stbi_image_free() with free_psram_heap() on the places where is is not in PSRAM

* .

* typos

* typo

* Added MQTT Outbox explanation/warning

* added CONFIG_SPIRAM_USE_MEMMAP explanation

---------

Co-authored-by: CaCO3 <[email protected]>

* Disable custom MQTT Outbox. This also moves the MQTT Publishing memory usage back to the internal RAM!

* log MQTT connection refused reasons (#2216)

* Revert PSRAM usage as it lead to memory fragmentation.
See #2200 for details

* fix missing value data

* Revert PSRAM usage as it lead to memory fragmentation. (#2224)

See #2200 for details

Co-authored-by: CaCO3 <[email protected]>

* Fix missing value data in graph (#2230)

* fix missing value data

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2231)

* Update interface_influxdb.cpp (#2233)

* update copyright year

* Cleanup

* Set prevalue using MQTT + set prevalue to RAW value (REST+MQTT) (#2252)

* Use double instead of float

* Error handling + set to RAW if newvalue < 0

* REST SetPrevalue: Set to RAW if newvalue < 0

* set prevalue with MQTT

* removed the stb_image files and re-add them as a submodule. (#2223)

- stb_image.h: Version update 2.25 -> 2.28
- stb_resize.h: Version update 0.96 -> 0.97
- stb_write.h: Version update 1.14 -> 1.16

Co-authored-by: CaCO3 <[email protected]>

* Remove obsolete ClassFlowWriteList (#2264)

* Renaming & cleanup of some modules / functions in source code (#2265)

* Rename module tag name

* Rename server_tflite.cpp -> MainFlowControl.cpp

* Remove redundandant MQTTMainTopic function

* Update

* Remove obsolete GetMQTTMainTopic

* Fix last element missing in digit model drop down (#2282)

* Debug influxdb (#2283)

* Fix time offset issues in InfluxDB component. (#2278)

Closes #2273
Closes #2150

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Improve Logging

* Implement TimeSync at beginning

* Update time_sntp.cpp

* Update time_sntp.cpp

* Set Time After WLAN Init

---------

Co-authored-by: Antonin Delpeuch <[email protected]>

* Implement a camera livestream handler (#2286)

* fix leading NaN (#2310)

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend InfluxDBv1 with individual topic names (#2319)

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Update main.cpp

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2344)

* Update Changelog.md

* Update Changelog.md

* merge conflicts

* merge conflicts

* update changelog

---------

Co-authored-by: Frank Haverland <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Antonin Delpeuch <[email protected]>
caco3 added a commit that referenced this issue Apr 27, 2023
* Testcase for #2145 and debug-log (#2151)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Fix timezone config parser (#2169)

* make sure to parse the whole config line

* fix crash on empty timezone parameter

---------

Co-authored-by: CaCO3 <[email protected]>

* Enhance ROI pages (#2161)

* Check if the ROIs are equidistant. Only if not, untick the checkbox

* renaming

* Check if the ROIs have same y, dy and dx. If so, tick the sync checkbox

* only allow editing space when box is checked

* fix sync check

* show inner frame on all ROIs

* cleanup

* Check if the ROIs have same dy and dx. If so, tick the sync checkbox

* checkbox position

* renaming

* renaming

* show inner frame and cross hairs on all ROIs

* update ROIs on ticking checkboxes

* show timezone hint

* fix deleting last ROI

* cleanup

---------

Co-authored-by: CaCO3 <[email protected]>

* restart timeout on progress, catch error (#2170)

* restart timeout on progress, catch error

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* BugFix #2167

* Release 15.1 preparations (#2171)

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update changelog

* Fix links to PR

* Formating

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>

* fix typo

* Replace relative documentation links with absolute ones pointing to the external documentation (#2180)

Co-authored-by: CaCO3 <[email protected]>

* Sort model files in configuration combobox (#2189)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Sort model files in combobox

* reboot task - increase stack size (#2201)

Avoid stack overflow

* Update interface_influxdb.cpp

* Update Changelog.md

* Show PSRAM usage (#2206)

* centralize PSRAM usage (application code only)

* update logging

* update logging

* fix use after free

* initialize buffer

* free rgb_image before ussing it for new allocation

* use wrapper function

* switch log level to debug

* .

* undo adding free() calls

* .

* add names to all CImage instances

* .

* .

* .

* revert changes of stbi_image_free() with free_psram_heap() on the places where is is not in PSRAM

* .

* typos

* typo

* Added MQTT Outbox explanation/warning

* added CONFIG_SPIRAM_USE_MEMMAP explanation

---------

Co-authored-by: CaCO3 <[email protected]>

* Disable custom MQTT Outbox. This also moves the MQTT Publishing memory usage back to the internal RAM!

* log MQTT connection refused reasons (#2216)

* Revert PSRAM usage as it lead to memory fragmentation.
See #2200 for details

* fix missing value data

* Revert PSRAM usage as it lead to memory fragmentation. (#2224)

See #2200 for details

Co-authored-by: CaCO3 <[email protected]>

* Fix missing value data in graph (#2230)

* fix missing value data

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2231)

* Update interface_influxdb.cpp (#2233)

* update copyright year

* Cleanup

* Set prevalue using MQTT + set prevalue to RAW value (REST+MQTT) (#2252)

* Use double instead of float

* Error handling + set to RAW if newvalue < 0

* REST SetPrevalue: Set to RAW if newvalue < 0

* set prevalue with MQTT

* removed the stb_image files and re-add them as a submodule. (#2223)

- stb_image.h: Version update 2.25 -> 2.28
- stb_resize.h: Version update 0.96 -> 0.97
- stb_write.h: Version update 1.14 -> 1.16

Co-authored-by: CaCO3 <[email protected]>

* Remove obsolete ClassFlowWriteList (#2264)

* Renaming & cleanup of some modules / functions in source code (#2265)

* Rename module tag name

* Rename server_tflite.cpp -> MainFlowControl.cpp

* Remove redundandant MQTTMainTopic function

* Update

* Remove obsolete GetMQTTMainTopic

* Fix last element missing in digit model drop down (#2282)

* Debug influxdb (#2283)

* Fix time offset issues in InfluxDB component. (#2278)

Closes #2273
Closes #2150

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Improve Logging

* Implement TimeSync at beginning

* Update time_sntp.cpp

* Update time_sntp.cpp

* Set Time After WLAN Init

---------

Co-authored-by: Antonin Delpeuch <[email protected]>

* Implement a camera livestream handler (#2286)

* fix leading NaN (#2310)

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend InfluxDBv1 with individual topic names (#2319)

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Update main.cpp

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2344)

* Update Changelog.md

* Update Changelog.md

* merge conflicts

* merge conflicts

* update changelog

* Update Changelog.md

* Fix alignment mark PSRAM issue (#2357)

* Add lock for shared PSRAM memory, use it for the relevant round steps and the Alignment Mark Task

* only allow taking a new image for the Alignment Mark while round got completed

* show success/denial of Alignment Mark Update Request

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2360)

---------

Co-authored-by: Frank Haverland <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Antonin Delpeuch <[email protected]>
caco3 added a commit that referenced this issue May 1, 2023
* Testcase for #2145 and debug-log (#2151)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Fix timezone config parser (#2169)

* make sure to parse the whole config line

* fix crash on empty timezone parameter

---------

Co-authored-by: CaCO3 <[email protected]>

* Enhance ROI pages (#2161)

* Check if the ROIs are equidistant. Only if not, untick the checkbox

* renaming

* Check if the ROIs have same y, dy and dx. If so, tick the sync checkbox

* only allow editing space when box is checked

* fix sync check

* show inner frame on all ROIs

* cleanup

* Check if the ROIs have same dy and dx. If so, tick the sync checkbox

* checkbox position

* renaming

* renaming

* show inner frame and cross hairs on all ROIs

* update ROIs on ticking checkboxes

* show timezone hint

* fix deleting last ROI

* cleanup

---------

Co-authored-by: CaCO3 <[email protected]>

* restart timeout on progress, catch error (#2170)

* restart timeout on progress, catch error

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* BugFix #2167

* Release 15.1 preparations (#2171)

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update changelog

* Fix links to PR

* Formating

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>

* fix typo

* Replace relative documentation links with absolute ones pointing to the external documentation (#2180)

Co-authored-by: CaCO3 <[email protected]>

* Sort model files in configuration combobox (#2189)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Sort model files in combobox

* reboot task - increase stack size (#2201)

Avoid stack overflow

* Update interface_influxdb.cpp

* Update Changelog.md

* Show PSRAM usage (#2206)

* centralize PSRAM usage (application code only)

* update logging

* update logging

* fix use after free

* initialize buffer

* free rgb_image before ussing it for new allocation

* use wrapper function

* switch log level to debug

* .

* undo adding free() calls

* .

* add names to all CImage instances

* .

* .

* .

* revert changes of stbi_image_free() with free_psram_heap() on the places where is is not in PSRAM

* .

* typos

* typo

* Added MQTT Outbox explanation/warning

* added CONFIG_SPIRAM_USE_MEMMAP explanation

---------

Co-authored-by: CaCO3 <[email protected]>

* Disable custom MQTT Outbox. This also moves the MQTT Publishing memory usage back to the internal RAM!

* log MQTT connection refused reasons (#2216)

* Revert PSRAM usage as it lead to memory fragmentation.
See #2200 for details

* fix missing value data

* Revert PSRAM usage as it lead to memory fragmentation. (#2224)

See #2200 for details

Co-authored-by: CaCO3 <[email protected]>

* Fix missing value data in graph (#2230)

* fix missing value data

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2231)

* Update interface_influxdb.cpp (#2233)

* update copyright year

* Cleanup

* Set prevalue using MQTT + set prevalue to RAW value (REST+MQTT) (#2252)

* Use double instead of float

* Error handling + set to RAW if newvalue < 0

* REST SetPrevalue: Set to RAW if newvalue < 0

* set prevalue with MQTT

* removed the stb_image files and re-add them as a submodule. (#2223)

- stb_image.h: Version update 2.25 -> 2.28
- stb_resize.h: Version update 0.96 -> 0.97
- stb_write.h: Version update 1.14 -> 1.16

Co-authored-by: CaCO3 <[email protected]>

* Remove obsolete ClassFlowWriteList (#2264)

* Renaming & cleanup of some modules / functions in source code (#2265)

* Rename module tag name

* Rename server_tflite.cpp -> MainFlowControl.cpp

* Remove redundandant MQTTMainTopic function

* Update

* Remove obsolete GetMQTTMainTopic

* Fix last element missing in digit model drop down (#2282)

* Debug influxdb (#2283)

* Fix time offset issues in InfluxDB component. (#2278)

Closes #2273
Closes #2150

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Improve Logging

* Implement TimeSync at beginning

* Update time_sntp.cpp

* Update time_sntp.cpp

* Set Time After WLAN Init

---------

Co-authored-by: Antonin Delpeuch <[email protected]>

* Implement a camera livestream handler (#2286)

* fix leading NaN (#2310)

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend InfluxDBv1 with individual topic names (#2319)

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Update main.cpp

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2344)

* Update Changelog.md

* Update Changelog.md

* merge conflicts

* merge conflicts

* update changelog

* Update Changelog.md

* Fix alignment mark PSRAM issue (#2357)

* Add lock for shared PSRAM memory, use it for the relevant round steps and the Alignment Mark Task

* only allow taking a new image for the Alignment Mark while round got completed

* show success/denial of Alignment Mark Update Request

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2360)

* Update platformio.ini (#2368)

* Allow the Alignment Mark step while status is "Initializing" or "Initialization (delayed)" or while in setup mode (#2373)

Co-authored-by: CaCO3 <[email protected]>

* Improve logging if Autostart is not enabled. Renamed/splitted functions (#2376)

Co-authored-by: CaCO3 <[email protected]>

* Cleanup web UI (#2378)

* fix wrong index on backup log

* remove no longer used files/functions

* improve logging on Alignment Mark Update

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md

---------

Co-authored-by: Frank Haverland <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Antonin Delpeuch <[email protected]>
caco3 added a commit that referenced this issue May 1, 2023
* Testcase for #2145 and debug-log (#2151)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Fix timezone config parser (#2169)

* make sure to parse the whole config line

* fix crash on empty timezone parameter

---------

Co-authored-by: CaCO3 <[email protected]>

* Enhance ROI pages (#2161)

* Check if the ROIs are equidistant. Only if not, untick the checkbox

* renaming

* Check if the ROIs have same y, dy and dx. If so, tick the sync checkbox

* only allow editing space when box is checked

* fix sync check

* show inner frame on all ROIs

* cleanup

* Check if the ROIs have same dy and dx. If so, tick the sync checkbox

* checkbox position

* renaming

* renaming

* show inner frame and cross hairs on all ROIs

* update ROIs on ticking checkboxes

* show timezone hint

* fix deleting last ROI

* cleanup

---------

Co-authored-by: CaCO3 <[email protected]>

* restart timeout on progress, catch error (#2170)

* restart timeout on progress, catch error

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* BugFix #2167

* Release 15.1 preparations (#2171)

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update changelog

* Fix links to PR

* Formating

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>

* fix typo

* Replace relative documentation links with absolute ones pointing to the external documentation (#2180)

Co-authored-by: CaCO3 <[email protected]>

* Sort model files in configuration combobox (#2189)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Sort model files in combobox

* reboot task - increase stack size (#2201)

Avoid stack overflow

* Update interface_influxdb.cpp

* Update Changelog.md

* Show PSRAM usage (#2206)

* centralize PSRAM usage (application code only)

* update logging

* update logging

* fix use after free

* initialize buffer

* free rgb_image before ussing it for new allocation

* use wrapper function

* switch log level to debug

* .

* undo adding free() calls

* .

* add names to all CImage instances

* .

* .

* .

* revert changes of stbi_image_free() with free_psram_heap() on the places where is is not in PSRAM

* .

* typos

* typo

* Added MQTT Outbox explanation/warning

* added CONFIG_SPIRAM_USE_MEMMAP explanation

---------

Co-authored-by: CaCO3 <[email protected]>

* Disable custom MQTT Outbox. This also moves the MQTT Publishing memory usage back to the internal RAM!

* log MQTT connection refused reasons (#2216)

* Revert PSRAM usage as it lead to memory fragmentation.
See #2200 for details

* fix missing value data

* Revert PSRAM usage as it lead to memory fragmentation. (#2224)

See #2200 for details

Co-authored-by: CaCO3 <[email protected]>

* Fix missing value data in graph (#2230)

* fix missing value data

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2231)

* Update interface_influxdb.cpp (#2233)

* update copyright year

* Cleanup

* Set prevalue using MQTT + set prevalue to RAW value (REST+MQTT) (#2252)

* Use double instead of float

* Error handling + set to RAW if newvalue < 0

* REST SetPrevalue: Set to RAW if newvalue < 0

* set prevalue with MQTT

* removed the stb_image files and re-add them as a submodule. (#2223)

- stb_image.h: Version update 2.25 -> 2.28
- stb_resize.h: Version update 0.96 -> 0.97
- stb_write.h: Version update 1.14 -> 1.16

Co-authored-by: CaCO3 <[email protected]>

* Remove obsolete ClassFlowWriteList (#2264)

* Renaming & cleanup of some modules / functions in source code (#2265)

* Rename module tag name

* Rename server_tflite.cpp -> MainFlowControl.cpp

* Remove redundandant MQTTMainTopic function

* Update

* Remove obsolete GetMQTTMainTopic

* Fix last element missing in digit model drop down (#2282)

* Debug influxdb (#2283)

* Fix time offset issues in InfluxDB component. (#2278)

Closes #2273
Closes #2150

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Improve Logging

* Implement TimeSync at beginning

* Update time_sntp.cpp

* Update time_sntp.cpp

* Set Time After WLAN Init

---------

Co-authored-by: Antonin Delpeuch <[email protected]>

* Implement a camera livestream handler (#2286)

* fix leading NaN (#2310)

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend InfluxDBv1 with individual topic names (#2319)

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Update main.cpp

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2344)

* Update Changelog.md

* Update Changelog.md

* merge conflicts

* merge conflicts

* update changelog

* Update Changelog.md

* Fix alignment mark PSRAM issue (#2357)

* Add lock for shared PSRAM memory, use it for the relevant round steps and the Alignment Mark Task

* only allow taking a new image for the Alignment Mark while round got completed

* show success/denial of Alignment Mark Update Request

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2360)

* Update platformio.ini (#2368)

* Allow the Alignment Mark step while status is "Initializing" or "Initialization (delayed)" or while in setup mode (#2373)

Co-authored-by: CaCO3 <[email protected]>

* Improve logging if Autostart is not enabled. Renamed/splitted functions (#2376)

Co-authored-by: CaCO3 <[email protected]>

* Cleanup web UI (#2378)

* fix wrong index on backup log

* remove no longer used files/functions

* improve logging on Alignment Mark Update

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md

* fix broken sysinfo (#2381)

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md

* Update Changelog.md

---------

Co-authored-by: Frank Haverland <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Antonin Delpeuch <[email protected]>
caco3 added a commit that referenced this issue May 2, 2023
* Testcase for #2145 and debug-log (#2151)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Fix timezone config parser (#2169)

* make sure to parse the whole config line

* fix crash on empty timezone parameter

---------

Co-authored-by: CaCO3 <[email protected]>

* Enhance ROI pages (#2161)

* Check if the ROIs are equidistant. Only if not, untick the checkbox

* renaming

* Check if the ROIs have same y, dy and dx. If so, tick the sync checkbox

* only allow editing space when box is checked

* fix sync check

* show inner frame on all ROIs

* cleanup

* Check if the ROIs have same dy and dx. If so, tick the sync checkbox

* checkbox position

* renaming

* renaming

* show inner frame and cross hairs on all ROIs

* update ROIs on ticking checkboxes

* show timezone hint

* fix deleting last ROI

* cleanup

---------

Co-authored-by: CaCO3 <[email protected]>

* restart timeout on progress, catch error (#2170)

* restart timeout on progress, catch error

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* BugFix #2167

* Release 15.1 preparations (#2171)

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update changelog

* Fix links to PR

* Formating

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

* Update Changelog.md

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>

* fix typo

* Replace relative documentation links with absolute ones pointing to the external documentation (#2180)

Co-authored-by: CaCO3 <[email protected]>

* Sort model files in configuration combobox (#2189)

* new models ana-cont-11.0.5, ana-class100-1.5.7, dig-class100-1.6.0

* Testcase for #2145
Added debug log, if allowNegativeRates is handeled

* Sort model files in combobox

* reboot task - increase stack size (#2201)

Avoid stack overflow

* Update interface_influxdb.cpp

* Update Changelog.md

* Show PSRAM usage (#2206)

* centralize PSRAM usage (application code only)

* update logging

* update logging

* fix use after free

* initialize buffer

* free rgb_image before ussing it for new allocation

* use wrapper function

* switch log level to debug

* .

* undo adding free() calls

* .

* add names to all CImage instances

* .

* .

* .

* revert changes of stbi_image_free() with free_psram_heap() on the places where is is not in PSRAM

* .

* typos

* typo

* Added MQTT Outbox explanation/warning

* added CONFIG_SPIRAM_USE_MEMMAP explanation

---------

Co-authored-by: CaCO3 <[email protected]>

* Disable custom MQTT Outbox. This also moves the MQTT Publishing memory usage back to the internal RAM!

* log MQTT connection refused reasons (#2216)

* Revert PSRAM usage as it lead to memory fragmentation.
See #2200 for details

* fix missing value data

* Revert PSRAM usage as it lead to memory fragmentation. (#2224)

See #2200 for details

Co-authored-by: CaCO3 <[email protected]>

* Fix missing value data in graph (#2230)

* fix missing value data

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2231)

* Update interface_influxdb.cpp (#2233)

* update copyright year

* Cleanup

* Set prevalue using MQTT + set prevalue to RAW value (REST+MQTT) (#2252)

* Use double instead of float

* Error handling + set to RAW if newvalue < 0

* REST SetPrevalue: Set to RAW if newvalue < 0

* set prevalue with MQTT

* removed the stb_image files and re-add them as a submodule. (#2223)

- stb_image.h: Version update 2.25 -> 2.28
- stb_resize.h: Version update 0.96 -> 0.97
- stb_write.h: Version update 1.14 -> 1.16

Co-authored-by: CaCO3 <[email protected]>

* Remove obsolete ClassFlowWriteList (#2264)

* Renaming & cleanup of some modules / functions in source code (#2265)

* Rename module tag name

* Rename server_tflite.cpp -> MainFlowControl.cpp

* Remove redundandant MQTTMainTopic function

* Update

* Remove obsolete GetMQTTMainTopic

* Fix last element missing in digit model drop down (#2282)

* Debug influxdb (#2283)

* Fix time offset issues in InfluxDB component. (#2278)

Closes #2273
Closes #2150

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Improve Logging

* Implement TimeSync at beginning

* Update time_sntp.cpp

* Update time_sntp.cpp

* Set Time After WLAN Init

---------

Co-authored-by: Antonin Delpeuch <[email protected]>

* Implement a camera livestream handler (#2286)

* fix leading NaN (#2310)

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend InfluxDBv1 with individual topic names (#2319)

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* analogROI: Activate save button after ROI creation (#2326)

* Migration of PlatformIO 5.2.0 to 6.1.0 (resp. ESP IDF from 4.4.2 to 5.0.1) (#2305)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* fix leading NaN (#2310)

* Migration to PlatformIO 6.1.0

* Disable RMTMEM usage as it is no longer allowed -> Smart LEDs not functional!

* moved miniz into subfolder of jomjol_fileserver_ota, else it does not build anymore.

* cleanup

* Task watchdog has new config name

* Fix return value check. It must be something else than ESP_FAIL, but it does not need to be ESP_OK!

* add missing strucures to work around new RMTMEM restriction (untested)

---------

Co-authored-by: CaCO3 <[email protected]>

* Keep MainFlowTask alive to handle reboot (#2325)

* Shared PSRAM memory (#2285)

* enable PSRAM logging

* add extra functions for psram shared memroy handling

* CImageBasis objects still should used dynamic memory (eg. rawImage), haw ever tmpImage must be placed inside the shared memory

* Place all STBI allocs inside the shared memory

* The models are placed in the shared PSRAM reagion and must be allocated through the dedicated functions

* .

* renaming

* fix cast warning

* add flag to switch STBI PSRAM usage

* improve PSRAM shared handling

* reserve shared PSRAM as early as possible

* init logging eralier so we can use it in PSRAM shared alloc

* move Wifi_LWIP, BSS_SEG and MQTT Outbox into PSRAM to ffree internal memory

* Check if model fits into reserved shared memory

* Update code/components/jomjol_tfliteclass/CTfLiteClass.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_flowcontroll/ClassFlowControll.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_image_proc/CImageBasis.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* Update code/components/jomjol_helper/psram.cpp

* .

* .

* .

* .

* Korrektur Merge Conflict in main.cpp

---------

Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>

* fix PSRAM init return value check

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Implement individual influx topic

* Update interface_influxdb.cpp

* Update interface_influxdb.cpp

* Update FieldName

* Extend incl. indiv. Measurement

* Implement UX

* Update ClassFlowInfluxDBv2.cpp

* Update main.cpp

---------

Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2344)

* Update Changelog.md

* Update Changelog.md

* merge conflicts

* merge conflicts

* update changelog

* Update Changelog.md

* Fix alignment mark PSRAM issue (#2357)

* Add lock for shared PSRAM memory, use it for the relevant round steps and the Alignment Mark Task

* only allow taking a new image for the Alignment Mark while round got completed

* show success/denial of Alignment Mark Update Request

* .

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md (#2360)

* Update platformio.ini (#2368)

* Allow the Alignment Mark step while status is "Initializing" or "Initialization (delayed)" or while in setup mode (#2373)

Co-authored-by: CaCO3 <[email protected]>

* Improve logging if Autostart is not enabled. Renamed/splitted functions (#2376)

Co-authored-by: CaCO3 <[email protected]>

* Cleanup web UI (#2378)

* fix wrong index on backup log

* remove no longer used files/functions

* improve logging on Alignment Mark Update

---------

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md

* fix broken sysinfo (#2381)

Co-authored-by: CaCO3 <[email protected]>

* Update Changelog.md

* Update Changelog.md

* Update sdcard_check.cpp (#2384)

* Update Changelog.md

---------

Co-authored-by: Frank Haverland <[email protected]>
Co-authored-by: CaCO3 <[email protected]>
Co-authored-by: jomjol <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Slider0007 <[email protected]>
Co-authored-by: Antonin Delpeuch <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants