[TW#17012] Wifi Interface Stop

[Pratikhyadav]

Hello,

I created HTTP server using lwip netcon library.

Also i applied changes as suggested by :
#784.

I have following scenario:

1. When i Create simple http server it works fine.(Tested it with more than 140000 requested).

2. But when used spiffs to store my data in file transition stop at random time and when i try to connect
   to http server , i can’t. I have to reboot Board.

3)To Check that is there any issue with spiffs i run standalone test in which i did same functionality of my
http server, without any socket(netcon) API. Which is works fine.

But when I merge Spiffs with simple http My server stop responding after some request(around 1500), And can’t reconnect with it.

My spiffs Partition is as following:

   Label            Usage          Type ST Offset   Length
0 nvs              WiFi data        01 02 00009000 00006000
1 phy_init         RF data          01 01 0000f000 00001000
2 factory          factory app      00 00 00010000 00100000
3 storage          Unknown data     01 82 00110000 00200000

My Wifi configuration is as following:

static esp_err_t event_handler(void *ctx, system_event_t *event)
{
    switch(event->event_id) {
    case SYSTEM_EVENT_STA_START:
        esp_wifi_connect();
        break;
    case SYSTEM_EVENT_STA_GOT_IP:
        xEventGroupSetBits(wifi_event_group, CONNECTED_BIT);
        printf("got ip\n");
        printf("ip: " IPSTR "\n", IP2STR(&event->event_info.got_ip.ip_info.ip));
        printf("netmask: " IPSTR "\n", IP2STR(&event->event_info.got_ip.ip_info.netmask));
        printf("gw: " IPSTR "\n", IP2STR(&event->event_info.got_ip.ip_info.gw));
        printf("\n");
        fflush(stdout);
        break;
    case SYSTEM_EVENT_STA_DISCONNECTED:
        /* This is a workaround as ESP32 WiFi libs don't currently
           auto-reassociate. */
           doReconnect=true;
        esp_wifi_connect();
        xEventGroupClearBits(wifi_event_group, CONNECTED_BIT);
        break;
    default:
        break;
    }
    return ESP_OK;
}

static void initialise_wifi(void)
{
    tcpip_adapter_init();
    wifi_event_group = xEventGroupCreate();
    ESP_ERROR_CHECK( esp_event_loop_init(event_handler, NULL) );
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK( esp_wifi_init(&cfg) );
    ESP_ERROR_CHECK( esp_wifi_set_storage(WIFI_STORAGE_RAM) );
    ESP_ERROR_CHECK( esp_wifi_set_mode(WIFI_MODE_AP) );
    wifi_config_t ap_config = {
        .ap = {
            .ssid = "HTTP_SERVER",
            .password = "9876543210",
            .authmode=WIFI_AUTH_WPA_WPA2_PSK,
            .max_connection=4,         
        }
    };

    ESP_ERROR_CHECK( esp_wifi_set_config(ESP_IF_WIFI_AP, &ap_config) );
    ESP_ERROR_CHECK( esp_wifi_start() );
}



My server Code is As following:

static void http_server(void *pvParameters)
{   
  ESP_LOGI(TAG,"system_get_free_heap_size in starting of server task =%d\n\n\n",system_get_free_heap_size());
  struct netconn *conn , *newconn;
 
  err_t err,err_listen,err_accept;
  conn = netconn_new(NETCONN_TCP);
  uint32_t total=0;
  uint32_t used=0;  
  if(conn!=NULL)
    { 
     err= netconn_bind(conn, NULL, 80);
      if (err == ERR_OK)
      ESP_LOGI(TAG,"socket bound");
      
     err_listen=netconn_listen(conn);
     
     if(err_listen==ERR_OK)
     {      
        do{        
            ESP_LOGI(TAG,"system_get_free_heap_size before accept in request number %d =%d",++counter,system_get_free_heap_size());
            ESP_LOGI(TAG,"system_get_time before accept in request number %d =%d",counter,system_get_time());
            err_accept = netconn_accept(conn, &newconn);
            
            if (err_accept == ERR_OK)
            {
               ESP_LOGI(TAG,"connection accept\n");
               
               ESP_LOGI(TAG,"\nsystem_get_free_heap_size after accept  =%d\n\n\n",system_get_free_heap_size());            
                ESP_LOGI(TAG,"\nsystem_get_free_heap_size befor close  newconn =%d\n\n\n",system_get_free_heap_size());
            
                newconn->recv_timeout=2000;

xTaskCreate(http_server_netconn_serve,"http_server_netconn_serve",5000,newconn,10,http_server_netconn_serve_Task);
            }         
         } while(err_accept == ERR_OK);

         netconn_close(conn);
         ESP_LOGI(TAG,"conn is close");
         netconn_delete(conn); 
     } /*err_listen end */
    
    }/*conn end */
   
   vTaskDelete(NULL);
     
   ESP_LOGI(TAG,"http exit");
}

void http_server_netconn_serve(void *pvParameters)
{  
     struct netbuf *inbuf;
     char *buf;
     
     u16_t buflen;
     err_t err;
      
      
     /* Read the data from the port, blocking if nothing yet there.
      We assume the request (the part we care about) is in one netbuf */
      ESP_LOGI(TAG,"waitting for data");
     err = netconn_recv(pvParameters, &inbuf);
     

     if (err == ERR_OK) 
     {
       
      err_t err1=netbuf_data(inbuf, (void**)&buf, &buflen);
      if (err1 != ERR_OK) 
      ESP_LOGI(TAG,"Data recived from client is not copied to buffer"); 
      
      ESP_LOGI(TAG,"\nbuflen=%d\n",buflen);
           
       ESP_LOGI(TAG,"system_get_free_heap_size befor paser function call=%d",system_get_free_heap_size());
       parser(buf,buflen,pvParameters);
      ESP_LOGI(TAG,"system_get_free_heap_size after paser function call=%d",system_get_free_heap_size());
           
           
      netbuf_delete(inbuf);
         
      err=netconn_close(pvParameters);
      if (err != ERR_OK) 
      ESP_LOGI(TAG,"pvParameters not closed succesfully in parser");
      
      err_t err_newconn_d=netconn_delete(pvParameters); 
      if (err_newconn_d != ERR_OK) 
      ESP_LOGI(TAG,"newconn not delete succesfully");
         
      ESP_LOGI(TAG,"system_get_free_heap_size after delete and befor free =%d",system_get_free_heap_size());      
           }
   vTaskDelete(http_server_netconn_serve_Task);
   task_delete=1;
    ESP_LOGI(TAG,"http_server_netconn_serve Task deleted");
  
}

Where my parser Function Check which kind of requests(POST, GET,DELETE,PUT) is requested from client, and send html formatted string in respond.

When POST is requested so file which is sent in POST request , I stored it in SPIFFS in http_parser's Body callback using fopen and fwrite.

After More than 1000 Requests Server stop responding , And has following log:

I (525892) wifi: station: 00:13:ef:d4:02:09 leave, AID = 1
I (525902) wifi: n:1 0, o:1 0, ap:1 1, sta:255 255, prof:1
I (529002) wifi: n:1 0, o:1 0, ap:1 1, sta:255 255, prof:1
I (529002) wifi: station: 00:13:ef:d4:02:09 join, AID=1, g, 20
I (580092) wifi: station: 00:13:ef:d4:02:09 leave, AID = 1
I (580092) wifi: n:1 0, o:1 0, ap:1 1, sta:255 255, prof:1
I (582152) wifi: n:1 0, o:1 0, ap:1 1, sta:255 255, prof:1
I (582152) wifi: station: 00:13:ef:d4:02:09 join, AID=1, g, 20
I (592152) wifi: station: 00:13:ef:d4:02:09 leave, AID = 1
I (592152) wifi: n:1 0, o:1 0, ap:1 1, sta:255 255, prof:1
I (595362) wifi: n:1 0, o:1 0, ap:1 1, sta:255 255, prof:1

Am I missing something?

[Pratikhyadav]

Is it WiFi Configuration or WiFi related issue or other?
Anyone can help me to suggest where to look specifically?

[Pratikhyadav]

I found that its not related to SPIFFS.

[jack0c]

@Pratikhyadav Can you describe relationship between this issue with SPIFFS. I am confused about this. If you can give difference between workable code and unworkable code, it will help a lot.

[Pratikhyadav]

Hello @jack0c

I Found That its not SPIFFS related Issue.

This Issue reproduced when i used Flash write Operation When i used Flash operation.
In my Working Code:
When I send Request In Post Method , My server Just read Hole File from Socket, But dont write it on Flash.Which is work fine in long run too.

Not Working Code:
When i Introduce Flash Write Operation to store socket's data to Flash My AP is disappear In overnight Test.

Only Change I applied is Added Flash Write Logic Which is Shown Below:
Following Code is Part of Parser Function

				while(1)
			{		
		 			

				  /* Read the data from the port, blocking if nothing yet there.
				   We assume the request (the part we care about) is in one netbuf */

				  err = netconn_recv(pvParameters, &inbuf1);

				  if (err == ERR_OK) 
				  {
					err=netbuf_data(inbuf1, (void**)&buf, &buflen);
					
					if (err != ERR_OK) 
					ESP_LOGE(TAG,"Data recived from client is not copied to buffer\n");
											
					http_parser_execute(&h_parser,&h_settings,buf,buflen);
					/*ADDED FLASH LOGIC*/
					 for(i=0;i<=buflen;i++)	
					{
						post_buffer_lenght++;
						
						Post_file[post_buffer_lenght]=buf[i];
						
					
						if(post_buffer_lenght==4095)
						{
							
							spi_flash_read(base_addr, checksum,SPI_FLASH_SEC_SIZE);								
							base_addr+=4096;								
							memset(Post_file,0,sizeof(Post_file));
							post_buffer_lenght=0;							
						}
						
						
					}
					
					/*END OF FLASH LOGIC*/
											
					netbuf_delete(inbuf1);					   
						
				  } 
				  
				  else
				  {
					  /*ADDED FLASH LOGIC*/
					  if(post_buffer_lenght>0)
					 {
						spi_flash_read(base_addr, checksum,post_buffer_lenght);						
						base_addr=0x300000;								
						memset(Post_file,0,sizeof(Post_file));
						post_buffer_lenght=0;					
						 
					 }
					 
					 else
					 {
						base_addr=0x300000;									
						 
					 }
					
					/*END OF FLASH LOGIC*/
					break;
				  }
		 
			}	

My partition Table is as following:
Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x4000,
otadata, data, ota, 0xd000, 0x2000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 0x100000,
ota_0, app, 0x10, 0x110000, 0x100000,
storage, data, 0x82, 0x210000, 0xC0000,

Let me know if you requires more detail.

[jack0c]

@Pratikhyadav
1.You said write operation, but you use spi_flash_read, why?
2.Didn't see the base_addr and post_buffer_lenght.
3.Should you use Post_file[post_buffer_lenght++]=buf[i];?
However, I didn't see the problem.
An advice, you can run ota example, which can be run on STA+AP mode.

[Pratikhyadav]

Hello @jack0c
Sorry I was mistaken ,instead of Read i used Flash Erase and Flash Write.

Yes I used Post_file[post_buffer_lenght++]=buf[i];

Code is As following.

int post_buffer_lenght=0; int base_addr=0x300000; char Pos_file[4096]={0};

while(1)
			{		
		 			

				  /* Read the data from the port, blocking if nothing yet there.
				   We assume the request (the part we care about) is in one netbuf */

				  err = netconn_recv(pvParameters, &inbuf1);

				  if (err == ERR_OK) 
				  {
					err=netbuf_data(inbuf1, (void**)&buf, &buflen);
					
					if (err != ERR_OK) 
					ESP_LOGE(TAG,"Data recived from client is not copied to buffer\n");
											
					http_parser_execute(&h_parser,&h_settings,buf,buflen);
					/*Flash Write Section*/
					 for(i=0;i<=buflen;i++)	
					{
						post_buffer_lenght++;
						
						Post_file[post_buffer_lenght]=buf[i];
						
					
						if(post_buffer_lenght==4095)
						{
							
							err=spi_flash_erase_sector(base_addr / SPI_FLASH_SEC_SIZE);
							if (err != ERR_OK) 
							ESP_LOGE(TAG,"flash section 0x%x is not cleared sucessfully\n",base_addr);
														
							err=spi_flash_write(base_addr, Post_file, SPI_FLASH_SEC_SIZE);
							if (err != ERR_OK) 
							ESP_LOGE(TAG,"flash section 0x%x is not written sucessfully\n",base_addr);							
							base_addr+=4096;								
							memset(Post_file,0,sizeof(Post_file));
							post_buffer_lenght=0;							
						}
						
						
					}

					/*Flash Write Section End*/

					
											
					netbuf_delete(inbuf1);					   
						
				  } 
				  
				  else
				  {
					  	/*Flash Write Section */
					  if(post_buffer_lenght>0)
					 {
						err=spi_flash_erase_sector(base_addr / SPI_FLASH_SEC_SIZE);
						if (err != ERR_OK) 
						ESP_LOGE(TAG,"flash section 0x%x is not cleared sucessfully\n",base_addr);
						err=spi_flash_write(base_addr, Post_file, SPI_FLASH_SEC_SIZE);
						if (err != ERR_OK) 
						ESP_LOGE(TAG,"flash section 0x%x is not written sucessfully\n",base_addr);							
						base_addr=0x300000;								
						memset(Post_file,0,sizeof(Post_file));
						post_buffer_lenght=0;					
						 
					 }
					 
					 else
					 {
						base_addr=0x300000;									
						 
					 }
					
					/*Flash Write Section End*/
					break;
				  }
		 
			}		

Let me know If you need More detail.

[Pratikhyadav]

Hi @jack0c
Is there anything i missed?
Please let me know if you have any suggestion.

[jack0c]

Again，
1.Should not put Post_file[post_buffer_lenght]=buf[i]; after post_buffer_lenght ++, here you will miss a byte.
2.My suggestion, you can run ota example, just change the mode to STA=AP.

[Pratikhyadav]

Hi @jack0c

yes, your first suggestion is to the point. and i rectified my mistake as well.

my set up is like this -
i am running a server in an ap mode. and trying to server post request form http client.
as soon as i get any post request, i starts reading data(file contents) chunk by chunk.
so, in my case, i am performing two operations :

1. reading the socket data(chunk)
2. writing this read chunk in my flash.

at the beginning these reading and writing to flash operation is happening properly. but, after several(about hundreds of) requests MY SERVER's AP IS GETTING DISAPPEARED & it is no more discoverable by any device running STA mode.

NOTE: this issue is seen only when i do flash write and flash erase operations after socket read.
even writing 1KB sized data into flash in this case i am able to reproduce this issue.

After AP disappear I got following messages after around 5 to 10 minutes:

W (4158142) wifi: pp task q full: s=11 p=3ffca928

W (4160142) wifi: pp task q full: s=11 p=3ffca938

W (4162142) wifi: pp task q full: s=11 p=3ffca948

W (4164142) wifi: pp task q full: s=11 p=3ffca958

W (4166142) wifi: pp task q full: s=11 p=3ffca968

W (4168142) wifi: pp task q full: s=11 p=3ffca978

W (4170142) wifi: pp task q full: s=11 p=3ffca988

W (4172142) wifi: pp task q full: s=11 p=3ffca998

W (4174142) wifi: pp task q full: s=11 p=3ffca9a8

W (4176142) wifi: pp task q full: s=11 p=3ffca9b8

W (4178142) wifi: pp task q full: s=11 p=3ffca9c8

W (4180142) wifi: pp task q full: s=11 p=3ffca9d8

W (4182142) wifi: pp task q full: s=11 p=3ffca9e8

W (4184142) wifi: pp task q full: s=11 p=3ffca9f8

W (4186142) wifi: pp task q full: s=11 p=3ffcaa08

W (4188142) wifi: pp task q full: s=11 p=3ffcaa18

W (4190142) wifi: pp task q full: s=11 p=3ffcaa28

I would like to know the reason why you are suggesting me to go through OTA example.

As I have tested Standalone task in which i have written 900 kB data into flash from 0x300000 location with http server which works fine.
In this test Case i was writing dummy data into flash in this standalone task while socket data reading operation was on going in http server task.

And what if i just want to store my file or data(rather than taking OTA, validating and then storing) into flash on client's http request.In this case i cant use OTA API.

[jack0c]

@Pratikhyadav What's your idf version?

[Pratikhyadav]

I am using idf 2.1 and its detail is as following

commit:27574a31e7c41f6e22e10e9b9d7713c6d49f795d
Merge: efdbc63 8134140
Author: Ivan Grokhotkov [email protected]
Date: Mon Jul 24 19:42:36 2017 +0800

[jack0c]

Can you please use idf 2.1.1, commit : 8bca703.
Your version is too old.

[Pratikhyadav]

I used 2.1.1 But still facing same issue.

But if i used 3.0rc1 its work for 24 hour and still working.
I will continue This Test and let you know if issue is reproduce.
I want to know that is it safe to use 3.0rc1 for development purpose?

Is it IDF related issue? is there any bug in WiFi stack or Flash APIs of older idf ?

[jack0c]

@Pratikhyadav Thank you for testing.
It's OK for you to use idf 3.0 rc1 for development purpose, idf3.0 release is coming soon.
It may be IDF related issue, but I can't tell which MR can resolve this problem.
Can you provide whole your test code or the binary.

[Pratikhyadav]

binfiles_send on git.zip

This Is bin file in which issue is reproduced
This bins was built on idf-2.1

Let me know if you want more detail about it.

[jack0c]

@zhangyanjiaoesp Please use this bin file to test.

[zhangyanjiaoesp]

@Pratikhyadav I have run your bin file, but I can't find the AP SSID and password. So how the http client connect to it ?

[Pratikhyadav]

After Flashing Provided REST with AP Interface Firmware into ESP32 Board, Execute below provided steps to start REST Interface testing with different GET/POST/PUT/DELETE requests

1. Connect to AP named “ESP_XXXXXXXX” with password: 0123456789 . (Where XXXXXXXXXXXX= MAC Address of ESP32)

2. Wait for Station to Connect with AP.

3)Send HTTP Rest POST request by any Rest client( I used POSTMAN) to Server with 900 Kbytes as attached image using below mentioned URL

http://192.168.4.1/

If complete file is received by server than it will send HTML page in response with message “POST was Requested”.

If File is NOT received completely than You will see Error message on console of ESP board :
“!!!!!!!!!!!!!complete file is not stored!!!!!!!!!!!!!!!!!!”

NOTE:Issue was reproduced in long run.

[zhangyanjiaoesp]

I use both ESP_30aea4047411 and ESP_30:ae:a4:04:74:11 , the client still can't connect to AP .

The print of AP like this:

[Pratikhyadav]

Use Uppercase instead of Lowercase .
In your case, Your AP's SSID will be
SSID: ESP_30AEA4047411
Password: 0123456789

Let me know if you need any other information about it.

[zhangyanjiaoesp]

Can you provide the test code for http_client?

[Pratikhyadav]

curl+simpleget.sh.zip
I used POST-MAN. application.

U can download it from following link
https://www.getpostman.com/apps

you can run following script for continues Rest request.
NOTE: This script is sending rest request using curl

Replace File location on POST requesting attached script.
curl -X POST -F 'file=@/file/location/' http://192.168.4.1:80/;

[zhangyanjiaoesp]

You say

NOTE:Issue was reproduced in long run.

,so I want to know how long it will be? several hours? or one day?

[Pratikhyadav]

Issue reproduce randomly every time. Some times its reproduce in 1 to 2 hour and some times its take more than 12 hours.

So better to run this test for One Day.

[Pratikhyadav]

Hello @zhangyanjiaoesp

Have you tried to test and to reproduce this issue on your end ?

If you have any doubt on "HOW TO TEST" let me know.

[zhangyanjiaoesp]

@Pratikhyadav I have reproduce your test, and I'm finding the reason of the issue

[zhangyanjiaoesp]

@Pratikhyadav Can you provide whole your test code? I test with the OTA example, in AP+STA mode, so the behavior is same to your test, and the idf version is v2.1. I have tested almost a whole day, but I can't reproduce the problem. So I need your test code.

[Pratikhyadav]

Thank you @zhangyanjiaoesp for putting efforts to reproduce issue on your end.
Let me know If you find anything regarding this issue.
Thanks again.

[FayeY]

Hi @Pratikhyadav ,
The ESP32 WiFi has been optimized a lot since the IDF 2.1, so please use the IDF3.0 rc1 for your development, we believe it has solved your problem.
We have done a lot of internal tests about this issue, but cannot reproduce it in the latest IDF3.0 RC1. So we are going to close this issue.
Please feel free to reopen it if the issue persists on the latest IDF3.0 RC1 and you can supply more details, such as your test code, for debugging.
Thanks.

[Pratikhyadav]

Hello,

If i provide you my Test code can you tell me exact root cause of this problem?

Thanks

[zhangyanjiaoesp]

If you provide your test code, it's better for us to find the root cause of this problem

[Pratikhyadav]

Hello

Have you debugged it ? Is there any issue in my code?

Let me know if you find anything.

[Pratikhyadav]

hello @zhangyanjiaoesp @FayeY
Have you found anything regarding this issue?

As my another apps are using esp-idf 2.1 i need to know which part cause this problem. SO request you to find exact root cause of this issue.and Let me know ASAP.

Thanks

[zhangyanjiaoesp]

@Pratikhyadav
sorry to reply later, we are testing to find the exact root cause with your test code. Once we have a result, we will inform you immediately.

[Pratikhyadav]

Thanks @zhangyanjiaoesp

waiting for your reply.

Many Thanks..

[Pratikhyadav]

Hello @zhangyanjiaoesp

Any update?

[FayeY]

Hi Pratik, sorry that these days are the Chinese Spring Festival Days, so all our engineers are on holiday. They will back to work in the next couple days. Sorry for the inconvenience.

[Pratikhyadav]

Hi @FayeY

OK. Humble request to do it ASAP.

[liuzfesp]

@zhangyanjiaoesp please help to update it when you back to work from holiday.

[zhangyanjiaoesp]

@Pratikhyadav I use your code to test before the holiday. When it runs on dual core, the problem can be reproduced , but when it runs on single core, the problem can not be reproduced. I need to test more to find the root cause.

[zhangyanjiaoesp]

@Pratikhyadav
I have found some useful change for idf v2.1, after the code be merged, you can see it on github

[Pratikhyadav]

@zhangyanjiaoesp

sorry, Can you tell more specific? which change is affected to this issue?

[Pratikhyadav]

Hello,
I have seen TW#18945 issue but ,
In my case AP was disappear , where in TW#18945 AP hangs for some time.(Correct me if i misunderstood )
Is there any root cause found yet?

[zhangyanjiaoesp]

@Pratikhyadav Please pull the latest idf release/v2.1 to check