Poor SD read speed of fread() (IDFGH-903)

[keelung-yang]

I'm using the last master branch since fopen() failed in v3.3 while opening large file which more than 7MB.

Test results on Kingston 32GB 90MB/s TF(Micro SD) Class10 UHS-I:

esp_err_t sd_connect()
{
    // On esp-idf v4.0-dev-225-g2f8b6cfc7
    //      SPI SDMMC_FREQ_DEFAULT   :   585 KB/s
    //      SPI SDMMC_FREQ_HIGHSPEED :   594 KB/s
    //      MMC(1-line) SDMMC_FREQ_DEFAULT   :   673 KB/s
    //      MMC(1-line) SDMMC_FREQ_HIGHSPEED :   775 KB/s
#if 0
    ESP_LOGI(LOG_TAG, "Using SPI peripheral");
    sdmmc_host_t host = SDSPI_HOST_DEFAULT();
    sdspi_slot_config_t slot_config = SDSPI_SLOT_CONFIG_DEFAULT();
#else
    ESP_LOGI(LOG_TAG, "Using SDMMC peripheral");
    sdmmc_host_t host = SDMMC_HOST_DEFAULT();
    sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();
    slot_config.width = 1;
#endif
    // host.max_freq_khz = SDMMC_FREQ_HIGHSPEED;
    // ...
}

And my test code:

int64_t sd_read_file(const char * path, uint8_t * buf, long int pos, size_t size)
{
    FILE * file = NULL;
    int ret = sd_cached_file(path, &file);  // To avoid call fopen() frequently
    if (ret != 0)
        return ret;
    
    ret = fseek(file, pos, SEEK_SET);
    if (ret != 0)
        return ret > 0 ? -ret : ret;
    else
        return fread(buf, 1, size, file);
}

double sd_test_read(const char * path, size_t blocksize)
{
    uint8_t * buf = (uint8_t *)malloc(blocksize);
    if (buf == NULL)
        return -1.0;
    
    int64_t bytes = 0;
    int64_t from = esp_timer_get_time();
    for (int64_t i = 0; ; i += blocksize)
    {
        int64_t ret = sd_read_file(path, buf, i, blocksize);
        if (ret > 0)
            bytes += ret;
        else
            break;
    }
    int64_t to = esp_timer_get_time();

    free(buf);
    
    return 1000000.0 * bytes / (to - from);
}

double speed = sd_test_read("/sdcard/37m.zip", 64*1024);
printf("---------------SD SPEED: %.2f KB/s\n", speed/1000);

[igrr]

Hi @keelung-yang, what read speed do you need to achieve?
If you use SD interface in 4-line mode, instead of SPI, you can get read speed around 3MB/s, from a FAT partition, and around 12MB/s if you read from card as a raw block device.

[keelung-yang]

Hi @igrr , > 50KB/s is enough to feed our 500kbps CAN bus application in theory. Since I've built a task which read data from SD card to a Ring Buffer, and then our CAN application fetch data from the Ring Buffer.

But, I found this while struggling with read spead: https://esp32.com/viewtopic.php?t=1942

esp_vfs_read() is being called to read 128 bytes at a time

I can write into SD card with the speed of ~6MB using fwrite but the reading speed from the SD card (fread) is aroung 1MB!!

It's not appropriate.
The chunk size 128 bytes should be configured by sdkconfig, but it's not.

[igrr]

Hi @keelung-yang, the default file buffer size is a compile-time constant of the C library (newlib). Since we are not building newlib as part of IDF build process, it is not easily possible to modify it using sdkconfig.

Currently you can do one of the following things to improve the read speed:

1. Increase the buffer size used by C library for the given FILE, after opening the file. This can be done using C function setvbuf. The C library will then perform reads from the VFS in chunks equal to the new buffer size.

2. Bypass the C library and its buffering layer, and read directly from the file descriptor using read(fileno(file), buf, buf_size) (where file is your FILE* stream).

We will also look into enabling the unbuffered stream read optimization in the next release of the toolchain. This will not change the behavior in the default (buffered) case, but will allow the fread function to read directly into user-provided buffer (without chunking) if buffering has been disabled with setvbuf.

[ruffle-b]

FWIW, if anyone else trips across this, I've just been doing some testing (SPI 1 bit on a Wemos D32-Pro) and with the standard buffering it was taking some 60 seconds to read a 50MB file.

With a 16k buffer, that's cut down to ~29 seconds. I tried 32 and 64k but that didn't improve it any more and an 8k buffer was slower.

Still not brilliant but twice as good as before.

HTH

[InfiniteYuan]

@igrr What is the highest rate if use SPI mode？

[tom-borcin]

Hello @keelung-yang,

thank you for reporting this issue. At Espressif we really appreciate it when our users report issues as it helps us make our products better. Unfortunately there hasn't been any progress with this issue for the last 2 years and it looks like it's been explained and there are workarounds proposed. Because of that we are closing it to keep our backlog manageable.

If you don't agree with this decision, please feel free to reopen the issue or contact me directly.

Regards,

Tomas