Update to SdFat 2.1.1 with UTF-8 support

[earlephilhower]

No description provided.

[earlephilhower]

Sorry, guys, for the churn. I seem to have lost the ability to run host tests on my machine so unfortunately have to debug the host builds using the CI system. Time to upgrade to Ubu20.04, I guess...

[mcspr]

...and the lib itself seems to have an issue - there is declaration of toUpcase, but it is defined as sdfat::toUpcase, so the linking fails

> xtensa-lx106-elf-nm -C .\.pio\build\nodemcuv2\lib025\libesp8266sdfat.a | select-string toUpcase

         U toUpcase(unsigned short)
         U toUpcase(unsigned short)
         U toUpcase(unsigned short)
         U toUpcase(unsigned short)
00000008 T sdfat::toUpcase(unsigned short)

[earlephilhower]

@mcspr hold off on testing this a bit. I've dug into the latest upstream changes and it should now be possible to remove all of the namespace patches. That would significantly shrink the diffs from upstream.

Now that I have an Ubu20 VM and can run host tests there, I'll debug anything that breaks before pushing an update.

[earlephilhower]

@mcspr you can try this version now. All tests but one valgrind one pass, and FWIW the dump points to a problem in the upstream SdFat library. I'll look into it tomorrow (well, later today since it's 1AM here), but OTW if you have a handy SD card this may be worth a run.

2021-11-03T08:00:01.6217614Z ==7070== Conditional jump or move depends on uninitialised value(s)
2021-11-03T08:00:01.6220064Z ==7070==    at 0x3150FD: FatFile::parsePathName(char const*, FatLfn_t*, char const**) (FatFileLFN.cpp:507)
2021-11-03T08:00:01.6222013Z ==7070==    by 0x3107A9: FatFile::open(FatFile*, char const*, unsigned char) (FatFile.cpp:448)
2021-11-03T08:00:01.6223860Z ==7070==    by 0x310638: FatFile::open(FatVolume*, char const*, unsigned char) (FatFile.cpp:422)
2021-11-03T08:00:01.6225630Z ==7070==    by 0x2F5187: FatVolume::exists(char const*) (FatVolume.h:86)
2021-11-03T08:00:01.6227292Z ==7070==    by 0x2F572A: sdfs::SDFSImpl::exists(char const*) (SDFS.h:85)
2021-11-03T08:00:01.6228874Z ==7070==    by 0x2D32F2: fs::FS::exists(char const*) (FS.cpp:379)
2021-11-03T08:00:01.6230462Z ==7070==    by 0x192CFD: sdfs_test::____C_A_T_C_H____T_E_S_T____110() (test_fs.inc:119)
2021-11-03T08:00:01.6232199Z ==7070==    by 0x24F4BC: Catch::FreeFunctionTestCase::invoke() const (catch.hpp:5874)
2021-11-03T08:00:01.6233964Z ==7070==    by 0x220F53: Catch::TestCase::invoke() const (catch.hpp:6779)
2021-11-03T08:00:01.6235739Z ==7070==    by 0x24A24F: Catch::RunContext::invokeActiveTestCase() (catch.hpp:5473)
2021-11-03T08:00:01.6238099Z ==7070==    by 0x249A8D: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:5445)
2021-11-03T08:00:01.6240343Z ==7070==    by 0x246DCB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:5284)
2021-11-03T08:00:01.6242064Z ==7070==  Uninitialised value was created by a stack allocation
2021-11-03T08:00:01.6244345Z ==7070==    at 0x310664: FatFile::open(FatFile*, char const*, unsigned char) (FatFile.cpp:425)
2021-11-03T08:00:01.6246169Z ==7070== 
2021-11-03T08:00:01.6251775Z ==7070== Conditional jump or move depends on uninitialised value(s)
2021-11-03T08:00:01.6254096Z ==7070==    at 0x31510D: FatFile::parsePathName(char const*, FatLfn_t*, char const**) (FatFileLFN.cpp:507)
2021-11-03T08:00:01.6256315Z ==7070==    by 0x3107A9: FatFile::open(FatFile*, char const*, unsigned char) (FatFile.cpp:448)
2021-11-03T08:00:01.6258461Z ==7070==    by 0x310638: FatFile::open(FatVolume*, char const*, unsigned char) (FatFile.cpp:422)
2021-11-03T08:00:01.6260848Z ==7070==    by 0x2F5187: FatVolume::exists(char const*) (FatVolume.h:86)
2021-11-03T08:00:01.6262891Z ==7070==    by 0x2F572A: sdfs::SDFSImpl::exists(char const*) (SDFS.h:85)
2021-11-03T08:00:01.6264840Z ==7070==    by 0x2D32F2: fs::FS::exists(char const*) (FS.cpp:379)
2021-11-03T08:00:01.6266761Z ==7070==    by 0x192CFD: sdfs_test::____C_A_T_C_H____T_E_S_T____110() (test_fs.inc:119)
2021-11-03T08:00:01.6268827Z ==7070==    by 0x24F4BC: Catch::FreeFunctionTestCase::invoke() const (catch.hpp:5874)
2021-11-03T08:00:01.6270938Z ==7070==    by 0x220F53: Catch::TestCase::invoke() const (catch.hpp:6779)
2021-11-03T08:00:01.6273049Z ==7070==    by 0x24A24F: Catch::RunContext::invokeActiveTestCase() (catch.hpp:5473)
2021-11-03T08:00:01.6275817Z ==7070==    by 0x249A8D: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:5445)
2021-11-03T08:00:01.6277717Z ==7070==    by 0x246DCB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:5284)
2021-11-03T08:00:01.6279120Z ==7070==  Uninitialised value was created by a stack allocation
2021-11-03T08:00:01.6280534Z ==7070==    at 0x310664: FatFile::open(FatFile*, char const*, unsigned char) (FatFile.cpp:425)
2021-11-03T08:00:01.6281714Z ==7070== 

[earlephilhower]

Testing on real HW (once I reverted to a working Ubuntu kernel!) looks good on this. The old SD.h examples go and even Unicode UTF-8 runs properly, too.

Looks good to go...

Type any character to begin
ls:
 11075648 demo.bin
        0 time_lapse/
       18 datalog.txt
       18 newdatalog.txt
        0 😀/
         20 россиянин
         17 très élégant
          9 狗.txt
After remove and rmdir
 11075648 demo.bin
        0 time_lapse/
       18 datalog.txt
       18 newdatalog.txt
Done!

[mcspr]

At your discretion. Seems to be working and 2.0.2 -> 2.1.1 update did not break anything. But, I also just tried the barebones example + small sketch utilizing SDFS

...

Card successfully initialized.

Card size: 2030 MB (MB = 1,000,000 bytes)

Volume is FAT64, Cluster size (bytes): 32768

Files found (date time size name):
2021-11-04 21:35            0 специально для #8355.txt
2021-11-04 21:35            0 создаем несколько файлов на sd.txt
2021-11-04 21:35            0 на русском языке.txt
2021-11-04 21:36            0 (проверяем unicode).txt

Success!  Type any character to restart.

I have also tried FSBrowser, with some small modifications to support FsLib APIs in the SDFAT.{h,cpp} to work with exfat volume. Web portion also works correctly for editing such files.

> [console]::InputEncoding | select -property encodingname
EncodingName
------------
Unicode (UTF-8)

> curl -s 'http://fsbrowser.local/list?dir=/' | convertfrom-json
type size time       name
---- ---- ----       ----
dir  -    1636061712 System Volume Information
file 0    1636061726 специально для #8355.txt
file 0    1636061736 создаем несколько файлов на sd.txt
file 0    1636061750 на русском языке.txt
file 0    1636061778 (проверяем unicode).txt

> curl -s -X PUT 'http://fsbrowser.local/edit?path=/проверка%20через%20FSBROWSER.txt'
> curl -s 'http://fsbrowser.local/list?dir=/' | convertfrom-json
type size time       name
---- ---- ----       ----
...
file 0    1636061800 проверка через FSBROWSER.txt

(specifically, changes are File32 -> FsFile, dirEntry() -> getModifyDateTime() / getCreateDateTime(), availableSpaceForWrite)

[earlephilhower]

(specifically, changes are File32 -> FsFile, dirEntry() -> getModifyDateTime() / getCreateDateTime(), availableSpaceForWrite)

We might want to consider an ExFatSDFS class (convert SDFS into a template and then instantiate one with File32 and one with FsFile with whatever specializations).

In any case, this new version of SdFat has massively fewer changes vs. upstream, thanks to greiman's recent changes, so future pulls should be much easier.