Can't open FLAC file with read on macOS (get unimplemented format error)

[ArnoXf]

Hi!

I got an issue when trying to open a FLAC file using the following simple code:

>>> import soundfile as sf
>>> data, sr = sf.read('test.flac')

That results in the following traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/envs/mqa/lib/python3.6/site-packages/soundfile.py", line 373, in read
    subtype, endian, format, closefd) as f:
  File "/usr/local/anaconda3/envs/mqa/lib/python3.6/site-packages/soundfile.py", line 740, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/usr/local/anaconda3/envs/mqa/lib/python3.6/site-packages/soundfile.py", line 1265, in _open
    "Error opening {0!r}: ".format(self.name))
  File "/usr/local/anaconda3/envs/mqa/lib/python3.6/site-packages/soundfile.py", line 1455, in _error_check
    raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))
RuntimeError: Error opening '/Users/afuhrmann/Desktop/test.flac': File contains data in an unimplemented format.

The available formats don't show FLAC:

>>> sf.available_formats()
{'AIFF': 'AIFF (Apple/SGI)', 'AU': 'AU (Sun/NeXT)', 'AVR': 'AVR (Audio Visual Research)', 'CAF': 'CAF (Apple Core Audio File)', 'HTK': 'HTK (HMM Tool Kit)', 'SVX': 'IFF (Amiga IFF/SVX8/SV16)', 'MAT4': 'MAT4 (GNU Octave 2.0 / Matlab 4.2)', 'MAT5': 'MAT5 (GNU Octave 2.1 / Matlab 5.0)', 'MPC2K': 'MPC (Akai MPC 2k)', 'PAF': 'PAF (Ensoniq PARIS)', 'PVF': 'PVF (Portable Voice Format)', 'RAW': 'RAW (header-less)', 'RF64': 'RF64 (RIFF 64)', 'SD2': 'SD2 (Sound Designer II)', 'SDS': 'SDS (Midi Sample Dump Standard)', 'IRCAM': 'SF (Berkeley/IRCAM/CARL)', 'VOC': 'VOC (Creative Labs)', 'W64': 'W64 (SoundFoundry WAVE 64)', 'WAV': 'WAV (Microsoft)', 'NIST': 'WAV (NIST Sphere)', 'WAVEX': 'WAVEX (Microsoft)', 'WVE': 'WVE (Psion Series 3)', 'XI': 'XI (FastTracker 2)'}

but the subformats with format flac does:

>>> sf.available_subtypes(format='FLAC')
{'PCM_S8': 'Signed 8 bit PCM', 'PCM_16': 'Signed 16 bit PCM', 'PCM_24': 'Signed 24 bit PCM'}

Also FLAC is in the supported formats in both soundfile.py and sndfile.h

    SF_FORMAT_FLAC          = 0x170000,     /* FLAC lossless file format */

Also, FLAC files can be played in macOS.

Does anyone know how to resolve that issue?

Thanks in advance

[bastibe]

How did you install soundfile? My first guess would be that you did not install the wheel version (which comes with a full-featured libsndfile), but used some other installation method instead that does not come with its own libsndfile, thus using your system's libsndfile, which is lacking FLAC support.

[asapsmc]

I am getting the same error. I did a conda install librosa, which took care of libsndfile dependencies
But I'm using libsndfile v1.0.28 (in the current conda environment), which is supposed to support FLAC.
@bastibe how do I correct this?

[asapsmc]

I'm working in a conda environment with:

• pysoundfile 0.9.0.post1
• libsndfile 1.0.28

[bastibe]

The problem in your case is probably that you're using pysoundfile, instead of soundfile. We deprecated the use of pysoundfile a while ago, and the version is no longer maintained.

[asapsmc]

That was a problem, but I uninstalled pysoundfile, and now installed soundfile (v0.10.3).
But I'm getting the same error: "RuntimeError: Error opening '/Users/.../audio/hains006.flac': File contains data in an unimplemented format.
Also, when I do soundfile.available_formats(), FLAC doesn't appear.

[bastibe]

You'll have to check which version of libsndfile is actually used by soundfile. It might still be using the one from conda, which would explain why the newer version of soundfile (which comes with a fully-configured libsndfile) didn't fix the problem.

Since you seem to be on macOS, you can also try installing libsndfile through homebrew. If I remember correctly, homebrew has options for enabling FLAC support.

If all else fails, you can use LD_PRELOAD to force loading a specific version of libsndfile and see if that fixes the issue. And then work backwards from there to figure out why soundfile is not loading the version you prefer.

[mgeier]

@MR-T77 You can try this to find out which libsndfile is used on your system:

import soundfile as sf
print(sf._libname)

[asapsmc]

Thanks for your help @bastibe and @mgeier. Unfortunately, I'm a little bit lost on what to do next...
Yesterday I had installed libsndfile through homebrew, but that didn't fix. (I tried also brew install libsndfile --with-flac but got "Error: invalid option: --with-flac" - on Macos Mojave - 10.14)
If I do print print(sf._libname) I get "/.../anaconda/envs/dynamite/bin/../lib/libsndfile.dylib" but I don't know how to interpret this info.
Any other path I may explore before LD_PRELOAD, which seems to require a higher level of expertise than the one I have.

[mgeier]

I get "/.../anaconda/envs/dynamite/bin/../lib/libsndfile.dylib" but I don't know how to interpret this info.

This tells you from which file on your system the libsndfile library is loaded.

The path you are getting points somewhere inside your Anaconda folder. It's not the one distributed with the soundfile wheel package.

There seem to be FLAC problems with the macOS version of the Anaconda package, see conda-forge/libsndfile-feedstock#2 and librosa/librosa#847 (comment). There doesn't seem to be a resolution.

You should try to uninstall libsndfile with conda, then it probably works.

[j9ac9k]

I just stumbled across this very issue, but the source of the conflicting libsndfile.dylib was from homebrew. libsndfile is a dependency of ffmpeg, portaudio, sox and other packages commonly distributed via the brew package manager on macOS.

Unfortunately, soundfile picks up the libsndfile.dylib file that homebrew distrubutes

$ python
Python 3.7.7 (default, Jun 19 2020, 13:21:43)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf._libname
'/usr/local/lib/libsndfile.dylib'
>>> import ctypes
>>> ctypes.util.find_library("sndfile")
'/usr/local/lib/libsndfile.dylib'

After uninstalling libsndfile from homebrew via brew uninstall --ignore-dependencies libsndfile soundfile works as expected

$ python
Python 3.7.7 (default, Jun 19 2020, 13:21:43)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf._libname
'libsndfile.dylib'

Unfortunately, tools like ffmpeg, sox, etc I use routinely, so removing the homebrew bundled dependency for it is rather problematic. Would there be a workaround to specify which libsndfile.dylib file should be used?

[bastibe]

You could probably LD_PRELOAD it. (I think that works on macOS, right?)

[j9ac9k]

AFAIK LD_PRELOAD does not work on macOS/OSX, but I'm operating out of my depth here. After re-installing libsndfile via homebrew:

For the record, this is the location of my libsndfile.dylib library provided by soundfile: /Users/ogi/.zinit/plugins/pyenv---pyenv/versions/3.7.7/envs/fred/lib/python3.7/site-packages/_soundfile_data/libsndfile.dylib

$ python
Python 3.7.7 (default, Jun 19 2020, 13:21:43)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf._libname
'/usr/local/lib/libsndfile.dylib'
>>> exit()

$ DYLD_INSERT_LIBRARIES="/Users/ogi/.zinit/plugins/pyenv---pyenv/versions/3.7.7/envs/fred/lib/python3.7/site-packages/_soundfile_data/libsndfile.dylib" python
Python 3.7.7 (default, Jun 19 2020, 13:21:43)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf._libname
'/usr/local/lib/libsndfile.dylib'
>>> exit()

$ LD_PRELOAD="/Users/ogi/.zinit/plugins/pyenv---pyenv/versions/3.7.7/envs/fred/lib/python3.7/site-packages/_soundfile_data/libsndfile.dylib" python
Python 3.7.7 (default, Jun 19 2020, 13:21:43)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile as sf
>>> sf._libname
'/usr/local/lib/libsndfile.dylib'
>>>

Looking through the documentation for ctypes.util.find_library() I found this blurb

If wrapping a shared library with ctypes, it may be better to determine the shared library name at development time, and hardcode that into the wrapper module instead of using find_library() to locate the library at runtime.

[mgeier]

If wrapping a shared library with ctypes, it may be better to determine the shared library name at development time, and hardcode that into the wrapper module instead of using find_library() to locate the library at runtime.

The soundfile module actually does that, but only on the second attempt.

First, it tries to find the library with ctypes.util.find_library(). If that fails, it tries the hard-coded path in _soundfile_data.

The problem here is that the first attempt does not fail ...

As a work-around, you can monkey-patch the find_library() function to make it fail unconditionally:

def dummy_find_library(name):
    return None

import ctypes.util
ctypes.util.find_library = dummy_find_library

import soundfile as sf

...

[mgeier]

Another idea: Why not just overwrite /usr/local/lib/libsndfile.dylib?

I assume this is a symbolic link provided by brew, right?

You could just copy the libsndfile.dylib from the soundfile module there, and hope that all other programs like ffmpeg, portaudio, sox continue working.

[j9ac9k]

First, it tries to find the library with ctypes.util.find_library(). If that fails, it tries the hard-coded path in _soundfile_data.

After I posted my reply, I noticed that too! I don't have enough experience doing this sort of thing to make a recommendation on how it should be done so I leave that to the judgment of more experienced folks. I am curious though if it would be worthwhile to try loading the hard-coded path first and then fall back on find_library()?

Another idea: Why not just overwrite /usr/local/lib/libsndfile.dylib?

I think this is what I'll end up doing for the time being. I must admit I am curious why their version of libsndfile does not support flac files, but that's not a topic for this issue tracker.

Thanks for your input @mgeier !

[j9ac9k]

If there would be consideration for implementing the change where first soundfile attempts to read the hard-coded paths before falling back on find_library() I'll gladly submit one.

[mgeier]

I don't think there is a "right" or "wrong" order in which to look for the library, each one has their advantages and disadvantages.

The original idea was to give users a simple way to override the pre-packaged library with their own.

As usual in Linux (and this works in macOS as well), they could put their version of the library into /usr/local/lib/libsndfile.so, and it would be preferred over the one installed by the package manager (which is typically in /usr/lib/libsndfile.so or something like that).

That's exactly what happened in your case, @j9ac9k. If someone puts a library into /usr/local/lib, they normally want it to be used, don't they? So I would say that's a feature, not a bug.

Now if we reverse the order of library loading, this wouldn't be possible anymore, people would not (easily) be able to provide their own library version. This might actually be an advantage for some users, but I think in general it's not.

If we disable the current customization opportunity, I think we should introduce a new one, do you have an idea for that?

I wanted to mention a related discussion in spatialaudio/python-sounddevice#130, but then I saw that you have also commented there!

UPDATE: another related issue: #222.

[lminer]

I'm getting this issue on ubuntu as well.

[j9ac9k]

I'm getting this issue on ubuntu as well.

I didn't come up w/ a good fix; in my case I just uninstalled the libsndfile that was packaged via homebrew. I do periodically think about this issue; I suspect an environment variable may be the way to go here; I'm not sure how difficult that would be to implement though.

[bastibe]

I suspect an environment variable may be the way to go here; I'm not sure how difficult that would be to implement though.

This would be relatively easy to implement. Pull requests would certainly be welcome!

[ghost]

I'm getting the same issue on ubuntu 20: 04

When I uninstall libsndfile, it uninstalled librosa as well and when I install just soundfile and libsndfile.
sf.read("text.flac"); works fine, but when I noticed librosa was uninstalled, I installed it again and since librosa takes care of libsnfile dependencies, it installed a different version and sf.read("text.flac") stopped working.

any ideas why this is so?

[ghost]

Find a work around with the issue hope it might help out,
still to try @mgeier solution with the def dummy_find_library() function,

but what I did was I uninstall librosa and it uninstalled all it dependent packages
:$ conda uninstall librosa -y

and I install soundfile and loaded the .flac file and it works.
but when I install librosa again installed libsnfile and soundfile._libname by default picks and "/.../anaconda//libsndfile.so.1"* found in anaconda.

I uninstall just that package
:$ conda remove --force libndfile

soundfile._libname default to "libsnfile.so.1", which works alongside librosa.

Hoping to get a far more better and accurate solution in the future.

[danielvachalek]

Hallefrickenluja! @vegasc you rescued me from 5 hours of package hell.

conda remove --force libndfile worked for me!