improved handling of glob when reading files

[ghislainp]

Solve #957

[martindurant]

One flake error:

fsspec/core.py:536:89: E501 line too long (97 > 88 characters)

[ghislainp]

I don't understand the complexity of this code in get_fs_token_paths (core.py):

    if isinstance(paths, (list, tuple, set)):
        paths = expand_paths_if_needed(paths, mode, num, fs, name_function)
    else:  #->>>>
        if "w" in mode and expand:
            paths = _expand_paths(paths, name_function, num)
        elif "*" in paths:
            paths = [f for f in sorted(fs.glob(paths)) if not fs.isdir(f)]
        else:
            paths = [paths]

It seems that expand_paths_if_needed duplicates more or less the part of the code after #->>>> for the case of sequences. Spreading the same code in two functions seems error-prone in general, and indeed here, we have different behaviors depending on whether we have a list or a string (I'd expect the same behavior):

For instance:

get_fs_token_paths(["dir/*.txt"], mode="w", expand=False)   #-> glob expansion is done, because expand_paths_if_needed don't have 'expand' arg.

get_fs_token_paths("dir/*.txt", mode="w", expand=False)   "-> glob expansion is done

Another unexpected difference:

get_fs_token_paths(["dir/*.path"], expand=False)   #-> the directories found with glob("dir/*.path") are not excluded because expand_paths_if_needed does not exclude dirs.

get_fs_token_paths("dir/*.path", expand=False)   #->  the directories are excluded

I'd recommend to refactorize the code to get the same behavior for a list or a string:

    if not isinstance(paths, (list, tuple, set)):   # or maybe better:  if isinstance(paths, str):

        paths = [paths]

    if expand:

        paths = expand_paths_if_needed(paths, mode, num, fs, name_function)

    paths = sorted([f for f in paths if not fs.isdir(f)])

This code does honor "expand" for all models. It also filter out directories for all modes.