Proposal to change stdlib and mapping file formats.

[darwinshameran]

The current formats of stdlib and mapping aren't very flexible, changing them to something more suitable like JSON or python dictionary would make them more flexible. This change would make it easier to include and exclude certain modules based on python version, and provide simple fixes to issues such as #61, #62 and #74.

[bndr]

I'm open to suggestions how the structure should look like :)

[darwinshameran]

Here's some suggestions and example usage. Personally, I'm more fond of the stdlib example 2 than example 1.

import sys

# stdlib example 1

stdlib = {
    "concurrent": {2: False, 3: True},
    "concurrent.futures": {2: False, 3: True},
    "configparser": {2: True, 3: True},
    "contextlib": {2: True, 3: True},
    "copy": {2: True, 3: False},
    "requests": {2: False, 3: False}
}

# stdlib example 2

stdlib_ = {
    3: [
        "concurrent",
        "concurrent.futures",
        "configparser",
        "contextlib"
    ],

    2: [
        "configparser",
        "contextlib",
        "copy"
    ]
}

# usage example 1

version = 3
data = [k for k, v in stdlib.items() if v[version]]
print(data) # ['concurrent', 'concurrent.futures', 'configparser', 'contextlib']

version = 2
data = [k for k, v in stdlib.items() if v[version]]
print(data) # ['configparser', 'contextlib', 'copy']

# usage example 2

version = 3
data = stdlib_[version]
print(data) # ['concurrent', 'concurrent.futures', 'configparser', 'contextlib']


version = 2
data = stdlib_[version]
print(data) # ['configparser', 'contextlib', 'copy']

# mapping example

mappings = {
    "AG_fft_tools": ["agpy"],
    "BB_jekyll_hook": ["bitbucket_jekyll_hook"],
    "ZServer": ["Zope2"],
    "sklearn": ["scikit_learn", "numpy", "scipy"]
}

#  usage example
candidates = ['concurrent', 'concurrent.futures', 'configparser',
              'contextlib', 'ZServer', 'sklearn']

result = mappings["sklearn"]
print(result)
result = mappings["ZServer"]
print(result)

def get_pkg_names(pkgs):
    result = [mappings[x] if x in mappings else x for x in candidates]
    result = ([x for y in result for x in y if isinstance(y, list)]
             + [x for x in result if not isinstance(x, list)])

    return result

print(get_pkg_names(candidates))
#  ['Zope2', 'scikit_learn', 'numpy', 'scipy', 'concurrent',
    'concurrent.futures', 'configparser', 'contextlib']

[bndr]

Your mappings structure looks solid, would definitely implement that.

Your stdlib second example is also quite nice, but I don't exactly like the idea of storing duplicates for both versions, because most of the std libs are the same for python2 and 3.

Perhaps we can have a general array of stdlib that work for python 2 and 3 and then extend that array with python specific version. So basically if a package is python 2 only, then it is present only in the python 2 set, if the package is available for python 2 and 3, then it is stored in the "general" set, and the same goes for python3 only packages.

Other than that, looks good.

[darwinshameran]

Building on PR #75:

if sys.version_info[0] > 2:
    open_func = open
    py2 = False
    py3_exclude = ["fabric",...]
else:
    open_func = codecs.open
    py2 = True
    py2_exclude = ["concurrent", "concurrent.futures"] 
...    

with open(join("stdlib"), "r") as f:
    data = [x.strip() for x in f.readlines()]

    if py2:
        data = [x for x in data if x not in py2_exclude]
    else:
        data = [x for x in data if x not in py3_exclude]

    return sorted(list(set(packages) - set(data)))

This would exclude modules based on python version, while keeping a general set in the first place.