Add Initial Debian Packaging[WIP]

[aj4ayushjain]

This is the directory which can be used to build the debian package from the github repository of scancode-toolkit in Python2.

[codecov]

Codecov Report

Merging #1636 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #1636   +/-   ##
========================================
  Coverage    81.16%   81.16%           
========================================
  Files          125      125           
  Lines        15474    15474           
========================================
  Hits         12559    12559           
  Misses        2915     2915

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fc44d1...e72934b. Read the comment docs.

[codecov]

Codecov Report

Merging #1636 into develop will increase coverage by 0.09%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #1636      +/-   ##
===========================================
+ Coverage    78.43%   78.53%   +0.09%     
===========================================
  Files          131      131              
  Lines        16727    16727              
===========================================
+ Hits         13120    13136      +16     
+ Misses        3607     3591      -16

Impacted Files	Coverage Δ
src/licensedcode/index.py	73.64% <0%> (+0.22%)	⬆️
src/licensedcode/query.py	75.93% <0%> (+0.26%)	⬆️
src/scancode/cli.py	76.63% <0%> (+0.27%)	⬆️
src/licensedcode/match.py	77.06% <0%> (+0.37%)	⬆️
src/textcode/markup.py	95.74% <0%> (+2.12%)	⬆️
src/scancode/api.py	97.46% <0%> (+5.06%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 629944e...4ea6b1f. Read the comment docs.

[pombredanne]

@aj4ayushjain you gave these instructions in the chatroom:

1.Obtain the tar for latest version.
2.Then extract it and cd into it.(keeping tar and extracted folder in same directory.)
3.After that place debian folder into extracted folder
4.Then run $debuild -us -uc but first install the dependencies from pip and the package( https://launchpad.net/~dank/+archive/ubuntu/python-fixes/) and then install the .deb obtained.

[pombredanne]

pycyptodome dos not look like the correct name?

[dankegel]

right, another typo, the correct name is python-pycryptodrome

[aj4ayushjain]

i checked python-cryptodome is the correct name.

[pombredanne]

This is not correct, but that's not a big deal for now

[dankegel]

style suggestion: sort dependencies alphabetically, and use a comma even on last line so that diffs are cleaner if you have to add one to the end later

[aj4ayushjain]

Alright will make the changes

[dankegel]

typo alert: it's python-unicodecsv

[dankegel]

Easier recipe if you just want to see if it produces good binary packages

1. mv etc/debian debian
2. debuild -b -uc -us
3. try installing the resulting .deb

Note the many lintian errors produced by debuild. At least some of them may need addressing.

If the ci infrastructure allows, you'd want to do a test install of the resulting package; that would have caught the two typos in package names.

Probably also want to support autopkgtest so that debian package builders can verify that the package works; see https://wiki.debian.org/ContinuousIntegration/autopkgtest

[dankegel]

Also, may as well just put the debian directory at the top level rather than etc until there's a reason to do anything else; that gets rid of one of the steps in the recipe.

[aj4ayushjain]

Yeah i have to do that..Thinking of keeping the debian/ top level directory in a branch so user can checkout that branch and use it .

[dankegel]

The lintian errors are along the lines of

I: dh_python2 pydist:228: Cannot find package that provides boolean.py. Please add package that provides it to Build-Depends or add "boolean.py python-boolean.py" line to debian/pydist-overrides or add proper dependency to Depends by hand and ignore this info.

and it has one of those complains for each of

boolean.py
extractcode_7z
extractcode_libarchive
fingerprints
jaraco.timing
javaproperties
license_expression
packageurl_python
patch
publicsuffix2
pyahocorasick
pycryptodome
pymaven_patch
saneyaml
spdx_tools
tempora
text_unidecode
typecode_libmagic
url
yg.lockfile

The ones that have debian packages need to be added (as you've started doing).

The ones that don't have a debian package... well, we could try installing scancode's
bundled copies of them to a private directory and add that to the pythonpath for this app.

Putting them in the user install is not really an option for a debian-packaged python app.

[aj4ayushjain]

These are the dependencies which need to bundled and packaged or updated version is not available right now for some so will package them later.

1. publicsuffix2
2. py2-ipaddress(need updation)
3. url
4. fingerprints
5. backports.os
6. future(need updation)
7. text-unidecode
8. saneyaml
9. boolean.py
10. license_expression
11. pyahocorasick
12. pdfminer.six
13. pefile
14. pymaven-patch
15. colorama(need updation)
16. spdx-tools
17. yg.lockfile
18. pytz
19. tempora
20. jaraco.timing

[aj4ayushjain]

The meaning of each of the file for debian packaging(debian/ dir) to give a context

debian/changelog:-

This log which maintains the changes done in package over different version .There is a specific format which is to followed as suggested by debian.

debian/control:-

It contains information about the source package and binary package. The first paragraph contains the info for source package and following paragraphs for the binary package.
Source Package:-
Build-Depends:list of packages required to build the package.
Standards-Version:-version of debian policy manual
Binary Package:-
These are the dependencies of the program being built without which it will not run unless these are present.

debian/copyright:-

It contains the information and license of the upstream sources

debian/rules:- (executable)

It is recipe which defines how to build and install the package just like an Makefile

debian/compat:-

It defines the debhelper compatibility level.
Debhelper-It is the tool which used to build the debian package from the debian/rules file.

For more details on debian packaging you can check :-
https://www.debian.org/doc/manuals/maint-guide/dreq.en.html

[pombredanne]

@aj4ayushjain Some of your commits are missing the DCO signoff per https://github.com/nexB/scancode-toolkit/pull/1636/checks?check_run_id=180817210

The first few have it though: Signed-off-by: aj4ayushjain <[email protected]>
Can you add this to all commits?

[pombredanne]

@aj4ayushjain do you think you could rebase your PR so we can eventually merge soon enough?
Thanks!

[pombredanne]

Thank you for rebasing. What is your evaluation of what is left todo before we can merge?

[dankegel]

What are the etc/debian/*.ex files for? Are they used? I'm kind of opposed to checking in
unused example files...

[aj4ayushjain]

@dankegel U're suggestions and feedback are most welcomed.
I need to address the lintian errors and add autopkgtest for debian builders .

To build the package the instructions:-

1. tar --exclude-vcs -czvf ../scancode-toolkit_3.0.2.orig.tar.gz -C .. scancode-toolkit
2. debuild -us -uc

And @pombredanne after doing this and installing the .deb
Issue is with the naming of plugins we created pointing to system provided binaries.

[pombredanne]

@aj4ayushjain Can you tell me if I can merge this now? Or what is missing?

[dankegel]

Seems like it could use some alpha testing before merging. Are there any motivated users? I may be too busy to do much.

[pombredanne]

@aj4ayushjain ping? Do you mind providing quick instructions to test this?

[dankegel]

I believe it should be something like

sudo apt install devscripts build-essential equivs
sudo mk-build-deps -i
debuild -b -uc -us

where the sudo's only have to be run ones to install stuff,
and the debuild is run at the top of the source tree.

However, that fails for me with

dh: error: Parse error in debian/control, line 71, read: .
make: *** [debian/rules:9: clean] Error 25

Removing the last few lines of debian/control fixes that.

The next problem is

/bin/sh: 1: pip: not found

Adding the missing build-depend on python-pip in debian/control fixes that... well, not quite,
as on Ubuntu 20.04, installing python-pip actually installs pip3.
But it's probably a good enough workaround on older systems like Ubuntu 18.04.

That's as far as I got. Sorry for not trying this months ago.

If the current pull request can be coaxed into working, it might be worth landing, even though a proper packaging
would involve also doing debian packages for the things currently installed via pip or bundled wheels.
https://wiki.debian.org/Python/LibraryStyleGuide is an entry point to the debian packaging world for anyone
interested in picking this up.

[aj4ayushjain]

Pong! Sorry about the late response.
I was looking into it and couldn't figure the things out as well.Mostly I guess there are changes/updates of packages.
I will give it a try this in a day and comeback.

[pombredanne]

@aj4ayushjain ping againm FYI @jelmer is working on a Debian packaging. What would be needed to wrap this here?

[abhilashi]

Thanks for doing this

[pombredanne]

@abhilashi do you think you could help with this?

[pombredanne]

I am closing this for now and this can be reopened or recreated as needed.