Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use DEP-14 branch names debian/latest and upstream/latest #225

Merged
merged 4 commits into from
Jan 19, 2025

Conversation

ottok
Copy link
Contributor

@ottok ottok commented Nov 20, 2024

In DEP-14, the preferred branch name for the Debian packaging target branch is debian/latest and the preferred name for the upstream import target branch is upstream/latest. Note that the upstream development branch name can be whatever and should stay as it is upstream, typically main or master. The branch upstream/latest should not point to the latest upstream development commit, but to the latest commit that was used as the upstream release that the Debian revision was derived from.

@ottok
Copy link
Contributor Author

ottok commented Nov 20, 2024

Please allow the CI to run so I can see the result, and give me enough time to fix all errors :)

@samueloph
Copy link
Member

Please allow the CI to run so I can see the result, and give me enough time to fix all errors :)

It looks like CI did run when you submitted the PR, am I missing something?

@ottok
Copy link
Contributor Author

ottok commented Nov 21, 2024

It looks like CI did run when you submitted the PR, am I missing something?

After submission there was a banner stating that "user does not have permissions to trigger the CI" and that my account must be allowlisted. Later seems the CI ran, not sure if somebody did something or not.

@creekorful
Copy link
Member

I did 😄

@ottok
Copy link
Contributor Author

ottok commented Nov 21, 2024

I tested this manually by running dh-make-golang make -dep14 -git_revision v0.1.0 -pristine-tar -type library -upstream_git_history -wrap-and-sort ast [github.com/muesli/mango-pflag](http://github.com/muesli/mango-pflag) and it works.

I also added further refinements based on my Debian packaging best practices knowledge. I am even tempted to add a generic README.source that says something like "This package is maintained by the Go team. Please follow policy at https://go-team.pages.debian.net/ to facilitate collaboration among team members". But for now I will first wait for your feedback.

Let me know if you want me to split this into multiple separate PRs.

ottok added a commit to ottok/dh-make-golang that referenced this pull request Nov 24, 2024
template.go Outdated Show resolved Hide resolved
template.go Outdated Show resolved Hide resolved
@ottok ottok force-pushed the dep14 branch 2 times, most recently from e354493 to e2bfbec Compare November 24, 2024 21:00
@ottok ottok requested a review from n-peugnet November 24, 2024 21:01
@ottok
Copy link
Contributor Author

ottok commented Nov 24, 2024

Thanks for the review @n-peugnet. Add remarks are resolved and branch updated.

I also have test binaries available at https://salsa.debian.org/otto/dh-make-golang/-/jobs/6641396/artifacts/browse/debian/output/ in case you want to test how this would behave eventually when the dh-make-golang package in Debian gets updated.

ottok added a commit to ottok/dh-make-golang that referenced this pull request Nov 24, 2024
ottok added a commit to ottok/dh-make-golang that referenced this pull request Nov 24, 2024
@ottok
Copy link
Contributor Author

ottok commented Nov 24, 2024

Example output when trying to package Glow:

# dh-make-golang make -dep14 -git_revision v1.2.0 -pristine-tar -type program -upstream_git_history -wrap-and-sort ast github.com/charmbracelet/glow

2024/11/24 05:18:14 Starting "dh-make-golang v0.7.0 linux/amd64"
2024/11/24 05:18:14 Downloading "github.com/charmbracelet/glow/..."
2024/11/24 05:18:17 Determining upstream version number
2024/11/24 05:18:17 Found latest tag "v1.2.0"
2024/11/24 05:18:17 Latest tag "v1.2.0" matches master
2024/11/24 05:18:17 Package version is "1.2.0"
2024/11/24 05:18:17 findMains: Running /usr/bin/go list -e -f {{.ImportPath}} {{.Name}} github.com/charmbracelet/glow/... in /tmp/dh-make-golang934566832/src/github.com/charmbracelet/glow
go: downloading github.com/charmbracelet/charm v0.8.2
go: downloading github.com/charmbracelet/glamour v0.2.1-0.20200829234023-6c0e29c4dae5
go: downloading github.com/charmbracelet/bubbletea v0.12.2
go: downloading github.com/meowgorithm/babyenv v1.3.0
...
go: downloading github.com/dlclark/regexp2 v1.2.0
go: downloading github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
2024/11/24 05:18:21 Determining dependencies
2024/11/24 05:18:22 Downloading https://github.com/charmbracelet/glow/archive/v1.2.0.tar.gz
2024/11/24 05:18:23 Moving tempfile to "glow_1.2.0.orig.tar.gz"
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
2024/11/24 05:18:23 Adding remote "origin" with URL "[email protected]:go-team/packages/glow.git"
2024/11/24 05:18:23 Adding remote "upstreamvcs" with URL "https://github.com/charmbracelet/glow"
2024/11/24 05:18:23 Running "git fetch upstreamvcs"
remote: Enumerating objects: 3762, done.
remote: Counting objects: 100% (1567/1567), done.
remote: Compressing objects: 100% (157/157), done.
remote: Total 3762 (delta 1465), reused 1428 (delta 1409), pack-reused 2195 (from 1)
Receiving objects: 100% (3762/3762), 2.82 MiB | 3.50 MiB/s, done.
Resolving deltas: 100% (2555/2555), done.
From https://github.com/charmbracelet/glow
 * [new branch]      beta                -> upstreamvcs/beta
 * [new branch]      fix/styles-from-env -> upstreamvcs/fix/styles-from-env
 * [new branch]      master              -> upstreamvcs/master
 * [new branch]      pixterm             -> upstreamvcs/pixterm
 * [new branch]      proposals-update    -> upstreamvcs/proposals-update
 * [new branch]      revert              -> upstreamvcs/revert
 * [new branch]      timeout             -> upstreamvcs/timeout
 * [new tag]         v0.1                -> v0.1
 * [new tag]         v0.1.1              -> v0.1.1
 * [new tag]         v0.1.2              -> v0.1.2
 * [new tag]         v0.1.3              -> v0.1.3
 * [new tag]         v0.1.4              -> v0.1.4
 * [new tag]         v0.1.5              -> v0.1.5
 * [new tag]         v0.1.6              -> v0.1.6
 * [new tag]         v0.2.0              -> v0.2.0
 * [new tag]         v1.0.0              -> v1.0.0
 * [new tag]         v1.0.1              -> v1.0.1
 * [new tag]         v1.0.2              -> v1.0.2
 * [new tag]         v1.1.0              -> v1.1.0
 * [new tag]         v1.2.0              -> v1.2.0
 * [new tag]         v1.2.1              -> v1.2.1
 * [new tag]         v1.3.0              -> v1.3.0
 * [new tag]         v1.4.0              -> v1.4.0
 * [new tag]         v1.4.1              -> v1.4.1
 * [new tag]         v1.5.0              -> v1.5.0
 * [new tag]         v1.5.1              -> v1.5.1
 * [new tag]         v2.0.0              -> v2.0.0
2024/11/24 05:18:26 Build-Dependency "github.com/charmbracelet/charm" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control
2024/11/24 05:18:27 Setting debian/watch to track release tarball
2024/11/24 05:18:27 Done!

Packaging successfully created in /tmp/build/source/packages/glow
    Source: glow
    Binary: glow

Resolve all TODOs in itp-glow.txt, then email it out:
    /usr/sbin/sendmail -t < itp-glow.txt

Resolve all the TODOs in debian/, find them using:
    grep -r TODO debian

To build the package, commit the packaging and use gbp buildpackage:
    git add debian && git commit -S -m 'Initial packaging'
    gbp buildpackage --git-pbuilder

To create the packaging git repository on salsa, use:
    dh-make-golang create-salsa-project glow

Once you are happy with your packaging, push it to salsa using:
    git push origin debian/latest
    gbp push

NOTE: Full upstream git history has been included as per pkg-go team's
      new workflow.  This feature is new and somewhat experimental,
      and all feedback are welcome!
      (For old behavior, use --upstream_git_history=false)

The upstream git history is being tracked with the remote named "upstreamvcs".
To upgrade to the latest upstream version, you may use something like:
    git fetch upstreamvcs     # note the latest tag or commit-ish
    uscan --report-status     # check we get the same tag or commit-ish
    gbp import-orig --sign-tags --uscan --upstream-vcs-tag=<commit-ish>
± cat debian/gbp.conf 
[DEFAULT]
debian-branch = debian/latest
upstream-branch = upstream/latest
dist = DEP14
pristine-tar = True

# Lax requirement to use branch name 'debian/latest' so that git-buildpackage
# will always build using the currently checked out branch as the Debian branch.
# This makes it easier for contributors to work with feature and bugfix
# branches.
ignore-branch = True

# Configure the upstream tag format below, so that 'gbp import-orig' will run
# correctly, and link tarball import branch ('upstream/latest') with the
# equivalent upstream release tag, showing a complete audit trail of what
# upstream released and what was imported into Debian.
#
# Most go packages have tags of form 'v1.0.0'
#upstream-vcs-tag = v%(version%~%-)s

# Ensure the Debian maintainer signs git tags automatically
#sign-tags = True

± git branch
* debian/latest
  pristine-tar
  upstream/latest

± g-log 
* d72b1d8 New upstream version 1.2.0 17 hours ago (HEAD -> debian/latest, tag: upstream/1.2.0, upstream/latest)
* e9d728c Add "hidden" command/config for switching on mouse wheel support 4 years ago (tag: v1.2.0)
* c16a146 Load the whole stash, page by page, automatically 4 years ago
* 5023d4a Remove some code duplication with stash and news message handling 4 years ago

@ottok
Copy link
Contributor Author

ottok commented Nov 24, 2024

@creekorful:

I did 😄

Can you add me to the Debian org on GitHub so GitHub knows I am to be trusted?

image

@n-peugnet
Copy link
Contributor

From what I understand, some of these change should first be proposed to the team as a whole, since the branch names have already been updated in the past, for example: https://go-team.pages.debian.net/workflow-changes.html

About the removal of the .gitignore file update, I wonder if it would be possible to edit the .git/info/exclude instead, and then add an option to gbp to update this file on gbp clone.

ottok added a commit to ottok/dh-make-golang that referenced this pull request Nov 25, 2024
@ottok
Copy link
Contributor Author

ottok commented Nov 29, 2024

About the removal of the .gitignore file update, I wonder if it would be possible to edit the .git/info/exclude instead, and then add an option to gbp to update this file on gbp clone.

I split this out into #230.

@ottok ottok force-pushed the dep14 branch 2 times, most recently from ef5745f to 8253f23 Compare December 1, 2024 01:36
@ottok
Copy link
Contributor Author

ottok commented Dec 22, 2024

Seems not all are aware of DEP-14 contents, so I'll re-iterate here the relevant part:

Go team policy in 2017 switched to use DEP-14:
https://go-team.pages.debian.net/workflow-changes.html#wf-2017-11-dep14

1.4. Adopt DEP-14 branch naming
Rationale
Consistency in our branch naming makes it easier for team-internal and team-external contributors to understand/interact with our packaging repositories.

In 2017 DEP-14 was still being changed, and later the Debian-wide agreement was that the branch names should be debian/latest and upstream/latest.

https://dep-team.pages.debian.net/deps/dep14/

In Debian this means that uploads to unstable should be prepared in the debian/latest branch ...
...
By default, the latest upstream version should be imported in the upstream/latest branch ...

This proposal is simply updating the implementation, while the intent of the policy stays the same.

Copy link

@penguin359 penguin359 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a few comments and one issue I see with the code changes.

make.go Outdated Show resolved Hide resolved
make.go Outdated Show resolved Hide resolved
template.go Show resolved Hide resolved
@ottok
Copy link
Contributor Author

ottok commented Jan 5, 2025

@penguin359 Would you like to test this PR?

Easy steps to do so:

cd $(mktemp -d)
curl -LO https://salsa.debian.org/otto/dh-make-golang/-/jobs/6866388/artifacts/raw/debian/output/dh-make-golang_0.7.0-1+salsaci+20250105+18_amd64.deb
apt install ./dh-make-golang_0.7.0-1+salsaci* 
dh-make-golang make -dep14 -git_revision v1.0.9 -type program -upstream_git_history -wrap-and-sort ast github.com/cilium/pwru

cd pwru

± git remote -v
origin	[email protected]:go-team/packages/pwru.git (fetch)
origin	[email protected]:go-team/packages/pwru.git (push)
upstreamvcs	https://github.com/cilium/pwru (fetch)
upstreamvcs	https://github.com/cilium/pwru (push)

± git branch -a
* debian/latest
  upstream/latest
  remotes/upstreamvcs/main
  ...

± cat debian/rules 
#!/usr/bin/make -f

%:
	dh $@ --builddirectory=debian/build --buildsystem=golang

override_dh_auto_install:
	dh_auto_install -- --no-source

± cat debian/gbp.conf 
[DEFAULT]
debian-branch = debian/latest
upstream-branch = upstream/latest
dist = DEP14

# Always use pristine tar to improve supply chain security and auditability
pristine-tar = True

# Lax requirement to use branch name 'debian/latest' so that git-buildpackage
# will always build using the currently checked out branch as the Debian branch.
# This makes it easier for contributors to work with feature and bugfix
# branches.
ignore-branch = True

# Configure the upstream tag format below, so that 'gbp import-orig' will run
# correctly, and link tarball import branch ('upstream/latest') with the
# equivalent upstream release tag, showing a complete audit trail of what
# upstream released and what was imported into Debian.
#
# Most Go packages have tags of form 'v1.0.0'
upstream-vcs-tag = v%(version%~%-)s

# If upstream publishes tarball signatures, git-buildpackage will by default
# import and use the them. Change this to 'on' to make 'gbp import-orig' abort
# if the signature is not found or is not valid.
#
# Most Go packages don't publish signatures for the tarball releases, so this is
# not enable by default.
#upstream-signatures = on

# Ensure the Debian maintainer signs git tags automatically
sign-tags = True


± uscan --report-status
...
uscan info: Filename (filenamemangled) for downloaded file: /cilium/pwru/archive/refs/tags/v1.0.10-pre.tar.gz
uscan: Newest version of pwru on remote site is 1.0.10~pre, local version is 1.0.9
uscan:    => Newer package available from
      https://github.com/cilium/pwru/archive/refs/tags/v1.0.10-pre.tar.gz

± git commit debian/ -m "WIP"

± gbp import-orig --uscan
--> does not work as there is no v1.0.10 release yet

@ottok
Copy link
Contributor Author

ottok commented Jan 5, 2025

I ran dh-make-golang make -dep14 -git_revision v1.0.8 -type program -upstream_git_history -wrap-and-sort ast github.com/cilium/pwru so I had 1.0.8 packaging draft, and then I was able to test the 1.0.9 import:

± gbp import-orig --uscan --upstream-version 1.0.9
gbp:info: Launching uscan...
gbp:info: Using uscan downloaded tarball ../pwru_1.0.9+ds1.orig.tar.xz
gbp:info: Importing '../pwru_1.0.9+ds1.orig.tar.xz' to branch 'upstream/latest'...
gbp:info: Source package is pwru
gbp:info: Upstream version is 1.0.9
gbp:info: Replacing upstream source on 'debian/latest'
gbp:info: Successfully imported version 1.0.9 of ../pwru_1.0.9.orig.tar.xz

± g-log 
*   cf1daee Update upstream source from tag 'upstream/1.0.9' 2 minutes ago (HEAD -> debian/latest)
|\  
| *   25262ac New upstream version 1.0.9 2 minutes ago (tag: upstream/1.0.9, upstream/latest)
| |\  
| | * aebb893 README.md: Update Clang/LLVM dependency to 12 3 months ago (tag: v1.0.9)
| | * 8a4dd24 build(deps): bump actions/checkout from 4.1.7 to 4.2.1 3 months ago
| | * 925df0e build(deps): bump golang.org/x/sys from 0.24.0 to 0.26.0 3 months ago
| | * c7c2ed7 build(deps): bump actions/upload-artifact from 4.3.6 to 4.4.3 3 months ago
| | * 82dbbf3 build(deps): bump golang.org/x/arch from 0.8.0 to 0.11.0 3 months ago

± git branch
* debian/latest
  pristine-tar
  upstream/latest

The import works correctly, including creating the pristine-tar branch and merging with full audit trail release git tag -> upstream/latest import branch -> copyright Filter application -> debian/latest.

template.go Outdated Show resolved Hide resolved
template.go Show resolved Hide resolved
Copy link

@penguin359 penguin359 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one stylistic nitpick, but otherwise - looks good to me!

make.go Outdated Show resolved Hide resolved
template.go Show resolved Hide resolved
@penguin359
Copy link

I ran through the test you provided. I did notice a few things, however, I suspect all of these are from upstream, not your PR. I'm just going to document it here for the moment.

One item I noticed was a big, fat warning from Git about the initial branch name defaulting to master because "init.defaultBranch" was not set in my container environment. Adding that as a -o option when you do git init would suppress that for any users who haven't set theirs yet.

I also saw a message about "Deleting upstream vendor/ directories", however, it appears that the vendor/ folder is still present in Git and the orig tarball so I don't know what that message is about.

Also, I haven't confirmed it yet, but it seems like it's signing upstream tags on uscan imports, but did not sign the initial tag of upstream import.

@ottok
Copy link
Contributor Author

ottok commented Jan 14, 2025

One item I noticed was a big, fat warning from Git about the initial branch name defaulting to master because "init.defaultBranch" was not set in my container environment. Adding that as a -o option when you do git init would suppress that for any users who haven't set theirs yet.

Yes indeed, git init emits that. I added now a commit to run git init -b debian/latest instead which had no errors:

$ git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /tmp/run/git2/.git/

->

$ git init -b debian/latest
Initialized empty Git repository in /tmp/run/git1/.git/

I actually did this by simply deleting code, as this solution was already envisioned by @elboulangero in 729d8db in 2021.

I also saw a message about "Deleting upstream vendor/ directories", however, it appears that the vendor/ folder is still present in Git and the orig tarball so I don't know what that message is about.

I am not sure what this is without a copy-paster of full output, but I assume this was git-buildpackage doing it on branch upstream/latest?

Also, I haven't confirmed it yet, but it seems like it's signing upstream tags on uscan imports, but did not sign the initial tag of upstream import.

Seems this code isn't running uscan at all:

arg := []string{
		"import-orig",
		"--no-interactive",
		"--debian-branch=" + debianBranch,
		"--upstream-branch=" + upstreamImportBranch,
	}
	if pristineTar {
		arg = append(arg, "--pristine-tar")
	}
	if includeUpstreamHistory {
		arg = append(arg, "--upstream-vcs-tag="+u.commitIsh)
	}
	arg = append(arg, filepath.Join(wd, orig))
	cmd := exec.Command("gbp", arg...)

->

gbp import-orig --no-interactive --debian-branch=debian/latest --upstream-branch=upstream/latest --pristine-tar --upstream-vcs-tag=v1.0.0

The final argument --uscan is missing from. We should have it yes, but it is out of scope for this DEP-14 rename PR, we should do it separately.

make.go Show resolved Hide resolved
ottok added 4 commits January 19, 2025 13:48
In DEP-14, the preferred branch name for the Debian packaging target
branch is `debian/latest` and the preferred name for the upstream import
target branch is `upstream/latest`.

Note that the upstream development branch name can be whatever and should
stay as it is upstream, typically `main` or `master`. The branch
`upstream/latest` should not point to the latest upstream development
commit, but to the latest commit that was used as the upstream release
that the Debian revision was derived from.
Instead of using various different upstream remote names, use the one and
same upstream git remote name consistently. As the name pick `upstreamvcs`
just as git-buildpackage does, so that if anybody runs `gbp clone` they
will automatically end up with the same git remotes and branches as anyone
in to go-team.
When creating a new package, populate the git-buildpackage with additional
configs and in-line comments on why and how to use them. This will make
go packaging easier, more consistent and more secure as the best practices
flow to all packages via good defaults.

Also add comment to explain why pristine-tar is beneficial.
Debian 11 "Bullseye" shipped with Git 2.30, so it is unlikely anybody
doing Debian packaging uses a Git version older than that anymore, at
least not in combination with this latest dh-make-golang version. Thus
it is possible to activate the shorthand git init form as planned in 2021
since 729d8db.
@ottok
Copy link
Contributor Author

ottok commented Jan 19, 2025

Two weeks passed since I pushed new version and responded to comments. I marked the items I fixed now as resolved as well and will proceed to merge this soon.

@ottok ottok merged commit acfef1a into Debian:master Jan 19, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants