Support packages in subdirs

[christopher-dG]

Starting from Julia 1.5, packages are no longer required to be rooted at the Git repo root.

A couple of things to consider here:

Currently we just look for Project.toml or JuliaProject.toml at the repo root, but now we need to look everywhere for those files.
We can still do that without a Git clone by looking through the Git tree:

head = repo.get_commit("HEAD")
tree = repo.get_git_tree(head.commit.tree.sha, recursive=True)
projects = []
for el in tree.tree:
    parts = el.path.split("/")
    if el.type == "blob" and parts[-1] in ["Project.toml", "JuliaProject.toml"]:
        project = toml.loads(repo.get_contents(el.path).decoded_content.decode())
        if "name" in project and "uuid" in project:
            projects.append({"name": project["name"], "uuid": project["uuid"], "path": "/".join(parts[:-1]}))
return projects

Then we need to check for new versions and make tags for all of these projects.

Conversion of Git tree SHA to commit SHA will have to change, because we're currently looking at each commit's tree SHA which is computed from the root. We'll have to look through the tree for each commit:

for commit in repo.get_commits():
    sha = commit.commit.tree.sha
    if sha == expected:  # Happy path, package is at repo root
        return commit.sha
    tree = repo.get_git_tree(sha, recursive=True)
    for el in tree.tree:
        if el.path == project["path"]:
            if el.sha == expected:
                return commit.sha
            else:  # This commit isn't it, skip to next
                break
return None  # Commit not found

We also need to name tags differently if there are multiple packages, maybe we can just expose a config option for this. We can probably do some nifty templating in the default to get vVERSION for packages at the repo root and PACKAGE-vVERSION otherwise.

[fredrikekre]

We also need to name tags differently if there are multiple packages, maybe we can just expose a config option for this. We can probably do some nifty templating in the default to get vVERSION for packages at the repo root and PACKAGE-vVERSION otherwise.

I think it would be good to standardize it. Tools like Documenter etc. look for tags with specific formats, cf. JuliaRegistries/Registrator.jl#230 (comment) and below.

[DilumAluthge]

Might as well choose ${NAME}-${VERSION} as the standard tag format.

[GunnarFarneback]

Currently we just look for Project.toml or JuliaProject.toml at the repo root, but now we need to look everywhere for those files.

Isn't the registry information available? Package.toml has a field subdir which tells where a subdir package is located.

[christopher-dG]

This is kind of sufficient, but we're currently not using the issue comments that trigger TagBot as an actual source of information so when TagBot runs it still needs to "discover" the Julia packages in the repo. If we changed the trigger comments to contain the UUID of the package it was triggering for, then yes we'd immediately know which package we should run for. But we previously declined to do that because of security concerns. I feel like if we only used a package UUID, there's no room for exploitation, but that needs some more thought.

[DilumAluthge]

I like the idea of not using the issue comment as a source of information.

You could iterate over all packages in the General registry and look until you find one where the repo field in Package.toml matches the URL of the repo. Even though it is a subdir package, the URL of the GitHub repo should still match the repo field in Package.toml.

[christopher-dG]

Might as well choose ${NAME}-${VERSION} as the standard tag format.

Forgot to address this: I personally would really prefer to only apply this format on subdir packages, I dislike the idea of all my tags no longer being "standard" semver format (on repos where a single Julia package is the only content). But if there's consensus otherwise, then I don't mind I suppose.

You could iterate over all packages in the General registry

I really like this strategy, it basically inverts my thought process on how it should work but I think it ought to work really well.

Coincidentally I'm working on #173 this week, perhaps I can sneak in some time to address this once I've re-familiarized myself with the code.

[DilumAluthge]

Might as well choose ${NAME}-${VERSION} as the standard tag format.

Forgot to address this: I personally would really prefer to only apply this format on subdir packages, I dislike the idea of all my tags no longer being "standard" semver format (on repos where a single Julia package is the only content). But if there's consensus otherwise, then I don't mind I suppose.

Oh, I should clarify. I meant that we would choose ${NAME}-${VERSION} as the standard tag format only for subdir packages. We would still use the current format for non-subdir packages.

[DilumAluthge]

I don't really see the point in having ${NAME} in the tag if it is not a subdir package.

[ericphanson]

You could iterate over all packages in the General registry and look until you find one where the repo field in Package.toml matches the URL of the repo. Even though it is a subdir package, the URL of the GitHub repo should still match the repo field in Package.toml.

Just to be sure, would this still work if there are two packages in the same repo, in different subdirs? (Eg SnoopCompile and SnoopCompileCore)

[DilumAluthge]

I guess you would:

1. Loop over all packages in the General registry, and make a list of all packages for which the repo field in the package's Package.toml file matches the URL of the GitHub repository on which TagBot is currently being run.
2. For each package in the list from step 1, create all of the tags for that package.

[hannahilea]

PR implementing individual subdir package support: #252