Merge branch 'master' into ao-fix-backing-filtering-on-disputes

* master: (167 commits)
  Upgrade accidentally downgraded deps (#5365)
  [Pools] Fix issues with member migration to `DelegateStake` (#4822)
  Unify `no_genesis` check (#5360)
  [CI] Fix prdoc command (#5358)
  Beefy: add benchmarks for `report_fork_voting()` (#5188)
  Fix OurViewChange small race (#5356)
  Make ticket non-optional and add ensure_successful method to Consideration trait (#5359)
  [tests] dedup test code, add more tests, improve naming and docs (#5338)
  Stop running the wishlist workflow on forks (#5297)
  Migrate foreign assets v3::Location to v4::Location (#4129)
  Minor clean up (#5284)
  [Pools] Ensure members can always exit the pool gracefully (#4998)
  StorageWeightReclaim: set to node pov size if higher (#5281)
  [Bot] Add prdoc generation (#5331)
  Small nits found accidentally along the way (#5341)
  Create subsystem-benchmarks.yml (#5325)
  Bump libp2p-identity from 0.2.8 to 0.2.9 (#5232)
  Bump authoring duration for async backing to 2s. (#5195)
  Fix spelling issues (#5206)
  Bump the known_good_semver group across 1 directory with 3 updates (#5315)
  ...

.config/zepter.yaml

@@ -27,7 +27,7 @@ workflows:
       ]
   # The umbrella crate uses more features, so we to check those too:
   check_umbrella:
-    - [ $check.0, '--features=serde,experimental,with-tracing,tuples-96,with-tracing', '-p=polkadot-sdk' ]
+    - [ $check.0, '--features=serde,experimental,runtime,with-tracing,tuples-96,with-tracing', '-p=polkadot-sdk' ]
   # Same as `check_*`, but with the `--fix` flag.
   default:
     - [ $check.0, '--fix' ]

.github/commands-readme.md

@@ -10,6 +10,7 @@ The current available command actions are:
 
 - [Command FMT](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-fmt.yml)
 - [Command Update UI](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-update-ui.yml)
+- [Command Prdoc](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-prdoc.yml)
 - [Command Sync](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-sync.yml)
 - [Command Bench](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-bench.yml)
 - [Command Bench All](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-bench-all.yml)
@@ -25,12 +26,36 @@ Each command will have the same two required values, but it could have more.
 
 GitHub's official documentation: [Manually running a workflow](https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow)
 
+#### Running from CLI
+
+You can use [`gh cli`](https://cli.github.com/) to run the commands too. Refers to the [`gh workflow run`](https://cli.github.com/manual/gh_workflow_run) section from the documentation for more information.
+
 ### Number of the Pull Request
 
 The number of the pull request. Required so the action can fetch the correct branch and comment if it fails.
 
 ## Action configurations
 
+### FMT
+
+For FMT you only need the PR number.
+
+You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
+
+```bash
+gh workflow run command-fmt.yml -f pr=1000
+```
+
+### Update UI
+
+For Update UI you only need the PR number.
+
+You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
+
+```bash
+gh workflow run command-update-ui.yml -f pr=1000
+```
+
 ### Bench
 
 Runs `benchmark pallet` or `benchmark overhead` against your PR and commits back updated weights.
@@ -136,6 +161,12 @@ Posible combinations based on the `benchmark` dropdown.
   - Requires `Runtime Dir` to be `testing`
   - Requires `Target Directory` to be `cumulus`
 
+You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
+
+```bash
+gh workflow run command-bench.yml -f pr=1000 -f benchmark=polkadot-pallet -f subcommand=pallet -f runtime=rococo -f pallet=pallet_name -f target_dir=polkadot
+```
+
 ### Bench-all
 
 This is a wrapper to run `bench` for all pallets.
@@ -174,6 +205,12 @@ Posible combinations based on the `benchmark` dropdown.
     - `people-westend`
   - Requires `Target Directory` to be `cumulus`
 
+You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
+
+```bash
+gh workflow run command-bench-all.yml -f pr=1000 -f benchmark=pallet -f pallet=pallet_name -f target_dir=polkadot -f runtime=rococo
+```
+
 ### Bench-overhead
 
 Run benchmarks overhead and commit back results to PR.
@@ -193,6 +230,45 @@ Posible combinations based on the `benchmark` dropdown.
     - `asset-hub-westend`
   - Requires `Target directory` to be `cumulus`
 
+You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
+
+```bash
+gh workflow run command-bench-overheard.yml -f pr=1000 -f benchmark=substrate -f runtime=rococo -f target_dir=substrate
+```
+
+### PrDoc
+
+Generate a PrDoc with the crates populated by all modified crates.
+
+Options:
+- `pr`: The PR number to generate the PrDoc for.
+- `audience`: The audience of whom the changes may concern.
+- `bump`: A default bump level for all crates. The PrDoc will likely need to be edited to reflect the actual changes after generation.
+- `overwrite`: Whether to overwrite any existing PrDoc.
+
+### Sync
+
+Run sync and commit back results to PR.
+
+Posible combinations based on the `benchmark` dropdown.
+
+- `chain`
+  - Requires one of the following:
+    - `rococo`
+    - `westend`
+- `sync-type`
+  - Requires one of the following:
+    - `warp`
+    - `full`
+    - `fast`
+    - `fast-unsafe`
+
+You can use the following [`gh cli`](https://cli.github.com/) inside the repo:
+
+```bash
+gh workflow run command-sync.yml -f pr=1000 -f chain=rococo -f sync-type=full
+```
+
 ## How to modify an action
 
 If you want to modify an action and test it, you can do by simply pushing your changes and then selecting your branch in the `Use worflow from` option.

.github/dependabot.yml

@@ -5,13 +5,17 @@ updates:
     directory: '/'
     labels: ["A1-insubstantial", "R0-silent"]
     schedule:
-      interval: daily
+      interval: weekly
+    groups:
+      ci_dependencies:
+        patterns:
+        - "*"
   # Update Rust dependencies:
   - package-ecosystem: "cargo"
     directory: "/"
     labels: ["A1-insubstantial", "R0-silent"]
     schedule:
-      interval: "daily"
+      interval: "weekly"
     groups:
       # We assume these crates to be semver abiding and can therefore group them together.
       known_good_semver:

.github/scripts/common/lib.sh

@@ -315,6 +315,7 @@ function import_gpg_keys() {
     ) &
   done
   wait
+  gpg -k $SEC
 }
 
 # Check the GPG signature for a given binary
@@ -444,3 +445,28 @@ get_latest_release_tag() {
     latest_release_tag=$(curl -s -H "$TOKEN" $api_base/paritytech/polkadot-sdk/releases/latest | jq -r '.tag_name')
     printf $latest_release_tag
 }
+
+function get_polkadot_node_version_from_code() {
+   # list all the files with node version
+  git grep -e "\(NODE_VERSION[^=]*= \)\".*\"" |
+  # fetch only the one we need
+  grep  "primitives/src/lib.rs:" |
+  # Print only the version
+  awk '{ print $7 }' |
+  # Remove the quotes
+  sed 's/"//g' |
+  # Remove the semicolon
+  sed 's/;//g'
+}
+
+validate_stable_tag() {
+    tag="$1"
+    pattern='^stable[0-9]+(-[0-9]+)?$'
+
+    if [[ $tag =~ $pattern ]]; then
+        echo $tag
+    else
+        echo "The input '$tag' does not match the pattern."
+        exit 1
+    fi
+}

.github/scripts/generate-prdoc.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+
+"""
+Generate the PrDoc for a Pull Request with a specific number, audience and bump level.
+
+It downloads and parses the patch from the GitHub API to opulate the prdoc with all modified crates.
+This will delete any prdoc that already exists for the PR if `--force` is passed.
+
+Usage:
+	python generate-prdoc.py --pr 1234 --audience "TODO" --bump "TODO"
+"""
+
+import argparse
+import os
+import re
+import sys
+import subprocess
+import toml
+import yaml
+import requests
+
+from github import Github
+import whatthepatch
+from cargo_workspace import Workspace
+
+# Download the patch and pass the info into `create_prdoc`.
+def from_pr_number(n, audience, bump, force):
+	print(f"Fetching PR '{n}' from GitHub")
+	g = Github()
+
+	repo = g.get_repo("paritytech/polkadot-sdk")
+	pr = repo.get_pull(n)
+
+	patch_url = pr.patch_url
+	patch = requests.get(patch_url).text
+
+	create_prdoc(n, audience, pr.title, pr.body, patch, bump, force)
+
+def create_prdoc(pr, audience, title, description, patch, bump, force):
+	path = f"prdoc/pr_{pr}.prdoc"
+
+	if os.path.exists(path):
+		if force == True:
+			print(f"Overwriting existing PrDoc for PR {pr}")
+		else:
+			print(f"PrDoc already exists for PR {pr}. Use --force to overwrite.")
+			sys.exit(1)
+	else:
+		print(f"No preexisting PrDoc for PR {pr}")
+
+	prdoc = { "doc": [{}], "crates": [] }
+
+	prdoc["title"] = title
+	prdoc["doc"][0]["audience"] = audience
+	prdoc["doc"][0]["description"] = description
+
+	workspace = Workspace.from_path(".")
+
+	modified_paths = []
+	for diff in whatthepatch.parse_patch(patch):
+		modified_paths.append(diff.header.new_path)
+
+	modified_crates = {}
+	for p in modified_paths:
+		# Go up until we find a Cargo.toml
+		p = os.path.join(workspace.path, p)
+		while not os.path.exists(os.path.join(p, "Cargo.toml")):
+			p = os.path.dirname(p)
+
+		with open(os.path.join(p, "Cargo.toml")) as f:
+			manifest = toml.load(f)
+
+		if not "package" in manifest:
+			print(f"File was not in any crate: {p}")
+			continue
+
+		crate_name = manifest["package"]["name"]
+		if workspace.crate_by_name(crate_name).publish:
+			modified_crates[crate_name] = True
+		else:
+			print(f"Skipping unpublished crate: {crate_name}")
+
+	print(f"Modified crates: {modified_crates.keys()}")
+
+	for crate_name in modified_crates.keys():
+		entry = { "name": crate_name }
+
+		if bump == 'silent' or bump == 'ignore' or bump == 'no change':
+			entry["validate"] = False
+		else:
+			entry["bump"] = bump
+
+		print(f"Adding crate {entry}")
+		prdoc["crates"].append(entry)
+
+	# write the parsed PR documentation back to the file
+	with open(path, "w") as f:
+		yaml.dump(prdoc, f)
+
+def parse_args():
+	parser = argparse.ArgumentParser()
+	parser.add_argument("--pr", type=int, required=True)
+	parser.add_argument("--audience", type=str, default="TODO")
+	parser.add_argument("--bump", type=str, default="TODO")
+	parser.add_argument("--force", type=str)
+	return parser.parse_args()
+
+if __name__ == "__main__":
+	args = parse_args()
+	force = True if args.force.lower() == "true" else False
+	print(f"Args: {args}, force: {force}")
+	from_pr_number(args.pr, args.audience, args.bump, force)

.github/scripts/update-wishlist-leaderboard.py

@@ -0,0 +1,79 @@
+from github import Github
+import re
+import os
+from datetime import date
+
+g = Github(os.getenv("GH_TOKEN"))
+
+# Regex pattern to match wish format:
+wish_pattern = re.compile(
+    r"I wish for:? (https://github\.com/([a-zA-Z0-9_.-]+)/([a-zA-Z0-9_.-]+)/(issues|pull)/(\d+))"
+)
+
+wishlist_issue = g.get_repo(os.getenv("WISHLIST_REPOSITORY")).get_issue(
+    int(os.getenv("WISHLIST_ISSUE_NUMBER"))
+)
+new_leaderboard = (
+    "| Feature Request | Summary | Votes | Status |\n| --- | --- | --- | --- |\n"
+)
+wishes = {}
+issue_details = {}
+
+for comment in wishlist_issue.get_comments():
+    # in the comment body, if there is a string `#(\d)`, replace it with
+    # https://github.com/paritytech/polkadot-sdk/issues/(number)
+    updated_body = re.sub(
+        r"#(\d+)", r"https://github.com/paritytech/polkadot-sdk/issues/\1", comment.body
+    )
+
+    matches = wish_pattern.findall(updated_body)
+    for match in matches:
+        url, org, repo_name, _, issue_id = match
+        issue_key = (url, org, repo_name, issue_id)
+        if issue_key not in wishes:
+            wishes[issue_key] = []
+
+        # Get the author and upvoters of the wish comment.
+        wishes[issue_key].append(comment.user.id)
+        wishes[issue_key].extend(
+            [
+                reaction.user.id
+                for reaction in comment.get_reactions()
+                if reaction.content in ["+1", "heart", "rocket"]
+            ]
+        )
+
+        # Get upvoters of the desired issue.
+        desired_issue = g.get_repo(f"{org}/{repo_name}").get_issue(int(issue_id))
+        wishes[issue_key].extend(
+            [
+                reaction.user.id
+                for reaction in desired_issue.get_reactions()
+                if reaction.content in ["+1", "heart", "rocket"]
+            ]
+        )
+        issue_details[url] = [
+            desired_issue.title,
+            "👾 Open" if desired_issue.state == "open" else "✅Closed",
+        ]
+
+# Count unique wishes - the author of the wish, upvoters of the wish, and upvoters of the desired issue.
+for key in wishes:
+    wishes[key] = len(list(set(wishes[key])))
+
+# Sort wishes by count and add to the markdown table
+sorted_wishes = sorted(wishes.items(), key=lambda x: x[1], reverse=True)
+for (url, _, _, _), count in sorted_wishes:
+    [summary, status] = issue_details.get(url, "No summary available")
+    new_leaderboard += f"| {url} | {summary} | {count} | {status} |\n"
+new_leaderboard += f"\n> Last updated: {date.today().strftime('%Y-%m-%d')}\n"
+print(new_leaderboard)
+
+new_content = re.sub(
+    r"(\| Feature Request \|)(.*?)(> Last updated:)(.*?\n)",
+    new_leaderboard,
+    wishlist_issue.body,
+    flags=re.DOTALL,
+)
+
+wishlist_issue.edit(body=new_content)