PEP 458: RSTUF Integration #13943

kairoaraujo · 2023-06-15T05:24:04Z

This PR implements PEP 458 by adding a setup for Repository Service for TUF (RSTUF) and connecting Warehouse to it.

Context

Unlike previous attempts (#7488, #10870) to implement PEP 458, this PR does not deeply integrate TUF repository management into Warehouse, but instead configures a stand-alone RSTUF service, which maintains the TUF metadata repository as a black box, and which accepts calls to a REST API, so that Warehouse can indicate addition or removal of release files, and trigger TUF metadata changes.

Additionally, RSTUF provides a CLI for root signing in order to initialize the RSTUF metadata repository.

See RSTUF docs for details.

Description of this PR

Configure RSTUF (development instance)
- Add development dependencies
- Add RSTUF worker services to docker-compose.yml
  - RSTUF uses the same Redis Server, but unique Redis DB ids 1 and 2
  - RSTUF uses the same PostgreSQL, but a specific rstuf database
- Add a pre-generated bootstrap file, with RSTUF config and root of trust
- Add RSTUF commands Makefile
  - make tufinit to bootstrap the RSTUF service
  - make tufimport to create TUF metadata for all existing release files from the example Warehouse database
    it is a make command to the added Warehouse cli command (warehouse tuf dev import-all)
Add calls to RSTUF API upon package addition and removal

Status of RSTUF

RSTUF is close to releasing a beta version, called Minimum Working Version (MWV). Actually, two of three components (RSTUF Worker and API) are already tagged MWV. The third component (RSTUF CLI) is missing one feature, which is not relevant for this PR, and not expected to break compatibility for the MWV release.

kairoaraujo · 2023-06-15T06:55:28Z

Open questions / Next steps

How to test
Documentation about how to roll out

kairoaraujo · 2023-06-15T06:56:11Z

warehouse/packaging/tasks.py

@@ -508,3 +509,9 @@ def populate_data_using_schema(file):
                json_rows, table_name, job_config=LoadJobConfig(schema=table_schema)
            ).result()
            break
+
+
+@tasks.task(ignore_result=True, acks_late=True)


About this task

I need it to have the request object to be used by the render_simple_detail function.
I couldn't find a way to retrieve the request , only using tasks.
You can see it is used during the process of the warehouse tuf dev import-all command https://github.com/kairoaraujo/warehouse/blob/706478ac2f375e6f2d03259ee75cba67c91f21ae/warehouse/cli/tuf.py#L136

I also intended to use it later during the Manage packages/project and simple details to TUF commit. Still, it wasn't possible because I could not send the simple detail generated in the background. I need the response to use in the TUF metadata.

If I can retrieve the request object or generate one for use with the render_simple_detail, we can remove it.

kairoaraujo · 2023-06-15T07:10:59Z

warehouse/tuf/targets.py

+    if response.status_code != 202:
+        raise HTTPBadGateway(f"Unexpected TUF Server response: {response.text}")
+
+    return response.json()


RSTUF, by design, is asynchronous.
Every request to add or remove an artifact in TUF metadata generates a task id.
With this task id is possible to do call backs to receive the status. RSTUF gives the control to monitor this task to the requester.

Some integration design questions about Warehouse

Should the Warehouse create an asynchronous monitoring/action for the tasks to take some action depending on the result?

Should Warehouse store the task result in some Warehouse table as in RSTUF use the default [Celery result expires]? Currently, we store the task id in the events, but the task result is not persistent as mentioned, for example:

warehouse/warehouse/forklift/legacy.py

Lines 1410 to 1427 in 818210a

task = targets.add_file(request, project, file_)

file_.record_event(

tag=EventTag.File.FileAdd,

request=request,

additional={

"filename": file_.filename,

"submitted_by": request.user.username

if request.user

else "OpenID created token",

"canonical_version": release.canonical_version,

"publisher_url": request.oidc_publisher.publisher_url(

request.oidc_claims

)

if request.oidc_publisher

else None,

"project_id": str(project.id),

"tuf": task["data"]["task_id"],

kairoaraujo · 2023-06-15T07:16:05Z

I'm tagging here some TUF folks.

@lukpueh -- Thanks for pre-draft-PR review ❤️
@joshuagl -- It's following the RSTUF work from the very beginning

Note: Feel free to tag others

lukpueh · 2023-06-15T07:30:31Z

cc @JustinCappos @mnm678 @adityasaky @trishankatdatadog @brainwane

simi · 2023-07-28T15:02:03Z

warehouse/config.py

@@ -241,6 +241,12 @@ def configure(settings=None):
        coercer=int,
        default=100,
    )
+    maybe_set(settings, "tuf.database.url", "TUF_DATABASE_URL")
+    maybe_set(settings, "tuf.metadata.url", "TUF_METADATA_URL")
+    maybe_set(settings, "tuf.api.url", "TUF_API_URL")


Would it make sense to rename this to RSTUF_API_URL since RSTUF is providing the API in here?

Adds the RSTUF in the Warehouse infrastructure * Include the RSTUF Ceremony payload file - It is generated using `rstuf admin ceremony`, and the keys * Add the development dependencies - RSTUF CLI and dependencies * Include RSTUF components to the `docker-compose.yml` - RSTUF uses the same Redis Server but uses unique Redis DB ids `1` and `2` - RSTUF uses the same PostgreSQL, but a specific database rstuf * Add the RSTUF environment configuration for development * Define the Makefile commands for RSTUF - `make tufinit` to bootstrap the RSTUF service - `make tufimport` to import all project packages to the RSTUF service * Define the basic commands for RSTUF within Warehouse - Command to import all existent packages and indexes to TUF metadata (`warehouse tuf dev import-all`) * Add TUF development documentation Signed-off-by: Kairo de Araujo <[email protected]>

* Adding packages After adding a package to the Warehouse database, it generates and stores the Simple Index with a request to the RSTUF backend to include the package and its simple index in TUF Metadata. * Removing package or Project Release On PyPI Management, when a user removes a file or a project release it also removes it from TUF metadata and updates the simple details index. Co-authored-by: Lukas Puehringer <[email protected]> Signed-off-by: Kairo de Araujo <[email protected]> simplify code in warehouse.tuf.targets Signed-off-by: Kairo de Araujo <[email protected]>

Reduce the number of delegated hash-bin roles for the development enviroment. Signed-off-by: Kairo de Araujo <[email protected]>

Rename the environment variable setting `TUF_API_URL` to `RSTUF_API_URL` as this API is provided by Repository Service for TUF (RSTUF). Signed-off-by: Kairo de Araujo <[email protected]>

ewdurbin · 2024-01-17T16:25:18Z

Thanks for this PR @kairoaraujo.

I think to make progress, we would want to deliver chunks of this iteratively rather than all at once.

Break out a separate PR that introduces a development mode rstuf container even if nothing talks to it... ideally that can be treated as much like a "blackbox" as possible. The current setup with api/worker/worker adds a lot of complexity the makes review difficult. This PR could also remove the un-used vault service in dev intended for the original tuf implementation.
A follow on PR that demonstrates integration with the rstuf service locally. This will help in review so that we can focus on warehouse's concerns rather than RSTUFs.

miketheman · 2024-01-31T15:37:54Z

Superseded by #15241

kairoaraujo commented Jun 15, 2023

View reviewed changes

miketheman added the security Security-related issues and pull requests label Jun 28, 2023

simi reviewed Jul 28, 2023

View reviewed changes

kairoaraujo force-pushed the rstuf_integration branch from 818210a to 5751be8 Compare July 29, 2023 10:25

Kairo de Araujo added 4 commits October 2, 2023 16:48

Reduce the number of delegated hash-bin dev

2f52784

Reduce the number of delegated hash-bin roles for the development enviroment. Signed-off-by: Kairo de Araujo <[email protected]>

Rename the TUF_API_URL to RSTUF_API_URL

b86baea

Rename the environment variable setting `TUF_API_URL` to `RSTUF_API_URL` as this API is provided by Repository Service for TUF (RSTUF). Signed-off-by: Kairo de Araujo <[email protected]>

kairoaraujo force-pushed the rstuf_integration branch from 5751be8 to b86baea Compare October 2, 2023 14:49

trishankatdatadog mentioned this pull request Nov 8, 2023

TUF Initialization using python-tuf 2.0.0 #10870

Closed

3 tasks

kairoaraujo mentioned this pull request Jan 25, 2024

PEP458: Add minimal RSTUF service configuration #15241

Merged

miketheman closed this Jan 31, 2024

lukpueh mentioned this pull request Feb 5, 2024

[WIP] warehouse: TUF initialization #7488

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 458: RSTUF Integration #13943

PEP 458: RSTUF Integration #13943

kairoaraujo commented Jun 15, 2023 •

edited

Loading

kairoaraujo commented Jun 15, 2023

kairoaraujo Jun 15, 2023

kairoaraujo Jun 15, 2023

kairoaraujo commented Jun 15, 2023

lukpueh commented Jun 15, 2023

simi Jul 28, 2023

ewdurbin commented Jan 17, 2024

miketheman commented Jan 31, 2024

	task = targets.add_file(request, project, file_)

	file_.record_event(
	tag=EventTag.File.FileAdd,
	request=request,
	additional={
	"filename": file_.filename,
	"submitted_by": request.user.username
	if request.user
	else "OpenID created token",
	"canonical_version": release.canonical_version,
	"publisher_url": request.oidc_publisher.publisher_url(
	request.oidc_claims
	)
	if request.oidc_publisher
	else None,
	"project_id": str(project.id),
	"tuf": task["data"]["task_id"],

PEP 458: RSTUF Integration #13943

PEP 458: RSTUF Integration #13943

Conversation

kairoaraujo commented Jun 15, 2023 • edited Loading

Context

Description of this PR

Status of RSTUF

kairoaraujo commented Jun 15, 2023

Open questions / Next steps

kairoaraujo Jun 15, 2023

Choose a reason for hiding this comment

kairoaraujo Jun 15, 2023

Choose a reason for hiding this comment

kairoaraujo commented Jun 15, 2023

lukpueh commented Jun 15, 2023

simi Jul 28, 2023

Choose a reason for hiding this comment

ewdurbin commented Jan 17, 2024

miketheman commented Jan 31, 2024

kairoaraujo commented Jun 15, 2023 •

edited

Loading