Fix vfs.ls with access_credentials_name

[johnkerl]

Overview

When attempting to use the run_ingest_workflow function to ingest H5AD files into SOMA, users are expected to provide AWS credentials through the extra_tiledb_config argument. This is a departure from other 1-line ingestors, which allow users to provide only an access_credential_name (or a config) referencing their role-based credential.

The function fails if AWS credentials are not defined in the local environment or passed through the extra_tiledb_config argument, even when access_credential_name is provided because a tiledb.VFS instance is created locally to determine whether the input_uri points to a file or directory.

Why this is

• run_ingest_workflow runs on the client
• We need vfs in order to:
  • Decide if the input URI is a leaf (.h5ad) or not (a prefix)
  • In the latter case, to enumerate over the .h5ad leaves at that prefix

Sample script

stamp=$(vdatetime)
prefix="s3://tiledb-johnkerl/s/a/stack-small"
scratch="tiledb://johnkerl-tiledb/s3://tiledb-johnkerl/scratch"
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_DEFAULT_REGION
./clingt.py $prefix $scratch/csi-ing-b-prefix-env-no-$stamp

clingt.py

#!/usr/bin/env python

import tiledb.cloud
import tiledb.cloud.soma
import datetime
import sys
import os

input_uri        = "s3://tiledb-johnkerl/s/a/stack-small"
stamp            = datetime.datetime.today().strftime('%Y%m%d-%H%M%S')
output_uri       = f"tiledb://johnkerl-tiledb/s3://tiledb-johnkerl/scratch/csi-{stamp}"
measurement_name = "RNA"
namespace        = "TileDB-Inc"
dry_run          = False

args = sys.argv[1:]

if len(args) >= 1 and args[0] == "-n":
    dry_run = True
    args = args[1:]

if len(args) == 1:
    input_uri  = args[0]
if len(args) == 2:
    input_uri  = args[0]
    output_uri = args[1]

print(input_uri)
print(output_uri)

dikt = tiledb.cloud.soma.run_ingest_workflow(
    output_uri=output_uri,
    input_uri=input_uri,
    measurement_name=measurement_name,
    namespace=namespace,
    access_credentials_name="tiledb-cloud-sandbox-role",
    resources={"cpu": "8", "memory": "32Gi"},
    extra_tiledb_config={"config.logging_level": "5"},
    dry_run=dry_run,
)
print(dikt)
print("https://cloud.tiledb.com/activity/taskgraphs/TileDB-Inc/" + dikt["graph_id"])

Expected behavior

$ aws s3 ls s3://tiledb-johnkerl/s/a/stack-small/
2023-08-11 15:37:15      20200 stack1.h5ad
2023-08-11 15:37:15      20200 stack2.h5ad
2023-08-11 15:37:15      20200 stack3.h5ad
2023-08-11 15:37:15      20200 stack4.h5ad

Since the prefix provided to clingt.py is s3://tiledb-johnkerl/s/a/stack-small, I expect one enumerator node to vfs.ls that prefix, and four leaves to be launched, one for each .h5ad file at that prefix.

Sample logs

The first one is user-visible, from clingt.py, and the second one is linked to from there:

• https://cloud.tiledb.com/compute/logs/taskgraphs/TileDB-Inc/725e8c29-3786-4c57-875e-bca451de5982
• https://cloud.tiledb.com/compute/logs/tasks/aca90987-2a52-47b1-a6a6-5c9d59312d2d

The next one (the enumerator node) can't be found from the ones above. The user has to discover it by searching compute logs :(. The first one is the enumerator (four-point candelabra); the ones after are the four leaves:

• https://cloud.tiledb.com/compute/logs/taskgraphs/TileDB-Inc/77b00b2d-1928-489e-a971-edbccdfa24f6
• https://cloud.tiledb.com/compute/logs/tasks/beaad255-3404-4c84-b8c3-c7d31c529ffe
• https://cloud.tiledb.com/compute/logs/tasks/b4a41857-507a-4506-a38e-c438cd318aaa
• https://cloud.tiledb.com/compute/logs/tasks/2f19b6c5-e112-4118-98b0-f8d4030d061d
• https://cloud.tiledb.com/compute/logs/tasks/2f19b6c5-e112-4118-98b0-f8d4030d061d

See also

[sc-35137]

[shortcut-integration]

This pull request has been linked to Shortcut Story #35137: 1-line SOMA ingest should only require an access credential name.

[johnkerl]

@thetorpedodog ping 🙏

[johnkerl]

@thetorpedodog thank you for punctuational review.

What I really need help with is the two big-picture questions I wrote about in the description field.

Thanks in advance for any help you can offer. I am in need of expert guidance here.

[thetorpedodog]

I guess the question I have at this stage is: why does the setup task run on the client side? It seems like if that were run on the server side, that would solve the problem.

[johnkerl]

I guess the question I have at this stage is: why does the setup task run on the client side? It seems like if that were run on the server side, that would solve the problem.

This PR has the setup task run server-side as described in the description section of this PR.

[thetorpedodog]

Coming back to this after way too long, I think I have a solution here. The change that needs to happen is in the run_ingest_workflow function.

Right now it starts the one-node graph to build the actual ingestion graph, but it returns the ID of the one-node graph.

Instead, we need to wait for that one-node graph to complete (i.e., for it to build and launch the actual ingestion graph). Then, the one-node graph will return the ID of the actual ingestion graph, and the run_ingest_workflow function can look the output of that node, which contains the graph ID, and return that to the user:

def run_ingest_workflow(...) -> Dict[str, str]:
    ...
    grf = build_ingest_workflow_graph(...)
    grf.compute()
    the_node = next(iter(grf.nodes.values()))
    real_graph_uuid = the_node.result()
    return {
        "status": "started",
        "graph_id": str(real_graph_uuid),
    }

[johnkerl]

@thetorpedodog ready for next round of review!

[thetorpedodog]

A few minor things, otherwise this looks pretty good!

[johnkerl]

@thetorpedodog ready for the next round of review!

[johnkerl]

@thetorpedodog ping 🙏

[thetorpedodog]

no need for str here; %s transforms everything into a string.