azul3 on ξ issues/dsotirho-ucsc/6620-duos-id ο daniel2.local π v3.12.8 (.venv)
at 01/08 16:35:08 β141 β― make auto_deploy ; sleep 60 ; make integration_test ; git status ; git rev-parse HEAD ; date
/Library/Developer/CommandLineTools/usr/bin/make -C lambdas
/Library/Developer/CommandLineTools/usr/bin/make -C layer layer
python /Users/daniel/repo/azul3/scripts/check_branch.py
python .chalice/config.json.template.py .chalice/config.json
Creating .chalice/config.json
python /Users/daniel/repo/azul3/scripts/stage_layer.py
2025-01-08 16:35:10,963 INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:35:11,000 INFO MainThread azul.deployment: Allocated new Boto3 client for 'iam' with ID 4385729776
2025-01-08 16:35:11,455 INFO MainThread azul.deployment: Allocated new Boto3 client for 'sts' with ID 4384360832
2025-01-08 16:35:11,498 INFO MainThread azul.lambda_layer: Checking for dependencies layer package at s3://edu-ucsc-gi-platform-anvil-dev-shared.us-east-1/azul/daniel2/lambda_layers/510ee02b48ce4f187752da6094c23fbb420613aa.zip.
2025-01-08 16:35:11,539 INFO MainThread azul.deployment: Allocated new Boto3 client for 's3' with ID 4584590704
2025-01-08 16:35:11,642 INFO MainThread azul.boto3: s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-dev-shared.us-east-1/azul/daniel2/lambda_layers/510ee02b48ce4f187752da6094c23fbb420613aa.zip
2025-01-08 16:35:11,642 INFO MainThread azul.boto3: s3.HeadObject: β¦ without request body
2025-01-08 16:35:11,952 INFO MainThread azul.boto3: s3.HeadObject: Got 200 response
2025-01-08 16:35:11,952 INFO MainThread azul.boto3: s3.HeadObject: β¦ with response body b''
2025-01-08 16:35:11,953 INFO MainThread azul.lambda_layer: Layer package already up-to-date.
/Library/Developer/CommandLineTools/usr/bin/make -C indexer package
python /Users/daniel/repo/azul3/scripts/check_branch.py
python .chalice/config.json.template.py .chalice/config.json
[INFO] 2025-01-08T16:35:13.883Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in shared credentials file: ~/.aws/credentials
[INFO] 2025-01-08T16:35:13.918Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'iam' with ID 4489269072
Creating .chalice/config.json
python vendor/resources/environ.json.template.py vendor/resources/environ.json
Creating vendor/resources/environ.json
PYTHONHASHSEED=0 python -m compileall \
-f -q --invalidation-mode checked-hash \
vendor vendor/humancellatlas vendor/azul
python -m azul.changelog vendor
chalice package --stage daniel2 --pkg-format terraform .chalice/terraform
[INFO] 2025-01-08T16:35:15.645Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in shared credentials file: ~/.aws/credentials
[INFO] 2025-01-08T16:35:15.680Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'iam' with ID 4401669344
Creating deployment package.
Reusing existing deployment package.
[INFO] 2025-01-08T16:35:16.103Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in environment variables.
/Library/Developer/CommandLineTools/usr/bin/make -C service package
python /Users/daniel/repo/azul3/scripts/check_branch.py
python .chalice/config.json.template.py .chalice/config.json
[INFO] 2025-01-08T16:35:18.006Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in shared credentials file: ~/.aws/credentials
[INFO] 2025-01-08T16:35:18.043Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'iam' with ID 4404669152
Creating .chalice/config.json
python vendor/resources/environ.json.template.py vendor/resources/environ.json
Creating vendor/resources/environ.json
PYTHONHASHSEED=0 python -m compileall \
-f -q --invalidation-mode checked-hash \
vendor vendor/humancellatlas vendor/azul
python -m azul.changelog vendor
chalice package --stage daniel2 --pkg-format terraform .chalice/terraform
Creating deployment package.
Reusing existing deployment package.
[INFO] 2025-01-08T16:35:19.509Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in environment variables.
/Library/Developer/CommandLineTools/usr/bin/make -C terraform auto_apply
python /Users/daniel/repo/azul3/scripts/check_branch.py
set -o pipefail && git ls-files --ignored --others --directory --exclude-standard | (grep -v '/[^/]' || test $? -eq 1) | xargs -r rm -rv
__pycache__//cloudwatch_dashboard.template.json.cpython-312.pyc
__pycache__/
api_gateway.tf.json
authentication.tf.json
backend.tf.json
cloudwatch.tf.json
common.tf.json
dynamo.tf.json
lambda_layer.tf.json
plan.bin
plan.json
providers.tf.json
s3.tf.json
sqs.tf.json
step_function.tf.json
python providers.tf.json.template.py providers.tf.json
Creating providers.tf.json
python backend.tf.json.template.py backend.tf.json
Creating backend.tf.json
terraform init -reconfigure -lockfile=readonly
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/google from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of opensearch-project/opensearch from the dependency lock file
- Reusing previous version of hashicorp/external from the dependency lock file
- Using previously-installed hashicorp/null v3.2.0
- Using previously-installed hashicorp/google v4.58.0
- Using previously-installed hashicorp/aws v5.80.0
- Using previously-installed opensearch-project/opensearch v2.2.1
- Using previously-installed hashicorp/external v2.2.0
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
python /Users/daniel/repo/azul3/scripts/terraform_schema.py check \
|| (echo "Schema is stale. Run 'make update_schema' and commit." ; false)
2025-01-08 16:35:25,653 INFO MainThread azul.terraform: Running ['terraform', 'version', '-json']
2025-01-08 16:35:25,705 INFO MainThread azul.terraform: Terraform output:
{
"terraform_version": "1.9.8",
"platform": "darwin_arm64",
"provider_selections": {
"registry.terraform.io/hashicorp/aws": "5.80.0",
"registry.terraform.io/hashicorp/external": "2.2.0",
"registry.terraform.io/hashicorp/google": "4.58.0",
"registry.terraform.io/hashicorp/null": "3.2.0",
"registry.terraform.io/opensearch-project/opensearch": "2.2.1"
},
"terraform_outdated": true
}
python api_gateway.tf.json.template.py api_gateway.tf.json
[INFO] 2025-01-08T16:35:26.962Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'es' with ID 4431086176
Creating api_gateway.tf.json
python authentication.tf.json.template.py authentication.tf.json
Creating authentication.tf.json
python cloudwatch.tf.json.template.py cloudwatch.tf.json
Creating cloudwatch.tf.json
python common.tf.json.template.py common.tf.json
Creating common.tf.json
python dynamo.tf.json.template.py dynamo.tf.json
Creating dynamo.tf.json
python elasticsearch.tf.json.template.py elasticsearch.tf.json
python lambda_layer.tf.json.template.py lambda_layer.tf.json
Creating lambda_layer.tf.json
python s3.tf.json.template.py s3.tf.json
Creating s3.tf.json
python sqs.tf.json.template.py sqs.tf.json
Creating sqs.tf.json
python step_function.tf.json.template.py step_function.tf.json
[INFO] 2025-01-08T16:35:33.165Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in shared credentials file: ~/.aws/credentials
[INFO] 2025-01-08T16:35:33.202Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'iam' with ID 4404343680
[INFO] 2025-01-08T16:35:33.604Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'sts' with ID 4430603248
Creating step_function.tf.json
terraform validate
Success! The configuration is valid.
python /Users/daniel/repo/azul3/scripts/rename_resources.py
2025-01-08 16:35:36,017 INFO MainThread azul.terraform: Running ['terraform', 'state', 'list']
terraform plan -no-color -out=plan.bin
null_resource.hmac_secret: Refreshing state... [id=5577006791947779410]
data.external.elasticsearch_nodes: Reading...
data.aws_subnet.gitlab_public_1: Reading...
data.aws_subnet.gitlab_public_0: Reading...
data.aws_region.current: Reading...
aws_cloudwatch_event_rule.service_servicecachehealth: Refreshing state... [id=azul-servicecachehealth-daniel2]
aws_cloudwatch_log_group.service: Refreshing state... [id=/aws/apigateway/azul-service-daniel2]
aws_iam_role.states: Refreshing state... [id=azul-statemachine-daniel2]
aws_iam_role.service: Refreshing state... [id=azul-service-daniel2]
data.aws_region.current: Read complete after 0s [id=us-east-1]
aws_cloudwatch_log_group.waf_api_gateway: Refreshing state... [id=aws-waf-logs-azul-api_gateway-daniel2]
data.aws_subnet.gitlab_private_1: Reading...
data.aws_region.service: Reading...
data.aws_region.service: Read complete after 0s [id=us-east-1]
aws_sqs_queue.tallies_fail: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_fail-daniel2.fifo]
data.aws_wafv2_ip_set.allowed_v4_ips: Reading...
data.aws_subnet.gitlab_private_1: Read complete after 0s [id=subnet-029fa823faef9be96]
aws_kms_key.manifest: Refreshing state... [id=57f8a9db-d62d-4d34-bf71-dec9de4e5a68]
data.aws_subnet.gitlab_public_1: Read complete after 1s [id=subnet-02a80653dfdfd4280]
data.aws_s3_bucket.logs: Reading...
data.aws_subnet.gitlab_public_0: Read complete after 1s [id=subnet-0dfee4332c4f7383a]
data.aws_subnet.gitlab_private_0: Reading...
data.aws_vpc.gitlab: Reading...
data.aws_wafv2_ip_set.blocked_v4_ips: Reading...
data.aws_caller_identity.current: Reading...
data.aws_subnet.gitlab_private_0: Read complete after 0s [id=subnet-0410795269b2194be]
data.aws_partition.indexer: Reading...
data.aws_partition.indexer: Read complete after 0s [id=aws]
aws_s3_bucket.storage: Refreshing state... [id=edu-ucsc-gi-platform-anvil-dev-storage-daniel2.us-east-1]
data.aws_caller_identity.current: Read complete after 0s [id=289950828509]
data.aws_route53_zone.anvil_gi_ucsc_edu: Reading...
data.aws_caller_identity.indexer: Reading...
data.aws_region.indexer: Reading...
data.aws_region.indexer: Read complete after 0s [id=us-east-1]
aws_sqs_queue.notifications_fail: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_fail-daniel2]
data.aws_caller_identity.indexer: Read complete after 0s [id=289950828509]
data.aws_partition.service: Reading...
data.aws_partition.service: Read complete after 0s [id=aws]
aws_lambda_layer_version.dependencies: Refreshing state... [id=arn:aws:lambda:us-east-1:289950828509:layer:azul-dependencies-daniel2:8]
data.external.elasticsearch_nodes: Read complete after 1s [id=-]
aws_iam_role.indexer: Refreshing state... [id=azul-indexer-daniel2]
data.aws_caller_identity.service: Reading...
data.aws_s3_bucket.logs: Read complete after 0s [id=edu-ucsc-gi-platform-anvil-dev-logs.us-east-1]
aws_cloudwatch_log_group.indexer: Refreshing state... [id=/aws/apigateway/azul-indexer-daniel2]
data.aws_caller_identity.service: Read complete after 1s [id=289950828509]
aws_cloudwatch_event_rule.indexer_indexercachehealth: Refreshing state... [id=azul-indexercachehealth-daniel2]
aws_dynamodb_table.sources_cache_by_auth: Refreshing state... [id=azul-sources_cache_by_auth-daniel2]
data.aws_wafv2_ip_set.allowed_v4_ips: Read complete after 1s [id=cc183ccf-c615-4fb3-a4d3-a688e8855843]
aws_acm_certificate.service_0: Refreshing state... [id=arn:aws:acm:us-east-1:289950828509:certificate/f45019e8-e108-4dc6-bcd6-f0e2191f1f49]
data.aws_wafv2_ip_set.blocked_v4_ips: Read complete after 1s [id=e94dbda0-d614-4e1f-bc7a-8bab52c577b7]
aws_acm_certificate.indexer_0: Refreshing state... [id=arn:aws:acm:us-east-1:289950828509:certificate/85eeb04f-c530-49d9-9427-e655f010e33f]
google_project_iam_custom_role.azul: Refreshing state... [id=projects/platform-anvil-dev/roles/azul_daniel2_6]
data.google_client_config.current: Reading...
aws_sqs_queue.tallies_retry: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo]
data.google_client_config.current: Read complete after 0s [id=projects/platform-anvil-dev/regions//zones/]
aws_iam_role_policy.states: Refreshing state... [id=azul-statemachine-daniel2:azul-statemachine-daniel2]
data.aws_vpc.gitlab: Read complete after 1s [id=vpc-053fc6eb1daf602f9]
aws_sfn_state_machine.manifest: Refreshing state... [id=arn:aws:states:us-east-1:289950828509:stateMachine:azul-manifest-daniel2]
aws_sqs_queue.notifications_retry: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2]
aws_kms_alias.manifest: Refreshing state... [id=alias/azul-manifest-daniel2]
aws_security_group.indexer: Refreshing state... [id=sg-0c923c16c2eb6df98]
aws_security_group.service: Refreshing state... [id=sg-0d4e2784edc0c51ba]
data.aws_route53_zone.anvil_gi_ucsc_edu: Read complete after 1s [id=Z05898353G87FKJOLNP7N]
aws_cloudwatch_dashboard.dashboard: Refreshing state... [id=azul-dashboard-daniel2]
aws_sqs_queue.tallies: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo]
google_service_account.azul_public: Refreshing state... [id=projects/platform-anvil-dev/serviceAccounts/azul-ucsc-6-public-daniel2@platform-anvil-dev.iam.gserviceaccount.com]
google_service_account.azul_unregistered: Refreshing state... [id=projects/platform-anvil-dev/serviceAccounts/azul-ucsc-6-unreg-daniel2@platform-anvil-dev.iam.gserviceaccount.com]
google_service_account.azul: Refreshing state... [id=projects/platform-anvil-dev/serviceAccounts/azul-ucsc-6-daniel2@platform-anvil-dev.iam.gserviceaccount.com]
aws_sqs_queue.notifications: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2]
aws_lambda_function.indexer_aggregate_retry: Refreshing state... [id=azul-indexer-daniel2-aggregate_retry]
aws_lambda_function.indexer_contribute: Refreshing state... [id=azul-indexer-daniel2-contribute]
aws_lambda_function.indexer_indexercachehealth: Refreshing state... [id=azul-indexer-daniel2-indexercachehealth]
aws_lambda_function.indexer_aggregate: Refreshing state... [id=azul-indexer-daniel2-aggregate]
aws_lambda_function.indexer: Refreshing state... [id=azul-indexer-daniel2]
aws_wafv2_web_acl.api_gateway: Refreshing state... [id=e8d9c8ae-c8b0-479a-852d-eb64685fc1ed]
aws_lambda_function.indexer_contribute_retry: Refreshing state... [id=azul-indexer-daniel2-contribute_retry]
aws_lambda_function.service_servicecachehealth: Refreshing state... [id=azul-service-daniel2-servicecachehealth]
aws_lambda_function.service_manifest: Refreshing state... [id=azul-service-daniel2-manifest]
aws_lambda_function.service: Refreshing state... [id=azul-service-daniel2]
aws_route53_record.service_domain_validation_0: Refreshing state... [id=Z05898353G87FKJOLNP7N__b5c7cd26db92935ac3523b9e7386f0e4.service.daniel2.anvil.gi.ucsc.edu._CNAME]
aws_route53_record.indexer_domain_validation_0: Refreshing state... [id=Z05898353G87FKJOLNP7N__e069388fdcb911a9cd22fab225055e26.indexer.daniel2.anvil.gi.ucsc.edu._CNAME]
google_project_iam_member.azul: Refreshing state... [id=platform-anvil-dev/projects/platform-anvil-dev/roles/azul_daniel2_6/serviceAccount:azul-ucsc-6-daniel2@platform-anvil-dev.iam.gserviceaccount.com]
aws_lambda_event_source_mapping.indexer_aggregate_retry: Refreshing state... [id=efac04d4-a9cd-40a2-955e-d485ce373ea9]
aws_lambda_event_source_mapping.indexer_contribute: Refreshing state... [id=295b3db0-05b2-4040-bc24-85712e87f3a8]
aws_s3_bucket_public_access_block.storage: Refreshing state... [id=edu-ucsc-gi-platform-anvil-dev-storage-daniel2.us-east-1]
aws_s3_bucket_logging.storage: Refreshing state... [id=edu-ucsc-gi-platform-anvil-dev-storage-daniel2.us-east-1]
aws_s3_bucket_lifecycle_configuration.storage: Refreshing state... [id=edu-ucsc-gi-platform-anvil-dev-storage-daniel2.us-east-1]
aws_iam_role_policy.indexer: Refreshing state... [id=azul-indexer-daniel2:indexer]
aws_s3_bucket_inventory.storage: Refreshing state... [id=edu-ucsc-gi-platform-anvil-dev-storage-daniel2.us-east-1:azul-inventory-daniel2]
aws_iam_role_policy.service: Refreshing state... [id=azul-service-daniel2:service]
aws_lambda_event_source_mapping.indexer_aggregate: Refreshing state... [id=106b9384-54e6-442b-924e-68fed790ceb0]
aws_lambda_function_event_invoke_config.indexer_indexercachehealth: Refreshing state... [id=azul-indexer-daniel2-indexercachehealth]
aws_cloudwatch_event_target.indexer_indexercachehealth: Refreshing state... [id=azul-indexercachehealth-daniel2-azul-indexercachehealth-daniel2]
aws_lambda_permission.indexer_indexercachehealth: Refreshing state... [id=terraform-20241004005133472500000003]
aws_lambda_event_source_mapping.indexer_contribute_retry: Refreshing state... [id=92d9eafa-1951-4f37-b20d-8b5afe9ee981]
aws_lambda_function_event_invoke_config.service_servicecachehealth: Refreshing state... [id=azul-service-daniel2-servicecachehealth]
aws_cloudwatch_event_target.service_servicecachehealth: Refreshing state... [id=azul-servicecachehealth-daniel2-azul-servicecachehealth-daniel2]
aws_lambda_permission.service_servicecachehealth: Refreshing state... [id=terraform-20241004005149287500000004]
aws_api_gateway_rest_api.indexer: Refreshing state... [id=cg8vga0kph]
aws_acm_certificate_validation.service_0: Refreshing state... [id=2024-10-04 00:42:25.007 +0000 UTC]
aws_api_gateway_rest_api.service: Refreshing state... [id=pjlb65sk4c]
aws_acm_certificate_validation.indexer_0: Refreshing state... [id=2024-10-04 00:43:30.395 +0000 UTC]
aws_api_gateway_domain_name.service_0: Refreshing state... [id=service.daniel2.anvil.gi.ucsc.edu]
aws_api_gateway_domain_name.indexer_0: Refreshing state... [id=indexer.daniel2.anvil.gi.ucsc.edu]
aws_route53_record.service_0: Refreshing state... [id=Z05898353G87FKJOLNP7N_service.daniel2.anvil.gi.ucsc.edu_A]
aws_route53_record.indexer_0: Refreshing state... [id=Z05898353G87FKJOLNP7N_indexer.daniel2.anvil.gi.ucsc.edu_A]
aws_api_gateway_deployment.indexer: Refreshing state... [id=cetc10]
aws_lambda_permission.indexer: Refreshing state... [id=terraform-20241004005039045700000002]
aws_api_gateway_deployment.service: Refreshing state... [id=gf4kds]
aws_lambda_permission.service: Refreshing state... [id=terraform-20241004004654204200000001]
aws_api_gateway_stage.indexer: Refreshing state... [id=ags-cg8vga0kph-daniel2]
aws_api_gateway_stage.service: Refreshing state... [id=ags-pjlb65sk4c-daniel2]
aws_api_gateway_base_path_mapping.indexer_0: Refreshing state... [id=indexer.daniel2.anvil.gi.ucsc.edu/]
aws_api_gateway_method_settings.indexer_0: Refreshing state... [id=cg8vga0kph-daniel2-*/*]
aws_api_gateway_base_path_mapping.service_0: Refreshing state... [id=service.daniel2.anvil.gi.ucsc.edu/]
aws_api_gateway_method_settings.service_0: Refreshing state... [id=pjlb65sk4c-daniel2-*/*]
aws_wafv2_web_acl_association.indexer: Refreshing state... [id=arn:aws:wafv2:us-east-1:289950828509:regional/webacl/azul-api_gateway-daniel2/e8d9c8ae-c8b0-479a-852d-eb64685fc1ed,arn:aws:apigateway:us-east-1::/restapis/cg8vga0kph/stages/daniel2]
aws_wafv2_web_acl_association.service: Refreshing state... [id=arn:aws:wafv2:us-east-1:289950828509:regional/webacl/azul-api_gateway-daniel2/e8d9c8ae-c8b0-479a-852d-eb64685fc1ed,arn:aws:apigateway:us-east-1::/restapis/pjlb65sk4c/stages/daniel2]
aws_wafv2_web_acl_logging_configuration.waf_api_gateway: Refreshing state... [id=arn:aws:wafv2:us-east-1:289950828509:regional/webacl/azul-api_gateway-daniel2/e8d9c8ae-c8b0-479a-852d-eb64685fc1ed]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# aws_lambda_function.indexer will be updated in-place
~ resource "aws_lambda_function" "indexer" {
id = "azul-indexer-daniel2"
~ last_modified = "2025-01-09T00:32:28.000+0000" -> (known after apply)
~ source_code_hash = "fQKUWqcta0ZMuMuhrOyqzRchs70gJB4p5nfWnrI/iHE=" -> "ij0Ap18Oa8nXPY/bS2NTBT7tizNr2IpLpOzGlxYGxDs="
tags = {
"Name" = "azul-indexer-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-indexer"
"billing" = "anvil"
"component" = "azul-indexer"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.indexer_aggregate will be updated in-place
~ resource "aws_lambda_function" "indexer_aggregate" {
id = "azul-indexer-daniel2-aggregate"
~ last_modified = "2025-01-09T00:32:17.000+0000" -> (known after apply)
~ source_code_hash = "fQKUWqcta0ZMuMuhrOyqzRchs70gJB4p5nfWnrI/iHE=" -> "ij0Ap18Oa8nXPY/bS2NTBT7tizNr2IpLpOzGlxYGxDs="
tags = {
"Name" = "azul-indexer_aggregate-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-indexer"
"billing" = "anvil"
"component" = "azul-indexer_aggregate"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.indexer_aggregate_retry will be updated in-place
~ resource "aws_lambda_function" "indexer_aggregate_retry" {
id = "azul-indexer-daniel2-aggregate_retry"
~ last_modified = "2025-01-09T00:31:58.000+0000" -> (known after apply)
~ source_code_hash = "fQKUWqcta0ZMuMuhrOyqzRchs70gJB4p5nfWnrI/iHE=" -> "ij0Ap18Oa8nXPY/bS2NTBT7tizNr2IpLpOzGlxYGxDs="
tags = {
"Name" = "azul-indexer_aggregate_retry-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-indexer"
"billing" = "anvil"
"component" = "azul-indexer_aggregate_retry"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.indexer_contribute will be updated in-place
~ resource "aws_lambda_function" "indexer_contribute" {
id = "azul-indexer-daniel2-contribute"
~ last_modified = "2025-01-09T00:32:07.000+0000" -> (known after apply)
~ source_code_hash = "fQKUWqcta0ZMuMuhrOyqzRchs70gJB4p5nfWnrI/iHE=" -> "ij0Ap18Oa8nXPY/bS2NTBT7tizNr2IpLpOzGlxYGxDs="
tags = {
"Name" = "azul-indexer_contribute-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-indexer"
"billing" = "anvil"
"component" = "azul-indexer_contribute"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.indexer_contribute_retry will be updated in-place
~ resource "aws_lambda_function" "indexer_contribute_retry" {
id = "azul-indexer-daniel2-contribute_retry"
~ last_modified = "2025-01-09T00:32:38.000+0000" -> (known after apply)
~ source_code_hash = "fQKUWqcta0ZMuMuhrOyqzRchs70gJB4p5nfWnrI/iHE=" -> "ij0Ap18Oa8nXPY/bS2NTBT7tizNr2IpLpOzGlxYGxDs="
tags = {
"Name" = "azul-indexer_contribute_retry-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-indexer"
"billing" = "anvil"
"component" = "azul-indexer_contribute_retry"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.indexer_indexercachehealth will be updated in-place
~ resource "aws_lambda_function" "indexer_indexercachehealth" {
id = "azul-indexer-daniel2-indexercachehealth"
~ last_modified = "2025-01-09T00:31:37.000+0000" -> (known after apply)
~ source_code_hash = "fQKUWqcta0ZMuMuhrOyqzRchs70gJB4p5nfWnrI/iHE=" -> "ij0Ap18Oa8nXPY/bS2NTBT7tizNr2IpLpOzGlxYGxDs="
tags = {
"Name" = "azul-indexer_indexercachehealth-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-indexer"
"billing" = "anvil"
"component" = "azul-indexer_indexercachehealth"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.service will be updated in-place
~ resource "aws_lambda_function" "service" {
id = "azul-service-daniel2"
~ last_modified = "2025-01-09T00:32:48.000+0000" -> (known after apply)
~ source_code_hash = "bn9ex0mVCvhnY4tnMIVmD/scGfWlQ74JSOFKtae+dqY=" -> "qnqlBXy0XAbqE0t2iuqfiyYh6M5s7ffK2Y7wiX933c4="
tags = {
"Name" = "azul-service-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-service"
"billing" = "anvil"
"component" = "azul-service"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.service_manifest will be updated in-place
~ resource "aws_lambda_function" "service_manifest" {
id = "azul-service-daniel2-manifest"
~ last_modified = "2025-01-09T00:32:58.000+0000" -> (known after apply)
~ source_code_hash = "bn9ex0mVCvhnY4tnMIVmD/scGfWlQ74JSOFKtae+dqY=" -> "qnqlBXy0XAbqE0t2iuqfiyYh6M5s7ffK2Y7wiX933c4="
tags = {
"Name" = "azul-service_manifest-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-service"
"billing" = "anvil"
"component" = "azul-service_manifest"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
# aws_lambda_function.service_servicecachehealth will be updated in-place
~ resource "aws_lambda_function" "service_servicecachehealth" {
id = "azul-service-daniel2-servicecachehealth"
~ last_modified = "2025-01-09T00:31:48.000+0000" -> (known after apply)
~ source_code_hash = "bn9ex0mVCvhnY4tnMIVmD/scGfWlQ74JSOFKtae+dqY=" -> "qnqlBXy0XAbqE0t2iuqfiyYh6M5s7ffK2Y7wiX933c4="
tags = {
"Name" = "azul-service_servicecachehealth-daniel2"
"aws-chalice" = "version=1.31.3:stage=daniel2:app=azul-service"
"billing" = "anvil"
"component" = "azul-service_servicecachehealth"
"deployment" = "daniel2"
"owner" = "dsotirho@ucsc.edu"
"service" = "azul"
}
# (27 unchanged attributes hidden)
~ environment {
~ variables = {
~ "azul_git_commit" = "27c948d342a0753551ee37d31effb72e1156e981" -> "00c13a2241e3c0453e51e70535b971a1df502ea3"
# (41 unchanged elements hidden)
}
}
# (4 unchanged blocks hidden)
}
Plan: 0 to add, 9 to change, 0 to destroy.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Saved the plan to: plan.bin
To perform exactly these actions, run the following command to apply:
terraform apply "plan.bin"
terraform show -json plan.bin > plan.json
terraform apply -auto-approve plan.bin
aws_lambda_function.indexer_aggregate: Modifying... [id=azul-indexer-daniel2-aggregate]
aws_lambda_function.indexer_contribute_retry: Modifying... [id=azul-indexer-daniel2-contribute_retry]
aws_lambda_function.indexer_indexercachehealth: Modifying... [id=azul-indexer-daniel2-indexercachehealth]
aws_lambda_function.service_servicecachehealth: Modifying... [id=azul-service-daniel2-servicecachehealth]
aws_lambda_function.service: Modifying... [id=azul-service-daniel2]
aws_lambda_function.indexer: Modifying... [id=azul-indexer-daniel2]
aws_lambda_function.indexer_aggregate_retry: Modifying... [id=azul-indexer-daniel2-aggregate_retry]
aws_lambda_function.indexer_contribute: Modifying... [id=azul-indexer-daniel2-contribute]
aws_lambda_function.service_manifest: Modifying... [id=azul-service-daniel2-manifest]
aws_lambda_function.service_servicecachehealth: Still modifying... [id=azul-service-daniel2-servicecachehealth, 10s elapsed]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 10s elapsed]
aws_lambda_function.indexer_indexercachehealth: Still modifying... [id=azul-indexer-daniel2-indexercachehealth, 10s elapsed]
aws_lambda_function.indexer_contribute_retry: Still modifying... [id=azul-indexer-daniel2-contribute_retry, 10s elapsed]
aws_lambda_function.service: Still modifying... [id=azul-service-daniel2, 10s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 10s elapsed]
aws_lambda_function.indexer_contribute: Still modifying... [id=azul-indexer-daniel2-contribute, 10s elapsed]
aws_lambda_function.service_manifest: Still modifying... [id=azul-service-daniel2-manifest, 10s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 10s elapsed]
aws_lambda_function.service_manifest: Modifications complete after 15s [id=azul-service-daniel2-manifest]
aws_lambda_function.indexer_indexercachehealth: Still modifying... [id=azul-indexer-daniel2-indexercachehealth, 20s elapsed]
aws_lambda_function.service: Still modifying... [id=azul-service-daniel2, 20s elapsed]
aws_lambda_function.service_servicecachehealth: Still modifying... [id=azul-service-daniel2-servicecachehealth, 20s elapsed]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 20s elapsed]
aws_lambda_function.indexer_contribute_retry: Still modifying... [id=azul-indexer-daniel2-contribute_retry, 20s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 20s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 20s elapsed]
aws_lambda_function.indexer_contribute: Still modifying... [id=azul-indexer-daniel2-contribute, 20s elapsed]
aws_lambda_function.service_servicecachehealth: Modifications complete after 25s [id=azul-service-daniel2-servicecachehealth]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 30s elapsed]
aws_lambda_function.indexer_contribute_retry: Still modifying... [id=azul-indexer-daniel2-contribute_retry, 30s elapsed]
aws_lambda_function.service: Still modifying... [id=azul-service-daniel2, 30s elapsed]
aws_lambda_function.indexer_indexercachehealth: Still modifying... [id=azul-indexer-daniel2-indexercachehealth, 30s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 30s elapsed]
aws_lambda_function.indexer_contribute: Still modifying... [id=azul-indexer-daniel2-contribute, 30s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 30s elapsed]
aws_lambda_function.indexer_indexercachehealth: Modifications complete after 34s [id=azul-indexer-daniel2-indexercachehealth]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 40s elapsed]
aws_lambda_function.indexer_contribute_retry: Still modifying... [id=azul-indexer-daniel2-contribute_retry, 40s elapsed]
aws_lambda_function.service: Still modifying... [id=azul-service-daniel2, 40s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 40s elapsed]
aws_lambda_function.indexer_contribute: Still modifying... [id=azul-indexer-daniel2-contribute, 40s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 40s elapsed]
aws_lambda_function.indexer_contribute: Modifications complete after 44s [id=azul-indexer-daniel2-contribute]
aws_lambda_function.service: Still modifying... [id=azul-service-daniel2, 50s elapsed]
aws_lambda_function.indexer_contribute_retry: Still modifying... [id=azul-indexer-daniel2-contribute_retry, 50s elapsed]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 50s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 50s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 50s elapsed]
aws_lambda_function.service: Modifications complete after 53s [id=azul-service-daniel2]
aws_lambda_function.indexer_contribute_retry: Still modifying... [id=azul-indexer-daniel2-contribute_retry, 1m0s elapsed]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 1m0s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 1m0s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 1m0s elapsed]
aws_lambda_function.indexer_contribute_retry: Modifications complete after 1m4s [id=azul-indexer-daniel2-contribute_retry]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 1m10s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 1m10s elapsed]
aws_lambda_function.indexer: Still modifying... [id=azul-indexer-daniel2, 1m10s elapsed]
aws_lambda_function.indexer: Modifications complete after 1m14s [id=azul-indexer-daniel2]
aws_lambda_function.indexer_aggregate: Still modifying... [id=azul-indexer-daniel2-aggregate, 1m20s elapsed]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 1m20s elapsed]
aws_lambda_function.indexer_aggregate: Modifications complete after 1m23s [id=azul-indexer-daniel2-aggregate]
aws_lambda_function.indexer_aggregate_retry: Still modifying... [id=azul-indexer-daniel2-aggregate_retry, 1m30s elapsed]
aws_lambda_function.indexer_aggregate_retry: Modifications complete after 1m32s [id=azul-indexer-daniel2-aggregate_retry]
Apply complete! Resources: 0 added, 9 changed, 0 destroyed.
python /Users/daniel/repo/azul3/scripts/post_deploy_tdr.py
2025-01-08 16:37:32,401 INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:37:32,458 INFO MainThread azul.deployment: Allocated new Boto3 client for 'secretsmanager' with ID 4416109312
2025-01-08 16:37:32,959 INFO MainThread azul.terra: Making POST request to 'https://sam.dsde-dev.broadinstitute.org/register/user/v1'
2025-01-08 16:37:32,959 DEBUG MainThread azul.terra: β¦ with request body ''
2025-01-08 16:37:33,141 INFO MainThread azul.terra: Got 409 response after 0.182s from POST to https://sam.dsde-dev.broadinstitute.org/register/user/v1
2025-01-08 16:37:33,142 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:32 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization,content-type,accept,origin,x-app-id', 'Access-Control-Allow-Methods': 'GET,POST,PUT,PATCH,DELETE,OPTIONS,HEAD', 'Access-Control-Max-Age': '1728000', 'Content-Type': 'application/json', 'Content-Length': '163', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:33,142 DEBUG MainThread azul.terra: β¦ with response body b'{"causes":[],"message":"user azul-ucsc-6-daniel2@platform-anvil-dev.iam.gserviceaccount.com is already registered","source":"...'
2025-01-08 16:37:33,142 INFO MainThread azul.terra: Google service account 'azul-ucsc-6-daniel2@platform-anvil-dev.iam.gserviceaccount.com' previously registered with SAM.
2025-01-08 16:37:33,143 INFO MainThread azul.terra: Making GET request to 'https://sam.dsde-dev.broadinstitute.org/register/user/v1'
2025-01-08 16:37:33,143 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:37:33,829 INFO MainThread azul.terra: Got 200 response after 0.686s from GET to https://sam.dsde-dev.broadinstitute.org/register/user/v1
2025-01-08 16:37:33,830 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:33 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization,content-type,accept,origin,x-app-id', 'Access-Control-Allow-Methods': 'GET,POST,PUT,PATCH,DELETE,OPTIONS,HEAD', 'Access-Control-Max-Age': '1728000', 'Content-Type': 'application/json', 'Content-Length': '229', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:33,830 DEBUG MainThread azul.terra: β¦ with response body b'{"enabled":{"tosAccepted":true,"google":true,"ldap":true,"allUsersGroup":true,"adminEnabled":true},"userInfo":{"userEmail":"a...'
2025-01-08 16:37:34,124 INFO MainThread azul.terra: Making POST request to 'https://sam.dsde-dev.broadinstitute.org/register/user/v1'
2025-01-08 16:37:34,124 DEBUG MainThread azul.terra: β¦ with request body ''
2025-01-08 16:37:34,300 INFO MainThread azul.terra: Got 409 response after 0.176s from POST to https://sam.dsde-dev.broadinstitute.org/register/user/v1
2025-01-08 16:37:34,300 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:33 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization,content-type,accept,origin,x-app-id', 'Access-Control-Allow-Methods': 'GET,POST,PUT,PATCH,DELETE,OPTIONS,HEAD', 'Access-Control-Max-Age': '1728000', 'Content-Type': 'application/json', 'Content-Length': '170', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:34,300 DEBUG MainThread azul.terra: β¦ with response body b'{"causes":[],"message":"user azul-ucsc-6-public-daniel2@platform-anvil-dev.iam.gserviceaccount.com is already registered","so...'
2025-01-08 16:37:34,300 INFO MainThread azul.terra: Google service account 'azul-ucsc-6-public-daniel2@platform-anvil-dev.iam.gserviceaccount.com' previously registered with SAM.
2025-01-08 16:37:34,301 INFO MainThread azul.terra: Making GET request to 'https://sam.dsde-dev.broadinstitute.org/register/user/v1'
2025-01-08 16:37:34,301 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:37:34,888 INFO MainThread azul.terra: Got 200 response after 0.587s from GET to https://sam.dsde-dev.broadinstitute.org/register/user/v1
2025-01-08 16:37:34,888 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:34 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization,content-type,accept,origin,x-app-id', 'Access-Control-Allow-Methods': 'GET,POST,PUT,PATCH,DELETE,OPTIONS,HEAD', 'Access-Control-Max-Age': '1728000', 'Content-Type': 'application/json', 'Content-Length': '236', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:34,888 DEBUG MainThread azul.terra: β¦ with response body b'{"enabled":{"tosAccepted":true,"google":true,"ldap":true,"allUsersGroup":true,"adminEnabled":true},"userInfo":{"userEmail":"a...'
2025-01-08 16:37:35,074 INFO ThreadPoolExecutor-0_0 azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732&limit=2'
2025-01-08 16:37:35,074 DEBUG ThreadPoolExecutor-0_0 azul.terra: β¦ without request body
2025-01-08 16:37:35,074 INFO ThreadPoolExecutor-0_1 azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520&limit=2'
2025-01-08 16:37:35,074 INFO ThreadPoolExecutor-0_2 azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CMG_Sample_1_20230225_ANV5_202302281509&limit=2'
2025-01-08 16:37:35,074 DEBUG ThreadPoolExecutor-0_1 azul.terra: β¦ without request body
2025-01-08 16:37:35,074 DEBUG ThreadPoolExecutor-0_2 azul.terra: β¦ without request body
2025-01-08 16:37:35,513 INFO ThreadPoolExecutor-0_1 azul.terra: Got 200 response after 0.439s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520&limit=2
2025-01-08 16:37:35,514 INFO ThreadPoolExecutor-0_0 azul.terra: Got 200 response after 0.440s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732&limit=2
2025-01-08 16:37:35,514 INFO ThreadPoolExecutor-0_2 azul.terra: Got 200 response after 0.439s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CMG_Sample_1_20230225_ANV5_202302281509&limit=2
2025-01-08 16:37:35,514 DEBUG ThreadPoolExecutor-0_1 azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:35 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': '6B4ZEdzA', 'Content-Type': 'application/json', 'Content-Length': '910', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:35,514 DEBUG ThreadPoolExecutor-0_0 azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:35 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'a5Y1KzJl', 'Content-Type': 'application/json', 'Content-Length': '903', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:35,514 DEBUG ThreadPoolExecutor-0_2 azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:35 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'eX0Bgkdg', 'Content-Type': 'application/json', 'Content-Length': '908', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:35,514 DEBUG ThreadPoolExecutor-0_2 azul.terra: β¦ with response body b'{"total":156,"filteredTotal":1,"items":[{"id":"7d632e2a-54f1-4021-81fd-dbaf893715cd","name":"ANVIL_CMG_Sample_1_20230225_ANV5...'
2025-01-08 16:37:35,514 DEBUG ThreadPoolExecutor-0_0 azul.terra: β¦ with response body b'{"total":156,"filteredTotal":1,"items":[{"id":"790795c4-49b1-4ac8-a060-207b92ea08c5","name":"ANVIL_1000G_2019_Dev_20230609_AN...'
2025-01-08 16:37:35,514 DEBUG ThreadPoolExecutor-0_1 azul.terra: β¦ with response body b'{"total":156,"filteredTotal":1,"items":[{"id":"5f55901c-4bfc-4d33-b85d-bd9f812f57cd","name":"ANVIL_CCDG_Sample_1_20230228_ANV...'
2025-01-08 16:37:35,515 INFO ThreadPoolExecutor-0_2 __main__: TDR client is authorized for API access to tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:.
2025-01-08 16:37:35,515 INFO ThreadPoolExecutor-0_0 __main__: TDR client is authorized for API access to tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:.
2025-01-08 16:37:35,515 INFO ThreadPoolExecutor-0_1 __main__: TDR client is authorized for API access to tdr:bigquery:gcp:datarepo-dev-42c70e6a:ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520:.
2025-01-08 16:37:35,516 DEBUG ThreadPoolExecutor-0_2 azul.terra: Query (200 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:35,516 DEBUG ThreadPoolExecutor-0_0 azul.terra: Query (202 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:35,516 DEBUG ThreadPoolExecutor-0_1 azul.terra: Query (201 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:36,421 DEBUG ThreadPoolExecutor-0_2 azul.terra: Job info: {"job_id": "987a4a8a-cf5e-42e4-99b9-b9b608c58c13", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:36,426 DEBUG ThreadPoolExecutor-0_1 azul.terra: Job info: {"job_id": "b45a94bf-b590-4530-9711-3e46bb7317c3", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:36,506 DEBUG ThreadPoolExecutor-0_0 azul.terra: Job info: {"job_id": "a3d208dd-1776-400f-92c2-278ff0a24750", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:36,612 DEBUG ThreadPoolExecutor-0_2 azul.terra: Query (198 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:36,641 DEBUG ThreadPoolExecutor-0_1 azul.terra: Query (199 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:36,688 DEBUG ThreadPoolExecutor-0_0 azul.terra: Query (200 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:37,418 DEBUG ThreadPoolExecutor-0_1 azul.terra: Job info: {"job_id": "b5930069-0fcd-4473-99fa-224a9dfbc244", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:37,514 DEBUG ThreadPoolExecutor-0_2 azul.terra: Job info: {"job_id": "a705556f-c397-4971-8236-1ee309ce5922", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:37,818 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 15 tables in source tdr:bigquery:gcp:datarepo-dev-42c70e6a:ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520:
2025-01-08 16:37:37,818 DEBUG ThreadPoolExecutor-0_1 azul.terra: Query (15940 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:37:37,866 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:
2025-01-08 16:37:37,866 DEBUG ThreadPoolExecutor-0_2 azul.terra: Query (16969 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER β¦'
2025-01-08 16:37:38,249 DEBUG ThreadPoolExecutor-0_0 azul.terra: Job info: {"job_id": "c355571b-e5c1-43a9-a68f-1d169435afe9", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:38,681 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:
2025-01-08 16:37:38,682 DEBUG ThreadPoolExecutor-0_0 azul.terra: Query (17025 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORβ¦'
2025-01-08 16:37:38,901 DEBUG ThreadPoolExecutor-0_1 azul.terra: Job info: {"job_id": "d1b98eb5-01a0-4d6e-b815-ea23f44c60d0", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:37:39,032 DEBUG ThreadPoolExecutor-0_2 azul.terra: Job info: {"job_id": "3e753453-277e-4e21-bab8-84339d9f4e04", "total_rows": 10, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER \u2026"}
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'subject' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'file_inventory' (average batch size 28.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 37.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_sequencingactivity' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sample' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_file' (average batch size 28.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_donor' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,060 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sequencing' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,061 DEBUG ThreadPoolExecutor-0_1 azul.terra: Query (201 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:39,218 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:37:39,218 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'file_inventory' (average batch size 50.0, num batches 1)
2025-01-08 16:37:39,218 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_sequencingactivity' (average batch size 50.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'subject' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_donor' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 37.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sequencing' (average batch size 50.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sample' (average batch size 25.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_file' (average batch size 50.0, num batches 1)
2025-01-08 16:37:39,219 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'family' (average batch size 13.0, num batches 1)
2025-01-08 16:37:39,220 DEBUG ThreadPoolExecutor-0_2 azul.terra: Query (200 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:39,896 DEBUG ThreadPoolExecutor-0_0 azul.terra: Job info: {"job_id": "34be9633-603f-4144-8998-5fdc1b7ef76a", "total_rows": 10, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n OR\u2026"}
2025-01-08 16:37:40,062 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 24.0, num batches 1)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'qc_result_sample' (average batch size 199.0, num batches 16)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'participant' (average batch size 200.1, num batches 16)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'pedigree' (average batch size 200.1, num batches 16)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'file_inventory' (average batch size 101.6, num batches 256)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 200.1, num batches 16)
2025-01-08 16:37:40,063 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_activity' (average batch size 101.0, num batches 256)
2025-01-08 16:37:40,064 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 200.1, num batches 16)
2025-01-08 16:37:40,064 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_file' (average batch size 101.6, num batches 256)
2025-01-08 16:37:40,064 DEBUG ThreadPoolExecutor-0_0 azul.terra: Query (204 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65')\n "
2025-01-08 16:37:40,092 DEBUG ThreadPoolExecutor-0_1 azul.terra: Job info: {"job_id": "50db0710-64a5-45ca-bd50-7b9a5c4fd0e7", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:40,104 DEBUG ThreadPoolExecutor-0_2 azul.terra: Job info: {"job_id": "b2350ab1-c422-4c96-a673-889fd8de37d8", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:40,252 DEBUG ThreadPoolExecutor-0_1 azul.terra: Query (199 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:40,335 DEBUG ThreadPoolExecutor-0_2 azul.terra: Query (198 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:37:40,797 DEBUG ThreadPoolExecutor-0_0 azul.terra: Job info: {"job_id": "286993c4-4e3e-4313-a399-09d31cac7208", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65')\n "}
2025-01-08 16:37:40,956 DEBUG ThreadPoolExecutor-0_0 azul.terra: Query (202 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '65')\n "
2025-01-08 16:37:41,002 DEBUG ThreadPoolExecutor-0_1 azul.terra: Job info: {"job_id": "a3cc20df-4682-4350-bb16-7824151aa2c2", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:41,143 DEBUG ThreadPoolExecutor-0_2 azul.terra: Job info: {"job_id": "ff87c262-e56a-495a-a682-e25f550ee9d3", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:37:41,381 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 15 tables in source tdr:bigquery:gcp:datarepo-dev-42c70e6a:ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520:/1
2025-01-08 16:37:41,382 DEBUG ThreadPoolExecutor-0_1 azul.terra: Query (15940 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:37:41,487 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:37:41,487 DEBUG ThreadPoolExecutor-0_2 azul.terra: Query (16969 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER β¦'
2025-01-08 16:37:41,776 DEBUG ThreadPoolExecutor-0_0 azul.terra: Job info: {"job_id": "b4266196-0ba5-414f-bdf2-2e848aeec281", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '65')\n "}
2025-01-08 16:37:42,168 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:37:42,169 DEBUG ThreadPoolExecutor-0_0 azul.terra: Query (17057 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 2 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 2 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 2 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 2 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 2 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 2 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 2 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 2 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 2 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 2 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:37:42,382 DEBUG ThreadPoolExecutor-0_1 azul.terra: Job info: {"job_id": "4409099e-5938-46fe-a99d-6509ce529781", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-42c70e6a.ANVIL_CCDG_Sample_1_20230228_ANV5_202302281520.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:37:42,520 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'subject' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,520 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'file_inventory' (average batch size 28.0, num batches 1)
2025-01-08 16:37:42,520 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 37.0, num batches 1)
2025-01-08 16:37:42,520 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_sequencingactivity' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,520 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sample' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,520 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:37:42,521 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_file' (average batch size 28.0, num batches 1)
2025-01-08 16:37:42,521 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_donor' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,521 INFO ThreadPoolExecutor-0_1 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sequencing' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,732 DEBUG ThreadPoolExecutor-0_2 azul.terra: Job info: {"job_id": "0a9d91ae-b537-4c71-ba15-41ebafd6b7d6", "total_rows": 10, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER \u2026"}
2025-01-08 16:37:42,922 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:37:42,923 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'file_inventory' (average batch size 50.0, num batches 1)
2025-01-08 16:37:42,923 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_sequencingactivity' (average batch size 50.0, num batches 1)
2025-01-08 16:37:42,923 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'subject' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,923 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_donor' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,924 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 37.0, num batches 1)
2025-01-08 16:37:42,924 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sequencing' (average batch size 50.0, num batches 1)
2025-01-08 16:37:42,924 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sample' (average batch size 25.0, num batches 1)
2025-01-08 16:37:42,924 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_file' (average batch size 50.0, num batches 1)
2025-01-08 16:37:42,924 INFO ThreadPoolExecutor-0_2 azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'family' (average batch size 13.0, num batches 1)
2025-01-08 16:37:43,381 DEBUG ThreadPoolExecutor-0_0 azul.terra: Job info: {"job_id": "6ecf7c14-107e-44bc-b952-01cc8a2a5385", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 2 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 2 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 2 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 2 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 2 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 2 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 2 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 2 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 2 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 2 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:37:43,547 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'qc_result_sample' (average batch size 22.0, num batches 1)
2025-01-08 16:37:43,547 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_donor' (average batch size 13.0, num batches 1)
2025-01-08 16:37:43,547 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'pedigree' (average batch size 11.0, num batches 1)
2025-01-08 16:37:43,547 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_file' (average batch size 114.0, num batches 1)
2025-01-08 16:37:43,547 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_activity' (average batch size 89.0, num batches 1)
2025-01-08 16:37:43,548 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'sample' (average batch size 10.0, num batches 1)
2025-01-08 16:37:43,548 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'participant' (average batch size 13.0, num batches 1)
2025-01-08 16:37:43,548 INFO ThreadPoolExecutor-0_0 azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'file_inventory' (average batch size 97.0, num batches 1)
2025-01-08 16:37:43,549 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/roleMap'
2025-01-08 16:37:43,550 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:37:43,964 INFO MainThread azul.terra: Got 200 response after 0.415s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/roleMap
2025-01-08 16:37:43,965 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:43 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'pomYJqkg', 'Content-Type': 'application/json', 'Content-Length': '13303', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:43,965 DEBUG MainThread azul.terra: β¦ with response body b'{"roleMap":{"a0925f5f-3d38-4f8a-a09f-f6997072dc23":["aggregate_data_reader","discoverer","reader"],"215793b0-b991-4b73-9584-1...'
2025-01-08 16:37:43,967 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/roleMap'
2025-01-08 16:37:43,967 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:37:44,195 INFO MainThread azul.terra: Got 200 response after 0.228s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/roleMap
2025-01-08 16:37:44,195 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:37:44 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'ebW8l3gQ', 'Content-Type': 'application/json', 'Content-Length': '13564', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:37:44,195 DEBUG MainThread azul.terra: β¦ with response body b'{"roleMap":{"a0925f5f-3d38-4f8a-a09f-f6997072dc23":["reader","discoverer","aggregate_data_reader"],"215793b0-b991-4b73-9584-1...'
python /Users/daniel/repo/azul3/scripts/check_branch.py
python /Users/daniel/repo/azul3/lambdas/service/.chalice/config.json.template.py /Users/daniel/repo/azul3/lambdas/service/.chalice/config.json
[INFO] 2025-01-08T16:38:45.631Z 00010ca1-b0ba-466f-8c58-dabbad000000 botocore.credentials Found credentials in shared credentials file: ~/.aws/credentials
[INFO] 2025-01-08T16:38:45.671Z 00010ca1-b0ba-466f-8c58-dabbad000000 azul.deployment Allocated new Boto3 client for 'iam' with ID 4395578624
Creating /Users/daniel/repo/azul3/lambdas/service/.chalice/config.json
python -m unittest -v integration_test
2025-01-08 16:38:46,670 INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:38:46,702 INFO MainThread azul.deployment: Allocated new Boto3 client for 'es' with ID 4368761344
/Users/daniel/repo/azul3/.venv/lib/python3.12/site-packages/botocore/auth.py:424: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
datetime_now = datetime.datetime.utcnow()
2025-01-08 16:38:47,204 DEBUG MainThread azul.es: Creating ES client [vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443]
2025-01-08 16:38:47,215 INFO MainThread azul.deployment: Allocated new Boto3 client for 'sts' with ID 4376309936
2025-01-08 16:38:47,227 INFO MainThread botocore.credentials: Found credentials in environment variables.
2025-01-08 16:38:47,227 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:38:47,227 INFO MainThread elasticsearch: β¦ without request body
/Users/daniel/repo/azul3/.venv/lib/python3.12/site-packages/aws_requests_auth/aws_auth.py:107: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
t = datetime.datetime.utcnow()
2025-01-08 16:38:47,667 INFO MainThread elasticsearch: Got 200 response after 0.440s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:38:47,667 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_activities":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":β¦'
2025-01-08 16:38:47,672 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:38:47,672 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:47,767 INFO MainThread elasticsearch: Got 200 response after 0.095s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:38:47,767 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_activities_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappβ¦'
2025-01-08 16:38:47,769 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:38:47,769 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:47,865 INFO MainThread elasticsearch: Got 200 response after 0.096s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:38:47,865 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_biosamples":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":β¦'
2025-01-08 16:38:47,867 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:38:47,867 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:47,959 INFO MainThread elasticsearch: Got 200 response after 0.091s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:38:47,959 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_biosamples_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappβ¦'
2025-01-08 16:38:47,961 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:38:47,961 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,055 INFO MainThread elasticsearch: Got 200 response after 0.094s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:38:48,055 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_bundles":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"stβ¦'
2025-01-08 16:38:48,057 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:38:48,057 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,147 INFO MainThread elasticsearch: Got 200 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:38:48,147 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_bundles_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappingβ¦'
2025-01-08 16:38:48,149 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:38:48,149 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,242 INFO MainThread elasticsearch: Got 200 response after 0.093s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:38:48,242 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_datasets":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"sβ¦'
2025-01-08 16:38:48,243 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:38:48,244 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,344 INFO MainThread elasticsearch: Got 200 response after 0.100s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:38:48,344 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_datasets_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappinβ¦'
2025-01-08 16:38:48,346 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:38:48,346 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,438 INFO MainThread elasticsearch: Got 200 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:38:48,439 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_donors":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"strβ¦'
2025-01-08 16:38:48,440 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:38:48,440 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,537 INFO MainThread elasticsearch: Got 200 response after 0.096s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:38:48,537 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_donors_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_β¦'
2025-01-08 16:38:48,539 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:38:48,539 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,630 INFO MainThread elasticsearch: Got 200 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:38:48,630 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_files":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"striβ¦'
2025-01-08 16:38:48,631 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:38:48,631 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,723 INFO MainThread elasticsearch: Got 200 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:38:48,723 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_files_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_tβ¦'
2025-01-08 16:38:48,726 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:38:48,726 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:38:48,816 INFO MainThread elasticsearch: Got 200 response after 0.091s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:38:48,817 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_replica":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"stβ¦'
test_local_chalice (integration_test.AzulChaliceLocalIntegrationTest.test_local_chalice) ... 2025-01-08 16:38:48,852 INFO Thread-2 (process_request_thread) azul.chalice: Received GET request for '/', with {"query": null, "headers": {"host": "127.0.0.1:8000", "user-agent": "python-requests/2.32.3", "accept-encoding": "gzip, deflate, br", "accept": "*/*", "connection": "keep-alive"}}.
2025-01-08 16:38:48,852 INFO Thread-2 (process_request_thread) azul.chalice: Did not authenticate request.
2025-01-08 16:38:48,853 DEBUG Thread-2 (process_request_thread) azul.chalice: Returning 301 response with headers {"Location": "http://127.0.0.1:8000/static/index.html", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}. See next line for the first 1024 characters of the body.
127.0.0.1 - - [08/Jan/2025 16:38:48] "GET / HTTP/1.1" 301 -
2025-01-08 16:38:48,853 INFO Thread-2 (process_request_thread) azul.chalice: Received GET request for '/static/index.html', with {"query": null, "headers": {"host": "127.0.0.1:8000", "user-agent": "python-requests/2.32.3", "accept-encoding": "gzip, deflate, br", "accept": "*/*", "connection": "keep-alive"}}.
2025-01-08 16:38:48,853 INFO Thread-2 (process_request_thread) azul.chalice: Did not authenticate request.
2025-01-08 16:38:48,854 DEBUG Thread-2 (process_request_thread) azul.chalice: Returning 200 response with headers {"Content-Type": "text/html", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}. See next line for the first 1024 characters of the body.
Swagger UI
127.0.0.1 - - [08/Jan/2025 16:38:48] "GET /static/index.html HTTP/1.1" 200 -
ok
test_local_chalice_health_endpoint (integration_test.AzulChaliceLocalIntegrationTest.test_local_chalice_health_endpoint) ... 2025-01-08 16:38:48,855 INFO Thread-3 (process_request_thread) azul.chalice: Received GET request for '/health', with {"query": null, "headers": {"host": "127.0.0.1:8000", "user-agent": "python-requests/2.32.3", "accept-encoding": "gzip, deflate, br", "accept": "*/*", "connection": "keep-alive"}}.
2025-01-08 16:38:48,855 INFO Thread-3 (process_request_thread) azul.chalice: Did not authenticate request.
2025-01-08 16:38:48,855 INFO Thread-3 (process_request_thread) azul.health: Getting health property 'api_endpoints'
2025-01-08 16:38:48,856 INFO Thread-3 (process_request_thread) azul.health: Making HEAD request to https://service.daniel2.anvil.gi.ucsc.edu/index/datasets?size=1
2025-01-08 16:38:49,269 INFO Thread-3 (process_request_thread) azul.health: Got 200 response after 0.413s from HEAD request to https://service.daniel2.anvil.gi.ucsc.edu/index/datasets?size=1
2025-01-08 16:38:49,270 INFO Thread-3 (process_request_thread) azul.health: Getting health property 'elasticsearch'
2025-01-08 16:38:49,293 INFO Thread-3 (process_request_thread) botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:38:49,342 INFO Thread-3 (process_request_thread) azul.deployment: Allocated new Boto3 client for 'es' with ID 4417609632
2025-01-08 16:38:49,831 INFO Thread-3 (process_request_thread) elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/
2025-01-08 16:38:49,832 INFO Thread-3 (process_request_thread) elasticsearch: β¦ without request body
2025-01-08 16:38:49,920 INFO Thread-3 (process_request_thread) elasticsearch: Got 200 response after 0.088s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/
2025-01-08 16:38:49,920 INFO Thread-3 (process_request_thread) elasticsearch: β¦ with response body ''
2025-01-08 16:38:49,920 INFO Thread-3 (process_request_thread) azul.health: Getting health property 'other_lambdas'
2025-01-08 16:38:49,920 INFO Thread-3 (process_request_thread) azul.health: Requesting furl('https://indexer.daniel2.anvil.gi.ucsc.edu/health/basic?catalog=anvil')
2025-01-08 16:38:53,382 INFO Thread-3 (process_request_thread) azul.health: Getting health property 'progress'
2025-01-08 16:38:53,382 INFO Thread-3 (process_request_thread) azul.health: Getting health property 'queues'
2025-01-08 16:38:53,418 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:53,418 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with request body b'{"QueueName": "azul-notifications-daniel2"}'
2025-01-08 16:38:53,801 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Got 200 response
2025-01-08 16:38:53,802 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with response body b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2"}'
2025-01-08 16:38:53,807 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:53,807 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with request body b'{"QueueUrl": "https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2", "AttributeNames": ["All"]}'
2025-01-08 16:38:53,893 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Got 200 response
2025-01-08 16:38:53,893 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with response body b'{"Attributes":{"QueueArn":"arn:aws:sqs:us-east-1:289950828509:azul-notifications-daniel2","ApproximateNumberOfMessages":"0","...'
2025-01-08 16:38:53,894 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:53,894 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with request body b'{"QueueName": "azul-notifications_retry-daniel2"}'
2025-01-08 16:38:53,981 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Got 200 response
2025-01-08 16:38:53,981 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with response body b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2"}'
2025-01-08 16:38:53,983 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:53,983 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with request body b'{"QueueUrl": "https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2", "AttributeNames": ["All"]}'
2025-01-08 16:38:54,068 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Got 200 response
2025-01-08 16:38:54,068 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with response body b'{"Attributes":{"QueueArn":"arn:aws:sqs:us-east-1:289950828509:azul-notifications_retry-daniel2","ApproximateNumberOfMessages"...'
2025-01-08 16:38:54,069 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,069 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with request body b'{"QueueName": "azul-tallies-daniel2.fifo"}'
2025-01-08 16:38:54,152 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Got 200 response
2025-01-08 16:38:54,152 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with response body b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo"}'
2025-01-08 16:38:54,153 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,153 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with request body b'{"QueueUrl": "https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo", "AttributeNames": ["All"]}'
2025-01-08 16:38:54,240 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Got 200 response
2025-01-08 16:38:54,240 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with response body b'{"Attributes":{"QueueArn":"arn:aws:sqs:us-east-1:289950828509:azul-tallies-daniel2.fifo","ApproximateNumberOfMessages":"0","A...'
2025-01-08 16:38:54,242 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,242 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with request body b'{"QueueName": "azul-tallies_retry-daniel2.fifo"}'
2025-01-08 16:38:54,330 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Got 200 response
2025-01-08 16:38:54,330 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with response body b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo"}'
2025-01-08 16:38:54,332 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,332 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with request body b'{"QueueUrl": "https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo", "AttributeNames": ["All"]}'
2025-01-08 16:38:54,419 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Got 200 response
2025-01-08 16:38:54,419 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with response body b'{"Attributes":{"QueueArn":"arn:aws:sqs:us-east-1:289950828509:azul-tallies_retry-daniel2.fifo","ApproximateNumberOfMessages":...'
2025-01-08 16:38:54,420 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,421 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with request body b'{"QueueName": "azul-tallies_fail-daniel2.fifo"}'
2025-01-08 16:38:54,505 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Got 200 response
2025-01-08 16:38:54,505 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with response body b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_fail-daniel2.fifo"}'
2025-01-08 16:38:54,506 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,506 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with request body b'{"QueueUrl": "https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_fail-daniel2.fifo", "AttributeNames": ["All"]}'
2025-01-08 16:38:54,592 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Got 200 response
2025-01-08 16:38:54,593 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with response body b'{"Attributes":{"QueueArn":"arn:aws:sqs:us-east-1:289950828509:azul-tallies_fail-daniel2.fifo","ApproximateNumberOfMessages":"...'
2025-01-08 16:38:54,594 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,594 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with request body b'{"QueueName": "azul-notifications_fail-daniel2"}'
2025-01-08 16:38:54,682 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: Got 200 response
2025-01-08 16:38:54,682 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueUrl: β¦ with response body b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_fail-daniel2"}'
2025-01-08 16:38:54,684 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Making POST request to https://sqs.us-east-1.amazonaws.com/
2025-01-08 16:38:54,684 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with request body b'{"QueueUrl": "https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_fail-daniel2", "AttributeNames": ["All"]}'
2025-01-08 16:38:54,770 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: Got 200 response
2025-01-08 16:38:54,770 INFO Thread-3 (process_request_thread) azul.boto3: sqs.GetQueueAttributes: β¦ with response body b'{"Attributes":{"QueueArn":"arn:aws:sqs:us-east-1:289950828509:azul-notifications_fail-daniel2","ApproximateNumberOfMessages":...'
2025-01-08 16:38:54,771 DEBUG Thread-3 (process_request_thread) azul.chalice: Returning 200 response with headers {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}. See next line for the first 1024 characters of the body.
{"api_endpoints": {"up": true}, "elasticsearch": {"up": true}, "other_lambdas": {"indexer": {"up": true}, "up": true}, "progress": {"up": true, "unindexed_bundles": 0, "unindexed_documents": 0}, "queues": {"up": true, "azul-notifications-daniel2": {"up": true, "messages": {"delayed": 0, "invisible": 0, "queued": 0}}, "azul-notifications_retry-daniel2": {"up": true, "messages": {"delayed": 0, "invisible": 0, "queued": 0}}, "azul-tallies-daniel2.fifo": {"up": true, "messages": {"delayed": 0, "invisible": 0, "queued": 0}}, "azul-tallies_retry-daniel2.fifo": {"up": true, "messages": {"delayed": 0, "invisible": 0, "queued": 0}}, "azul-tallies_fail-daniel2.fifo": {"up": true, "messages": {"delayed": 0, "invisible": 0, "queued": 0}}, "azul-notifications_fail-daniel2": {"up": true, "messages": {"delayed": 0, "invisible": 0, "queued": 0}}}, "up": true}
127.0.0.1 - - [08/Jan/2025 16:38:54] "GET /health HTTP/1.1" 200 -
ok
test_local_chalice_index_endpoints (integration_test.AzulChaliceLocalIntegrationTest.test_local_chalice_index_endpoints) ... 2025-01-08 16:38:54,774 INFO Thread-4 (process_request_thread) azul.chalice: Received GET request for '/index/files', with {"query": {"catalog": "anvil-it"}, "headers": {"host": "127.0.0.1:8000", "user-agent": "python-requests/2.32.3", "accept-encoding": "gzip, deflate, br", "accept": "*/*", "connection": "keep-alive"}}.
2025-01-08 16:38:54,774 INFO Thread-4 (process_request_thread) azul.chalice: Did not authenticate request.
2025-01-08 16:38:54,786 INFO Thread-4 (process_request_thread) botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:38:54,827 INFO Thread-4 (process_request_thread) azul.deployment: Allocated new Boto3 client for 'dynamodb' with ID 4720839904
2025-01-08 16:38:54,828 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.GetItem: Making POST request to https://dynamodb.us-east-1.amazonaws.com/
2025-01-08 16:38:54,828 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.GetItem: β¦ with request body b'{"TableName": "azul-sources_cache_by_auth-daniel2", "Key": {"identity": {"S": "anvil-it:"}}, "ProjectionExpression": "sources...'
2025-01-08 16:38:55,128 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.GetItem: Got 200 response
2025-01-08 16:38:55,128 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.GetItem: β¦ with response body b'{}'
2025-01-08 16:38:55,144 INFO Thread-4 (process_request_thread) azul.deployment: Allocated new Boto3 client for 'secretsmanager' with ID 4721336848
2025-01-08 16:38:55,675 INFO Thread-4 (process_request_thread) azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/roleMap'
2025-01-08 16:38:55,675 DEBUG Thread-4 (process_request_thread) azul.terra: β¦ without request body
2025-01-08 16:38:56,114 INFO Thread-4 (process_request_thread) azul.terra: Got 200 response after 0.439s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/roleMap
2025-01-08 16:38:56,114 DEBUG Thread-4 (process_request_thread) azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:38:56 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': '6JQgxwlp', 'Content-Type': 'application/json', 'Content-Length': '13303', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:38:56,115 DEBUG Thread-4 (process_request_thread) azul.terra: β¦ with response body b'{"roleMap":{"a0925f5f-3d38-4f8a-a09f-f6997072dc23":["reader","discoverer","aggregate_data_reader"],"215793b0-b991-4b73-9584-1...'
2025-01-08 16:38:56,119 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.PutItem: Making POST request to https://dynamodb.us-east-1.amazonaws.com/
2025-01-08 16:38:56,119 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.PutItem: β¦ with request body b'{"TableName": "azul-sources_cache_by_auth-daniel2", "Item": {"identity": {"S": "anvil-it:"}, "sources": {"S": "[\\"43983b5a-24...'
2025-01-08 16:38:56,293 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.PutItem: Got 200 response
2025-01-08 16:38:56,293 INFO Thread-4 (process_request_thread) azul.boto3: dynamodb.PutItem: β¦ with response body b'{}'
2025-01-08 16:38:56,310 INFO Thread-4 (process_request_thread) azul.deployment: Allocated new Boto3 client for 'es' with ID 4723837072
2025-01-08 16:38:56,808 INFO Thread-4 (process_request_thread) elasticsearch: Making POST request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate/_search
2025-01-08 16:38:56,808 INFO Thread-4 (process_request_thread) elasticsearch: β¦ with request body b'{"aggs":{"source_id":{"filter":{"bool":{}},"aggs":{"myTerms":{"terms":{"field":"sources.id.keyword","size":99999},"meta":{"pa...'
2025-01-08 16:38:57,312 INFO Thread-4 (process_request_thread) elasticsearch: Got 200 response after 0.504s from POST to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate/_search
2025-01-08 16:38:57,312 INFO Thread-4 (process_request_thread) elasticsearch: β¦ with response body '{"took":14,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":102,"relationβ¦'
2025-01-08 16:38:57,318 DEBUG Thread-4 (process_request_thread) azul.chalice: Returning 200 response with headers {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}. See next line for the first 1024 characters of the body.
{"hits": [{"entryId": "6185f612-ac54-43b7-88a2-3c9527973b84", "sources": [{"source_spec": "tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1", "source_id": "790795c4-49b1-4ac8-a060-207b92ea08c5"}], "bundles": [{"bundle_uuid": "6590c9b3-b815-ac65-a2df-4d5c855e53bb", "bundle_version": "2022-06-01T00:00:00.000000Z"}], "activities": [{"activity_type": ["Checksum", "Unknown"], "assay_type": [null], "data_modality": [null]}], "biosamples": [{"anatomical_site": [null], "biosample_type": [null], "disease": [null], "donor_age_at_collection_unit": [null], "donor_age_at_collection": [{"gte": null, "lte": null}]}], "datasets": [{"dataset_id": ["385290c3-dff5-fb6d-2501-fa0ba3ad1c35"], "title": ["ANVIL_1000G_2019_Dev"]}], "diagnoses": [], "donors": [{"organism_type": [null], "phenotypic_sex": [null], "reported_ethnicity": [null], "genetic_ancestry": [null]}], "files": [{"document_id": "6185f612-ac54-43b7-88a2-3c9527973b84", "source_datarepo_row_ids": ["file_inventory:143e6645-f584-
127.0.0.1 - - [08/Jan/2025 16:38:57] "GET /index/files?catalog=anvil-it HTTP/1.1" 200 -
ok
test_local_filtered_index_endpoints (integration_test.AzulChaliceLocalIntegrationTest.test_local_filtered_index_endpoints) ... 2025-01-08 16:38:57,322 INFO Thread-5 (process_request_thread) azul.chalice: Received GET request for '/index/files', with {"query": {"filters": "{\"donors.organism_type\": {\"is\": [\"Homo sapiens\"]}}", "catalog": "anvil-it"}, "headers": {"host": "127.0.0.1:8000", "user-agent": "python-requests/2.32.3", "accept-encoding": "gzip, deflate, br", "accept": "*/*", "connection": "keep-alive"}}.
2025-01-08 16:38:57,322 INFO Thread-5 (process_request_thread) azul.chalice: Did not authenticate request.
2025-01-08 16:38:57,341 INFO Thread-5 (process_request_thread) botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:38:57,380 INFO Thread-5 (process_request_thread) azul.deployment: Allocated new Boto3 client for 'dynamodb' with ID 4731538032
2025-01-08 16:38:57,381 INFO Thread-5 (process_request_thread) azul.boto3: dynamodb.GetItem: Making POST request to https://dynamodb.us-east-1.amazonaws.com/
2025-01-08 16:38:57,381 INFO Thread-5 (process_request_thread) azul.boto3: dynamodb.GetItem: β¦ with request body b'{"TableName": "azul-sources_cache_by_auth-daniel2", "Key": {"identity": {"S": "anvil-it:"}}, "ProjectionExpression": "sources...'
2025-01-08 16:38:57,658 INFO Thread-5 (process_request_thread) azul.boto3: dynamodb.GetItem: Got 200 response
2025-01-08 16:38:57,658 INFO Thread-5 (process_request_thread) azul.boto3: dynamodb.GetItem: β¦ with response body b'{"Item":{"expiration":{"N":"1736383196"},"sources":{"S":"[\\"43983b5a-24f5-4cc6-b37d-27901430b9aa\\", \\"a6ffed95-9d29-4484-a5db...'
2025-01-08 16:38:57,675 INFO Thread-5 (process_request_thread) elasticsearch: Making POST request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate/_search
2025-01-08 16:38:57,675 INFO Thread-5 (process_request_thread) elasticsearch: β¦ with request body b'{"aggs":{"source_id":{"filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.donors.organism_type.keyword"...'
2025-01-08 16:38:57,867 INFO Thread-5 (process_request_thread) elasticsearch: Got 200 response after 0.192s from POST to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate/_search
2025-01-08 16:38:57,867 INFO Thread-5 (process_request_thread) elasticsearch: β¦ with response body '{"took":10,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":β¦'
2025-01-08 16:38:57,870 DEBUG Thread-5 (process_request_thread) azul.chalice: Returning 200 response with headers {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}. See next line for the first 1024 characters of the body.
{"hits": [], "pagination": {"count": 0, "total": 0, "size": 10, "next": null, "previous": null, "pages": 0, "sort": "files.file_id", "order": "asc"}, "termFacets": {"diagnoses.phenotype": {"terms": [], "total": 0, "type": "terms"}, "biosamples.disease": {"terms": [], "total": 0, "type": "terms"}, "diagnoses.disease": {"terms": [], "total": 0, "type": "terms"}, "diagnoses.phenopacket": {"terms": [], "total": 0, "type": "terms"}, "files.is_supplementary": {"terms": [], "total": 0, "type": "terms"}, "files.data_modality": {"terms": [], "total": 0, "type": "terms"}, "donors.reported_ethnicity": {"terms": [], "total": 0, "type": "terms"}, "datasets.consent_group": {"terms": [], "total": 0, "type": "terms"}, "activities.assay_type": {"terms": [], "total": 0, "type": "terms"}, "datasets.title": {"terms": [], "total": 0, "type": "terms"}, "biosamples.anatomical_site": {"terms": [], "total": 0, "type": "terms"}, "donors.organism_type": {"terms": [{"term": null, "count": 102}], "total": 102, "type": "terms"}, "datasets
127.0.0.1 - - [08/Jan/2025 16:38:57] "GET /index/files?filters=%7B%22donors.organism_type%22%3A+%7B%22is%22%3A+%5B%22Homo+sapiens%22%5D%7D%7D&catalog=anvil-it HTTP/1.1" 200 -
ok
test_azul_client_error_handling (integration_test.AzulClientIntegrationTest.test_azul_client_error_handling) ... 2025-01-08 16:38:58,328 INFO MainThread test.integration_test: Using random seed 2129499653175755424
2025-01-08 16:38:58,329 INFO pool_0 azul.azulclient: Notifying https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 0.
2025-01-08 16:38:58,348 INFO pool_0 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:38:58,392 INFO pool_0 azul.deployment: Allocated new Boto3 client for 'secretsmanager' with ID 4738531152
2025-01-08 16:38:58,708 INFO pool_0 azul.azulclient: Making POST request to 'https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add'
2025-01-08 16:38:58,708 DEBUG pool_0 azul.azulclient: β¦ with request body b'{}'
2025-01-08 16:39:01,379 INFO pool_0 azul.azulclient: Got 400 response after 2.671s from POST to https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add
2025-01-08 16:39:01,379 DEBUG pool_0 azul.azulclient: β¦ with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '78', 'Connection': 'keep-alive', 'Date': 'Thu, 09 Jan 2025 00:39:01 GMT', 'X-Amzn-Trace-Id': 'Root=1-677f1aa2-6b93973d2009539802ff14b9;Parent=376580cff306999a;Sampled=0;Lineage=1:ba9bd050:0', 'x-amzn-RequestId': '73c388c2-6151-4978-bc66-6f0a55c0886c', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", 'x-amz-apigw-id': 'EGEZhHtQIAMESuA=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 e84fdc4f831d7aaaedd48d30a9d5a068.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'SFO5-C3', 'X-Amz-Cf-Id': 'qRHt1gjPDlONrHXoVVBAiLvbkQ1M_UP8J4wlTXKV-PBs9Ydc0Yn_Hg=='})
2025-01-08 16:39:01,379 DEBUG pool_0 azul.azulclient: β¦ with response body b'{"Code":"BadRequestError","Message":"Missing notification entry: bundle_fqid"}'
2025-01-08 16:39:01,379 WARNING pool_0 azul.azulclient: Retrying to notify https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 0, after error .
2025-01-08 16:39:01,380 INFO pool_1 azul.azulclient: Notifying https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 1.
2025-01-08 16:39:01,394 INFO pool_1 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:01,446 INFO pool_1 azul.deployment: Allocated new Boto3 client for 'secretsmanager' with ID 4745227552
2025-01-08 16:39:01,768 INFO pool_1 azul.azulclient: Making POST request to 'https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add'
2025-01-08 16:39:01,768 DEBUG pool_1 azul.azulclient: β¦ with request body b'{}'
2025-01-08 16:39:01,880 INFO pool_1 azul.azulclient: Got 400 response after 0.111s from POST to https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add
2025-01-08 16:39:01,880 DEBUG pool_1 azul.azulclient: β¦ with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '78', 'Connection': 'keep-alive', 'Date': 'Thu, 09 Jan 2025 00:39:01 GMT', 'X-Amzn-Trace-Id': 'Root=1-677f1aa5-5efa246c77dc4d3471aa92b7;Parent=7e35c34a5b2ed9c9;Sampled=0;Lineage=1:ba9bd050:0', 'x-amzn-RequestId': '81f63271-b8ff-4e6b-8440-c0932e17dd3d', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", 'x-amz-apigw-id': 'EGEZ-ELZIAMEvOw=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 e84fdc4f831d7aaaedd48d30a9d5a068.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'SFO5-C3', 'X-Amz-Cf-Id': 'QNK2XphEO7n0jQ5Mre_Fy4n043Vp0x_kDFPNOtrNK4oy8LAnBWv1yw=='})
2025-01-08 16:39:01,880 DEBUG pool_1 azul.azulclient: β¦ with response body b'{"Code":"BadRequestError","Message":"Missing notification entry: bundle_fqid"}'
2025-01-08 16:39:01,880 WARNING pool_1 azul.azulclient: Retrying to notify https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 1, after error .
2025-01-08 16:39:01,881 INFO pool_1 azul.azulclient: Notifying https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 2.
2025-01-08 16:39:01,882 INFO pool_1 azul.azulclient: Making POST request to 'https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add'
2025-01-08 16:39:01,882 DEBUG pool_1 azul.azulclient: β¦ with request body b'{}'
2025-01-08 16:39:02,117 INFO pool_1 azul.azulclient: Got 400 response after 0.235s from POST to https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add
2025-01-08 16:39:02,117 DEBUG pool_1 azul.azulclient: β¦ with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '78', 'Connection': 'keep-alive', 'Date': 'Thu, 09 Jan 2025 00:39:02 GMT', 'X-Amzn-Trace-Id': 'Root=1-677f1aa6-15ac4b0766523a6e5cadaa25;Parent=4dc1c6c1603e1ea6;Sampled=0;Lineage=1:ba9bd050:0', 'x-amzn-RequestId': 'd2cc85f5-7691-490f-a3ee-e98fb58c4c80', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", 'x-amz-apigw-id': 'EGEaAHmWIAMECOw=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 e84fdc4f831d7aaaedd48d30a9d5a068.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'SFO5-C3', 'X-Amz-Cf-Id': 'bLUoaglw2VC1b8k00QnY_a9jcE9xomHQHLCFmgqJaMpPsN-FIEmIDA=='})
2025-01-08 16:39:02,117 DEBUG pool_1 azul.azulclient: β¦ with response body b'{"Code":"BadRequestError","Message":"Missing notification entry: bundle_fqid"}'
2025-01-08 16:39:02,118 WARNING pool_1 azul.azulclient: Retrying to notify https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 2, after error .
2025-01-08 16:39:02,118 INFO pool_2 azul.azulclient: Notifying https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 3.
2025-01-08 16:39:02,139 INFO pool_2 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:02,180 INFO pool_2 azul.deployment: Allocated new Boto3 client for 'secretsmanager' with ID 4751659648
2025-01-08 16:39:02,495 INFO pool_2 azul.azulclient: Making POST request to 'https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add'
2025-01-08 16:39:02,495 DEBUG pool_2 azul.azulclient: β¦ with request body b'{}'
2025-01-08 16:39:02,730 INFO pool_2 azul.azulclient: Got 400 response after 0.234s from POST to https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add
2025-01-08 16:39:02,730 DEBUG pool_2 azul.azulclient: β¦ with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '78', 'Connection': 'keep-alive', 'Date': 'Thu, 09 Jan 2025 00:39:02 GMT', 'X-Amzn-Trace-Id': 'Root=1-677f1aa6-14a1f87b661f8eb8566e0a36;Parent=5ccdba6c8ddad289;Sampled=0;Lineage=1:ba9bd050:0', 'x-amzn-RequestId': '70168e0a-c03e-425b-ac85-849acac1a54a', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", 'x-amz-apigw-id': 'EGEaGE0jIAMEOPA=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 e84fdc4f831d7aaaedd48d30a9d5a068.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'SFO5-C3', 'X-Amz-Cf-Id': 'c66V4FzTSXRLsY0dckGhud4j1JFs-N5ciMnFxZ732bZtzdUp9bNuGA=='})
2025-01-08 16:39:02,730 DEBUG pool_2 azul.azulclient: β¦ with response body b'{"Code":"BadRequestError","Message":"Missing notification entry: bundle_fqid"}'
2025-01-08 16:39:02,730 WARNING pool_2 azul.azulclient: Failed to notify https://indexer.daniel2.anvil.gi.ucsc.edu/anvil-it/add about {}, attempt 3: after error .
2025-01-08 16:39:02,731 INFO MainThread azul.azulclient: Sent notifications for 0 of 1 bundles for catalog 'anvil-it'.
2025-01-08 16:39:02,732 ERROR MainThread azul.azulclient: Number of errors by HTTP status code:
{400: 1}
2025-01-08 16:39:02,732 ERROR MainThread azul.azulclient: Unsent notifications and their HTTP status code:
[({}, 400)]
ok
test_can_bundle_canned_repository (integration_test.CanBundleScriptIntegrationTest.test_can_bundle_canned_repository) ... 2025-01-08 16:39:02,736 INFO MainThread test.integration_test: Using random seed 2162954277956149835
2025-01-08 16:39:02,808 INFO MainThread azul.azulclient: Listing bundles with prefix '' in source CannedSourceRef(id='https://github.com/HumanCellAtlas/schema-test-data/tree/master/tests:/0', spec=SimpleSourceSpec(prefix=Prefix(common='', partition=0), name='https://github.com/HumanCellAtlas/schema-test-data/tree/master/tests')).
2025-01-08 16:39:10,359 INFO MainThread azul.azulclient: There are 7 bundle(s) with prefix '' in source CannedSourceRef(id='https://github.com/HumanCellAtlas/schema-test-data/tree/master/tests:/0', spec=SimpleSourceSpec(prefix=Prefix(common='', partition=0), name='https://github.com/HumanCellAtlas/schema-test-data/tree/master/tests')).
2025-01-08 16:39:10,359 INFO MainThread test.integration_test: Canning bundle CannedBundleFQID(uuid='1ed68210-eaba-531d-ba9e-db80164d65ef', version='2021-01-01T00:00:00.000000Z', source=CannedSourceRef(id='https://github.com/HumanCellAtlas/schema-test-data/tree/master/tests:/0', spec=SimpleSourceSpec(prefix=Prefix(common='', partition=0), name='https://github.com/HumanCellAtlas/schema-test-data/tree/master/tests'))) from catalog 'canned-it'
2025-01-08 16:39:10,363 INFO MainThread azul.plugins.repository.canned: It took 0.000s to download bundle 1ed68210-eaba-531d-ba9e-db80164d65ef.2021-01-01T00:00:00.000000Z
2025-01-08 16:39:10,363 INFO MainThread can_bundle: Fetched bundle '1ed68210-eaba-531d-ba9e-db80164d65ef' version '2021-01-01T00:00:00.000000Z' from catalog 'canned-it'.
2025-01-08 16:39:10,365 INFO MainThread can_bundle: Successfully wrote /var/folders/d1/1lnk2yfj2tj93pyr9_0mx7w00000gn/T/tmp6cuhs63t/1ed68210-eaba-531d-ba9e-db80164d65ef.gh.hca.json
ok
test_can_bundle_configured_catalogs (integration_test.CanBundleScriptIntegrationTest.test_can_bundle_configured_catalogs) ... 2025-01-08 16:39:10,366 INFO MainThread test.integration_test: Using random seed 2720170117190454623
2025-01-08 16:39:10,371 INFO MainThread azul.deployment: Allocated new Boto3 client for 'secretsmanager' with ID 4752021296
2025-01-08 16:39:10,862 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CMG_Sample_1_20230225_ANV5_202302281509&limit=2'
2025-01-08 16:39:10,862 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:39:11,343 INFO MainThread azul.terra: Got 200 response after 0.481s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CMG_Sample_1_20230225_ANV5_202302281509&limit=2
2025-01-08 16:39:11,344 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:39:11 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'aE4WNGBZ', 'Content-Type': 'application/json', 'Content-Length': '908', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:39:11,344 DEBUG MainThread azul.terra: β¦ with response body b'{"total":156,"filteredTotal":1,"items":[{"id":"7d632e2a-54f1-4021-81fd-dbaf893715cd","name":"ANVIL_CMG_Sample_1_20230225_ANV5...'
2025-01-08 16:39:11,344 INFO MainThread azul.azulclient: Listing bundles with prefix '' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=0), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:39:11,345 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:39:12,338 DEBUG MainThread azul.terra: Job info: {"job_id": "ecf0ca9e-19ce-4ef6-a64b-ce9c96f5ccfb", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:39:12,519 DEBUG MainThread azul.terra: Query (198 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:39:13,755 DEBUG MainThread azul.terra: Job info: {"job_id": "6cfa430e-5f25-4e63-879d-ac5406985a29", "total_rows": 25, "stats": {"estimatedBytesProcessed": "27702", "timeline": [{"elapsedMs": "305", "totalSlotMs": "108", "pendingUnits": "0", "completedUnits": "2", "estimatedRunnableUnits": "0"}], "totalPartitionsProcessed": "0", "totalBytesProcessed": "27702", "totalBytesBilled": "20971520", "billingTier": 1, "totalSlotMs": "108", "cacheHit": false, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "NOT_SUPPORTED_IN_STANDARD_EDITION", "message": "Index can not be used for query with Standard edition reservation. See https://cloud.google.com/bigquery/docs/editions-intro for more information."}]}, "transferredBytes": "0", "metadataCacheStatistics": {"tableMetadataCacheUsage": [{"tableReference": {"projectId": "datarepo-dev-067d6669", "datasetId": "datarepo_ANVIL_CMG_Sample_1_20230225", "tableId": "datarepo_raw_anvil_biosample_113eabbe_cf77_42a4_be99_bc3148948944"}, "unusedReason": "OTHER_REASON", "explanation": "Table does not have CMETA."}, {"tableReference": {"projectId": "datarepo-dev-97ad270b", "datasetId": "ANVIL_CMG_Sample_1_20230225_ANV5_202302281509", "tableId": "datarepo_row_ids"}, "unusedReason": "OTHER_REASON", "explanation": "Table does not have CMETA."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:39:14,151 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/0
2025-01-08 16:39:14,151 DEBUG MainThread azul.terra: Query (16969 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER β¦'
2025-01-08 16:39:15,359 DEBUG MainThread azul.terra: Job info: {"job_id": "0093f546-9781-47c1-9b59-c80ec8a53796", "total_rows": 10, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER \u2026"}
2025-01-08 16:39:15,566 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:39:15,566 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'file_inventory' (average batch size 50.0, num batches 1)
2025-01-08 16:39:15,566 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_sequencingactivity' (average batch size 50.0, num batches 1)
2025-01-08 16:39:15,566 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'subject' (average batch size 25.0, num batches 1)
2025-01-08 16:39:15,566 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_donor' (average batch size 25.0, num batches 1)
2025-01-08 16:39:15,567 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 37.0, num batches 1)
2025-01-08 16:39:15,567 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sequencing' (average batch size 50.0, num batches 1)
2025-01-08 16:39:15,567 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sample' (average batch size 25.0, num batches 1)
2025-01-08 16:39:15,567 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_file' (average batch size 50.0, num batches 1)
2025-01-08 16:39:15,567 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'family' (average batch size 13.0, num batches 1)
2025-01-08 16:39:15,569 INFO MainThread azul.azulclient: There are 36 bundle(s) with prefix '' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=0), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:39:15,570 INFO MainThread test.integration_test: Canning bundle TDRAnvilBundleFQID(uuid='70ce1a12-3f7d-a49c-968e-5095abc16f6c', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=0), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sequencing', batch_prefix='') from catalog 'anvil-it'
2025-01-08 16:39:15,574 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CMG_Sample_1_20230225_ANV5_202302281509&limit=2'
2025-01-08 16:39:15,574 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:39:15,965 INFO MainThread azul.terra: Got 200 response after 0.391s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_CMG_Sample_1_20230225_ANV5_202302281509&limit=2
2025-01-08 16:39:15,965 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:39:15 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': '6qWO5jLE', 'Content-Type': 'application/json', 'Content-Length': '908', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:39:15,965 DEBUG MainThread azul.terra: β¦ with response body b'{"total":156,"filteredTotal":1,"items":[{"id":"7d632e2a-54f1-4021-81fd-dbaf893715cd","name":"ANVIL_CMG_Sample_1_20230225_ANV5...'
2025-01-08 16:39:15,966 INFO MainThread azul.plugins.repository.tdr_anvil: Bundle '70ce1a12-3f7d-a49c-968e-5095abc16f6c' is a replica bundle
2025-01-08 16:39:15,966 DEBUG MainThread azul.terra: Query (294 characters total): '\n SELECT consent_group, data_modality, data_use_permission, datarepo_row_id, dataset_id, owner, principal_investigator, registered_identifier, source_datarepo_row_ids, title\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:39:16,849 DEBUG MainThread azul.terra: Job info: {"job_id": "cb6899c4-001b-4270-82ab-73c049a8fe39", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT consent_group, data_modality, data_use_permission, datarepo_row_id, dataset_id, owner, principal_investigator, registered_identifier, source_datarepo_row_ids, title\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:39:17,098 DEBUG MainThread azul.terra: Query (179 characters total): "\n SELECT *\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.sequencing`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:39:18,330 DEBUG MainThread azul.terra: Job info: {"job_id": "9e3b8600-8f9f-48aa-b896-83a3361bdf9d", "total_rows": 50, "stats": {"estimatedBytesProcessed": "45583", "timeline": [{"elapsedMs": "242", "totalSlotMs": "58", "pendingUnits": "0", "completedUnits": "2", "estimatedRunnableUnits": "0"}], "totalPartitionsProcessed": "0", "totalBytesProcessed": "45583", "totalBytesBilled": "20971520", "billingTier": 1, "totalSlotMs": "58", "cacheHit": false, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "NOT_SUPPORTED_IN_STANDARD_EDITION", "message": "Index can not be used for query with Standard edition reservation. See https://cloud.google.com/bigquery/docs/editions-intro for more information."}]}, "transferredBytes": "0", "metadataCacheStatistics": {"tableMetadataCacheUsage": [{"tableReference": {"projectId": "datarepo-dev-97ad270b", "datasetId": "ANVIL_CMG_Sample_1_20230225_ANV5_202302281509", "tableId": "datarepo_row_ids"}, "unusedReason": "OTHER_REASON", "explanation": "Table does not have CMETA."}, {"tableReference": {"projectId": "datarepo-dev-067d6669", "datasetId": "datarepo_ANVIL_CMG_Sample_1_20230225", "tableId": "datarepo_raw_sequencing_e377a987_baa8_4684_8cbe_0f051acb0a12"}, "unusedReason": "OTHER_REASON", "explanation": "Table does not have CMETA."}]}}, "query": "\n SELECT *\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.sequencing`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:39:18,510 INFO MainThread azul.plugins.repository.tdr: It took 2.545s to download bundle 70ce1a12-3f7d-a49c-968e-5095abc16f6c.2022-06-01T00:00:00.000000Z
2025-01-08 16:39:18,510 INFO MainThread can_bundle: Fetched bundle '70ce1a12-3f7d-a49c-968e-5095abc16f6c' version '2022-06-01T00:00:00.000000Z' from catalog 'anvil'.
2025-01-08 16:39:18,513 INFO MainThread can_bundle: Successfully wrote /var/folders/d1/1lnk2yfj2tj93pyr9_0mx7w00000gn/T/tmp7kygtva2/70ce1a12-3f7d-a49c-968e-5095abc16f6c.tdr.anvil.json
ok
test_version (integration_test.DeployedVersionIntegrationTest.test_version) ... ok
test (integration_test.DisableAutomaticIndexCreationTest.test) ... 2025-01-08 16:39:19,202 INFO MainThread test.integration_test: Using random seed 6552993134935875464
2025-01-08 16:39:19,202 INFO MainThread elasticsearch: Making POST request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/no-auto-create-51e41f79-it/_doc
2025-01-08 16:39:19,202 INFO MainThread elasticsearch: β¦ with request body b'{"foo":"bar"}'
2025-01-08 16:39:19,289 WARNING MainThread elasticsearch: Got 404 response after 0.087s from POST to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/no-auto-create-51e41f79-it/_doc
2025-01-08 16:39:19,289 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [no-auto-create-51e41f79-it]","resource.typβ¦'
2025-01-08 16:39:19,290 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/no-auto-create-51e41f79-it
2025-01-08 16:39:19,290 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:19,377 INFO MainThread elasticsearch: Got 404 response after 0.087s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/no-auto-create-51e41f79-it
2025-01-08 16:39:19,377 INFO MainThread elasticsearch: β¦ with response body ''
ok
test_catalog_listing (integration_test.IndexingIntegrationTest.test_catalog_listing) ... 2025-01-08 16:39:19,381 INFO MainThread test.integration_test: Using random seed 6718435747472879302
2025-01-08 16:39:19,382 INFO MainThread test.integration_test: Making GET request to 'https://service.daniel2.anvil.gi.ucsc.edu/index/catalogs'
2025-01-08 16:39:19,382 DEBUG MainThread test.integration_test: β¦ without request body
2025-01-08 16:39:19,666 INFO MainThread test.integration_test: Got 200 response after 0.284s from GET to https://service.daniel2.anvil.gi.ucsc.edu/index/catalogs
2025-01-08 16:39:19,666 DEBUG MainThread test.integration_test: β¦ with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '1696', 'Connection': 'keep-alive', 'Date': 'Thu, 09 Jan 2025 00:39:19 GMT', 'X-Amzn-Trace-Id': 'Root=1-677f1ab7-53193929663b2ae06a5784a9;Parent=50dcb8f2af5ad620;Sampled=0;Lineage=1:26435425:0', 'x-amzn-RequestId': 'aa14691d-456d-448c-898b-4f6ff3ab97ba', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", 'x-amz-apigw-id': 'EGEcvFc2IAMEGsw=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 3fa44d644afdb8749d0c126169e4edfc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'SFO53-P5', 'X-Amz-Cf-Id': 'BAhTvw5zrTXSl_kRFGXTGXvt92PkZISxrGecEA64ZglxFH0b3PbuiQ=='})
2025-01-08 16:39:19,666 DEBUG MainThread test.integration_test: β¦ with response body b'{"default_catalog":"anvil","catalogs":{"anvil":{"internal":false,"atlas":"anvil","plugins":{"metadata":{"name":"anvil","indic...'
ok
test_indexing (integration_test.IndexingIntegrationTest.test_indexing) ... 2025-01-08 16:39:19,667 INFO MainThread test.integration_test: Using random seed 510654809212837387
2025-01-08 16:39:20,355 INFO MainThread azul.azulclient: Disabling lambdas ...
2025-01-08 16:39:20,384 INFO MainThread azul.deployment: Allocated new Boto3 client for 'iam' with ID 4760008080
2025-01-08 16:39:20,824 INFO ThreadPoolExecutor-0_0 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:20,824 INFO ThreadPoolExecutor-0_2 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:20,824 INFO ThreadPoolExecutor-0_3 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:20,824 INFO ThreadPoolExecutor-0_1 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:21,090 INFO ThreadPoolExecutor-0_0 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4788043696
2025-01-08 16:39:21,092 INFO ThreadPoolExecutor-0_1 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4788029488
2025-01-08 16:39:21,093 INFO ThreadPoolExecutor-0_3 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4787802416
2025-01-08 16:39:21,095 INFO ThreadPoolExecutor-0_2 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4788272016
2025-01-08 16:39:21,502 INFO ThreadPoolExecutor-0_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Enabled'.
2025-01-08 16:39:21,502 INFO ThreadPoolExecutor-0_1 azul.queues: Disabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute'
2025-01-08 16:39:21,560 INFO ThreadPoolExecutor-0_0 azul.lambdas: Setting concurrency limit for azul-indexer-daniel2 to zero.
2025-01-08 16:39:21,775 INFO ThreadPoolExecutor-0_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Enabled'.
2025-01-08 16:39:21,775 INFO ThreadPoolExecutor-0_2 azul.queues: Disabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry'
2025-01-08 16:39:21,778 INFO ThreadPoolExecutor-0_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Enabled'.
2025-01-08 16:39:21,780 INFO ThreadPoolExecutor-0_3 azul.queues: Disabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate'
2025-01-08 16:39:22,038 INFO ThreadPoolExecutor-0_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Enabled'.
2025-01-08 16:39:22,038 INFO ThreadPoolExecutor-0_0 azul.queues: Disabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry'
2025-01-08 16:39:24,841 INFO ThreadPoolExecutor-0_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Disabling'.
2025-01-08 16:39:25,127 INFO ThreadPoolExecutor-0_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Disabling'.
2025-01-08 16:39:25,129 INFO ThreadPoolExecutor-0_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Disabling'.
2025-01-08 16:39:25,382 INFO ThreadPoolExecutor-0_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Disabling'.
2025-01-08 16:39:28,027 INFO ThreadPoolExecutor-0_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Disabling'.
2025-01-08 16:39:28,287 INFO ThreadPoolExecutor-0_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Disabling'.
2025-01-08 16:39:28,294 INFO ThreadPoolExecutor-0_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Disabled'.
2025-01-08 16:39:28,552 INFO ThreadPoolExecutor-0_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Disabling'.
2025-01-08 16:39:31,194 INFO ThreadPoolExecutor-0_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Disabling'.
2025-01-08 16:39:31,443 INFO ThreadPoolExecutor-0_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Disabled'.
2025-01-08 16:39:31,713 INFO ThreadPoolExecutor-0_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Disabling'.
2025-01-08 16:39:34,352 INFO ThreadPoolExecutor-0_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Disabled'.
2025-01-08 16:39:34,886 INFO ThreadPoolExecutor-0_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Disabled'.
2025-01-08 16:39:34,888 INFO MainThread azul.azulclient: Purging queues: azul-notifications-daniel2, azul-notifications_retry-daniel2, azul-tallies-daniel2.fifo, azul-tallies_retry-daniel2.fifo
2025-01-08 16:39:34,888 INFO ThreadPoolExecutor-1_0 azul.queues: Purging queue 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2'
2025-01-08 16:39:34,889 INFO ThreadPoolExecutor-1_1 azul.queues: Purging queue 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2'
2025-01-08 16:39:34,892 INFO ThreadPoolExecutor-1_2 azul.queues: Purging queue 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo'
2025-01-08 16:39:34,892 INFO ThreadPoolExecutor-1_3 azul.queues: Purging queue 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo'
2025-01-08 16:39:35,396 INFO MainThread azul.azulclient: Deleting indices ...
2025-01-08 16:39:35,397 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:35,397 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:35,488 INFO MainThread elasticsearch: Got 200 response after 0.090s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:35,488 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:35,488 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:35,488 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:35,612 INFO MainThread elasticsearch: Got 200 response after 0.124s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:35,612 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:35,612 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:35,612 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:35,704 INFO MainThread elasticsearch: Got 200 response after 0.092s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:35,704 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:35,705 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:35,705 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:35,826 INFO MainThread elasticsearch: Got 200 response after 0.121s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:35,826 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:35,826 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:35,827 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:35,916 INFO MainThread elasticsearch: Got 200 response after 0.089s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:35,916 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:35,916 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:35,916 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,045 INFO MainThread elasticsearch: Got 200 response after 0.128s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:36,045 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:36,045 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:36,045 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,136 INFO MainThread elasticsearch: Got 200 response after 0.090s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:36,136 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:36,136 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:36,136 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,254 INFO MainThread elasticsearch: Got 200 response after 0.118s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:36,254 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:36,254 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:36,254 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,344 INFO MainThread elasticsearch: Got 200 response after 0.089s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:36,344 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:36,344 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:36,344 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,467 INFO MainThread elasticsearch: Got 200 response after 0.123s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:36,467 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:36,467 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:36,467 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,558 INFO MainThread elasticsearch: Got 200 response after 0.091s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:36,558 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:36,559 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:36,559 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,680 INFO MainThread elasticsearch: Got 200 response after 0.121s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:36,681 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:36,681 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:36,681 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,772 INFO MainThread elasticsearch: Got 200 response after 0.092s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:36,773 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:36,773 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:36,773 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,895 INFO MainThread elasticsearch: Got 200 response after 0.122s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:36,895 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:36,896 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:36,896 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:36,989 INFO MainThread elasticsearch: Got 200 response after 0.093s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:36,989 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:36,989 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:36,989 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,113 INFO MainThread elasticsearch: Got 200 response after 0.124s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:37,113 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:37,113 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:37,113 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,203 INFO MainThread elasticsearch: Got 200 response after 0.090s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:37,203 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:37,203 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:37,203 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,327 INFO MainThread elasticsearch: Got 200 response after 0.124s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:37,327 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:37,327 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:37,327 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,420 INFO MainThread elasticsearch: Got 200 response after 0.093s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:37,420 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:37,421 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:37,421 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,542 INFO MainThread elasticsearch: Got 200 response after 0.121s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:37,542 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:37,542 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:37,542 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,635 INFO MainThread elasticsearch: Got 200 response after 0.093s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:37,636 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:37,636 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:37,636 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,760 INFO MainThread elasticsearch: Got 200 response after 0.124s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:37,760 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:37,760 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:37,760 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,851 INFO MainThread elasticsearch: Got 200 response after 0.091s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:37,851 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:37,851 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:37,851 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:37,977 INFO MainThread elasticsearch: Got 200 response after 0.126s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:37,977 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:37,978 INFO MainThread elasticsearch: Making HEAD request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:37,978 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:38,069 INFO MainThread elasticsearch: Got 200 response after 0.091s from HEAD to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:38,069 INFO MainThread elasticsearch: β¦ with response body ''
2025-01-08 16:39:38,069 INFO MainThread elasticsearch: Making DELETE request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:38,069 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:38,225 INFO MainThread elasticsearch: Got 200 response after 0.155s from DELETE to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:38,225 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true}'
2025-01-08 16:39:38,225 INFO MainThread azul.azulclient: Re-enabling lambdas ...
2025-01-08 16:39:38,265 INFO ThreadPoolExecutor-2_1 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:38,265 INFO ThreadPoolExecutor-2_0 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:38,265 INFO ThreadPoolExecutor-2_2 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:38,265 INFO ThreadPoolExecutor-2_3 botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2025-01-08 16:39:38,406 INFO ThreadPoolExecutor-2_1 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4818685584
2025-01-08 16:39:38,407 INFO ThreadPoolExecutor-2_2 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4791071136
2025-01-08 16:39:38,409 INFO ThreadPoolExecutor-2_3 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4818918176
2025-01-08 16:39:38,410 INFO ThreadPoolExecutor-2_0 azul.deployment: Allocated new Boto3 client for 'lambda' with ID 4819131936
2025-01-08 16:39:38,716 INFO ThreadPoolExecutor-2_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Disabled'.
2025-01-08 16:39:38,716 INFO ThreadPoolExecutor-2_2 azul.queues: Enabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry'
2025-01-08 16:39:38,720 INFO ThreadPoolExecutor-2_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Disabled'.
2025-01-08 16:39:38,721 INFO ThreadPoolExecutor-2_3 azul.queues: Enabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate'
2025-01-08 16:39:38,722 INFO ThreadPoolExecutor-2_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Disabled'.
2025-01-08 16:39:38,727 INFO ThreadPoolExecutor-2_1 azul.queues: Enabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute'
2025-01-08 16:39:38,887 INFO ThreadPoolExecutor-2_0 azul.lambdas: Removed concurrency limit for azul-indexer-daniel2.
2025-01-08 16:39:39,255 INFO ThreadPoolExecutor-2_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Disabled'.
2025-01-08 16:39:39,255 INFO ThreadPoolExecutor-2_0 azul.queues: Enabling push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry'
2025-01-08 16:39:42,065 INFO ThreadPoolExecutor-2_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Enabling'.
2025-01-08 16:39:42,077 INFO ThreadPoolExecutor-2_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Enabling'.
2025-01-08 16:39:42,098 INFO ThreadPoolExecutor-2_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Enabling'.
2025-01-08 16:39:42,617 INFO ThreadPoolExecutor-2_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Enabling'.
2025-01-08 16:39:45,226 INFO ThreadPoolExecutor-2_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Enabling'.
2025-01-08 16:39:45,236 INFO ThreadPoolExecutor-2_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Enabling'.
2025-01-08 16:39:45,252 INFO ThreadPoolExecutor-2_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Enabling'.
2025-01-08 16:39:45,765 INFO ThreadPoolExecutor-2_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Enabling'.
2025-01-08 16:39:48,387 INFO ThreadPoolExecutor-2_2 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications_retry-daniel2' to lambda function 'azul-indexer-daniel2-contribute_retry' is in state 'Enabled'.
2025-01-08 16:39:48,396 INFO ThreadPoolExecutor-2_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Enabling'.
2025-01-08 16:39:48,419 INFO ThreadPoolExecutor-2_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Enabling'.
2025-01-08 16:39:48,912 INFO ThreadPoolExecutor-2_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Enabling'.
2025-01-08 16:39:51,571 INFO ThreadPoolExecutor-2_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Enabling'.
2025-01-08 16:39:51,589 INFO ThreadPoolExecutor-2_3 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate' is in state 'Enabled'.
2025-01-08 16:39:52,069 INFO ThreadPoolExecutor-2_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Enabling'.
2025-01-08 16:39:54,731 INFO ThreadPoolExecutor-2_1 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-notifications-daniel2' to lambda function 'azul-indexer-daniel2-contribute' is in state 'Enabled'.
2025-01-08 16:39:55,235 INFO ThreadPoolExecutor-2_0 azul.queues: Push from 'https://sqs.us-east-1.amazonaws.com/289950828509/azul-tallies_retry-daniel2.fifo' to lambda function 'azul-indexer-daniel2-aggregate_retry' is in state 'Enabled'.
2025-01-08 16:39:55,236 INFO MainThread azul.azulclient: Creating indices ...
2025-01-08 16:39:55,237 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:55,237 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:55,328 WARNING MainThread elasticsearch: Got 404 response after 0.091s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:55,328 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_activities]","indβ¦'
2025-01-08 16:39:55,329 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:55,329 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:55,488 INFO MainThread elasticsearch: Got 200 response after 0.158s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:55,488 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_activities"}'
2025-01-08 16:39:55,489 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:55,489 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:55,579 INFO MainThread elasticsearch: Got 200 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities
2025-01-08 16:39:55,579 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_activities":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":β¦'
2025-01-08 16:39:55,580 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:55,581 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:55,672 WARNING MainThread elasticsearch: Got 404 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:55,672 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_activities_aggregβ¦'
2025-01-08 16:39:55,673 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:55,673 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:55,864 INFO MainThread elasticsearch: Got 200 response after 0.191s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:55,864 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_activities_aggregate"}'
2025-01-08 16:39:55,865 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:55,865 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:55,960 INFO MainThread elasticsearch: Got 200 response after 0.095s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_activities_aggregate
2025-01-08 16:39:55,960 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_activities_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappβ¦'
2025-01-08 16:39:55,961 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:55,961 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:56,055 WARNING MainThread elasticsearch: Got 404 response after 0.094s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:56,055 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_biosamples]","indβ¦'
2025-01-08 16:39:56,056 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:56,056 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:56,224 INFO MainThread elasticsearch: Got 200 response after 0.168s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:56,224 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_biosamples"}'
2025-01-08 16:39:56,224 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:56,224 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:56,314 INFO MainThread elasticsearch: Got 200 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples
2025-01-08 16:39:56,314 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_biosamples":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":β¦'
2025-01-08 16:39:56,315 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:56,315 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:56,405 WARNING MainThread elasticsearch: Got 404 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:56,405 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_biosamples_aggregβ¦'
2025-01-08 16:39:56,405 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:56,405 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:56,559 INFO MainThread elasticsearch: Got 200 response after 0.154s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:56,560 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_biosamples_aggregate"}'
2025-01-08 16:39:56,560 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:56,560 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:56,655 INFO MainThread elasticsearch: Got 200 response after 0.094s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_biosamples_aggregate
2025-01-08 16:39:56,655 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_biosamples_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappβ¦'
2025-01-08 16:39:56,656 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:56,656 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:56,746 WARNING MainThread elasticsearch: Got 404 response after 0.089s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:56,746 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_bundles]","index_β¦'
2025-01-08 16:39:56,746 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:56,746 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:56,908 INFO MainThread elasticsearch: Got 200 response after 0.161s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:56,908 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_bundles"}'
2025-01-08 16:39:56,908 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:56,908 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:56,999 INFO MainThread elasticsearch: Got 200 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles
2025-01-08 16:39:56,999 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_bundles":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"stβ¦'
2025-01-08 16:39:56,999 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:56,999 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:57,092 WARNING MainThread elasticsearch: Got 404 response after 0.093s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:57,092 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_bundles_aggregateβ¦'
2025-01-08 16:39:57,092 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:57,092 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:57,244 INFO MainThread elasticsearch: Got 200 response after 0.152s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:57,244 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_bundles_aggregate"}'
2025-01-08 16:39:57,245 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:57,245 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:57,336 INFO MainThread elasticsearch: Got 200 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_bundles_aggregate
2025-01-08 16:39:57,337 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_bundles_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappingβ¦'
2025-01-08 16:39:57,338 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:57,338 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:57,430 WARNING MainThread elasticsearch: Got 404 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:57,431 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_datasets]","resouβ¦'
2025-01-08 16:39:57,431 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:57,431 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:57,593 INFO MainThread elasticsearch: Got 200 response after 0.162s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:57,594 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_datasets"}'
2025-01-08 16:39:57,594 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:57,594 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:57,686 INFO MainThread elasticsearch: Got 200 response after 0.091s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets
2025-01-08 16:39:57,686 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_datasets":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"sβ¦'
2025-01-08 16:39:57,687 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:57,687 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:57,784 WARNING MainThread elasticsearch: Got 404 response after 0.097s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:57,785 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_datasets_aggregatβ¦'
2025-01-08 16:39:57,785 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:57,785 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:57,979 INFO MainThread elasticsearch: Got 200 response after 0.194s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:57,979 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_datasets_aggregate"}'
2025-01-08 16:39:57,980 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:57,980 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:58,073 INFO MainThread elasticsearch: Got 200 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_datasets_aggregate
2025-01-08 16:39:58,073 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_datasets_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mappinβ¦'
2025-01-08 16:39:58,074 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:58,074 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:58,164 WARNING MainThread elasticsearch: Got 404 response after 0.091s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:58,164 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_donors]","resourcβ¦'
2025-01-08 16:39:58,165 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:58,165 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:58,339 INFO MainThread elasticsearch: Got 200 response after 0.174s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:58,339 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_donors"}'
2025-01-08 16:39:58,340 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:58,340 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:58,430 INFO MainThread elasticsearch: Got 200 response after 0.089s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors
2025-01-08 16:39:58,430 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_donors":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"strβ¦'
2025-01-08 16:39:58,431 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:58,431 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:58,523 WARNING MainThread elasticsearch: Got 404 response after 0.091s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:58,523 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_donors_aggregate]β¦'
2025-01-08 16:39:58,523 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:58,524 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:58,685 INFO MainThread elasticsearch: Got 200 response after 0.161s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:58,685 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_donors_aggregate"}'
2025-01-08 16:39:58,686 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:58,686 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:58,779 INFO MainThread elasticsearch: Got 200 response after 0.093s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_donors_aggregate
2025-01-08 16:39:58,779 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_donors_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_β¦'
2025-01-08 16:39:58,780 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:58,780 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:58,869 WARNING MainThread elasticsearch: Got 404 response after 0.089s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:58,869 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_files]","resourceβ¦'
2025-01-08 16:39:58,870 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:58,870 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:59,031 INFO MainThread elasticsearch: Got 200 response after 0.161s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:59,031 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_files"}'
2025-01-08 16:39:59,032 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:59,032 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:59,121 INFO MainThread elasticsearch: Got 200 response after 0.089s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files
2025-01-08 16:39:59,122 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_files":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"striβ¦'
2025-01-08 16:39:59,123 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:59,123 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:59,215 WARNING MainThread elasticsearch: Got 404 response after 0.092s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:59,215 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_files_aggregate]"β¦'
2025-01-08 16:39:59,215 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:59,215 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:59,369 INFO MainThread elasticsearch: Got 200 response after 0.153s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:59,369 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_files_aggregate"}'
2025-01-08 16:39:59,370 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:59,370 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:59,463 INFO MainThread elasticsearch: Got 200 response after 0.093s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_files_aggregate
2025-01-08 16:39:59,463 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_files_aggregate":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_tβ¦'
2025-01-08 16:39:59,464 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:59,464 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:59,554 WARNING MainThread elasticsearch: Got 404 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:59,554 WARNING MainThread elasticsearch: β¦ with response body '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [azul_v2_daniel2_anvil-it_replica]","resourβ¦'
2025-01-08 16:39:59,555 INFO MainThread elasticsearch: Making PUT request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:59,555 INFO MainThread elasticsearch: β¦ with request body b'{"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"refresh_interval":"1s"}},"mappings":{"numeric_detection":f...'
2025-01-08 16:39:59,766 INFO MainThread elasticsearch: Got 200 response after 0.211s from PUT to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:59,767 INFO MainThread elasticsearch: β¦ with response body '{"acknowledged":true,"shards_acknowledged":true,"index":"azul_v2_daniel2_anvil-it_replica"}'
2025-01-08 16:39:59,767 INFO MainThread elasticsearch: Making GET request to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:59,767 INFO MainThread elasticsearch: β¦ without request body
2025-01-08 16:39:59,858 INFO MainThread elasticsearch: Got 200 response after 0.090s from GET to https://vpc-azul-index-anvilbox-y7lvuxpx7jaryr6rpxubm6ey6u.us-east-1.es.amazonaws.com:443/azul_v2_daniel2_anvil-it_replica
2025-01-08 16:39:59,858 INFO MainThread elasticsearch: β¦ with response body '{"azul_v2_daniel2_anvil-it_replica":{"aliases":{},"mappings":{"dynamic_templates":[{"strings_as_text":{"match_mapping_type":"stβ¦'
2025-01-08 16:39:59,860 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?offset=0&limit=1000&sort=created_date&direction=asc'
2025-01-08 16:39:59,860 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:40:00,665 INFO MainThread azul.terra: Got 200 response after 0.806s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?offset=0&limit=1000&sort=created_date&direction=asc
2025-01-08 16:40:00,666 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:40:00 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'eXEVKRJ7', 'Content-Type': 'application/json', 'Content-Length': '134709', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:40:00,666 DEBUG MainThread azul.terra: β¦ with response body b'{"total":153,"filteredTotal":153,"items":[{"id":"0c86170e-312d-4b39-a0a4-2a2bfaa24c7a","name":"CromwellTestSnapshotSimpleAfte...'
2025-01-08 16:40:00,669 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?offset=0&limit=1000&sort=created_date&direction=asc'
2025-01-08 16:40:00,669 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:40:00,973 INFO MainThread azul.terra: Got 200 response after 0.305s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?offset=0&limit=1000&sort=created_date&direction=asc
2025-01-08 16:40:00,974 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:40:00 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': '6vW5rEOE', 'Content-Type': 'application/json', 'Content-Length': '137284', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:40:00,974 DEBUG MainThread azul.terra: β¦ with response body b'{"total":156,"filteredTotal":156,"items":[{"id":"0c86170e-312d-4b39-a0a4-2a2bfaa24c7a","name":"CromwellTestSnapshotSimpleAfte...'
2025-01-08 16:40:00,979 INFO MainThread azul.terra: Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732&limit=2'
2025-01-08 16:40:00,979 DEBUG MainThread azul.terra: β¦ without request body
2025-01-08 16:40:01,204 INFO MainThread azul.terra: Got 200 response after 0.224s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots?filter=ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732&limit=2
2025-01-08 16:40:01,204 DEBUG MainThread azul.terra: β¦ with response headers HTTPHeaderDict({'Date': 'Thu, 09 Jan 2025 00:40:01 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'eX0lBydW', 'Content-Type': 'application/json', 'Content-Length': '903', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-08 16:40:01,205 DEBUG MainThread azul.terra: β¦ with response body b'{"total":156,"filteredTotal":1,"items":[{"id":"790795c4-49b1-4ac8-a060-207b92ea08c5","name":"ANVIL_1000G_2019_Dev_20230609_AN...'
2025-01-08 16:40:01,206 DEBUG MainThread azul.terra: Query (202 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:40:02,241 DEBUG MainThread azul.terra: Job info: {"job_id": "85e54fe3-a499-4735-8336-0fe95a39b108", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:40:02,666 DEBUG MainThread azul.terra: Query (200 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:40:03,548 DEBUG MainThread azul.terra: Job info: {"job_id": "4ab7e8c7-825a-4e21-9107-883c0f86a353", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:40:04,100 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:
2025-01-08 16:40:04,101 DEBUG MainThread azul.terra: Query (17025 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORβ¦'
2025-01-08 16:40:05,328 DEBUG MainThread azul.terra: Job info: {"job_id": "5abafae9-e794-48e4-9c15-622ea28e8bb4", "total_rows": 10, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n OR\u2026"}
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 24.0, num batches 1)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'qc_result_sample' (average batch size 199.0, num batches 16)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'participant' (average batch size 200.1, num batches 16)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'pedigree' (average batch size 200.1, num batches 16)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'file_inventory' (average batch size 101.6, num batches 256)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 200.1, num batches 16)
2025-01-08 16:40:05,534 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_activity' (average batch size 101.0, num batches 256)
2025-01-08 16:40:05,535 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 200.1, num batches 16)
2025-01-08 16:40:05,535 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 2 for table 'anvil_file' (average batch size 101.6, num batches 256)
2025-01-08 16:40:05,535 INFO MainThread azul.azulclient: Listing bundles with prefix '650' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:05,536 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:06,557 DEBUG MainThread azul.terra: Job info: {"job_id": "167d5209-3079-4a89-a6b9-be3bbdad9f8c", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:06,709 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '650')\n "
2025-01-08 16:40:07,584 DEBUG MainThread azul.terra: Job info: {"job_id": "3e262a17-2bc6-482d-aeca-b339be34352f", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '650')\n "}
2025-01-08 16:40:07,985 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '650' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:07,985 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'650\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'650\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:08,976 DEBUG MainThread azul.terra: Job info: {"job_id": "1b735609-8299-47b8-b77d-51383533be42", "total_rows": 4, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '650')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '650')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:09,154 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 5.0, num batches 1)
2025-01-08 16:40:09,154 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 7.0, num batches 1)
2025-01-08 16:40:09,154 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 1.0, num batches 1)
2025-01-08 16:40:09,154 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:40:09,155 INFO MainThread azul.azulclient: There are 4 bundle(s) with prefix '650' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:09,155 INFO MainThread azul.azulclient: Listing bundles with prefix '651' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:09,156 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:10,038 DEBUG MainThread azul.terra: Job info: {"job_id": "3746b106-efec-4fc8-99f3-fb679e483392", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:10,221 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '651')\n "
2025-01-08 16:40:11,055 DEBUG MainThread azul.terra: Job info: {"job_id": "2c9b26ee-a0f5-48b6-9a91-26625a3e7691", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '651')\n "}
2025-01-08 16:40:11,414 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '651' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:11,415 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'651\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'651\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:12,572 DEBUG MainThread azul.terra: Job info: {"job_id": "7ed2d45f-2de0-4e3d-a21f-6ada4acd47bb", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '651')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '651')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:12,753 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:40:12,754 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 1.0, num batches 1)
2025-01-08 16:40:12,754 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:12,754 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 2.0, num batches 1)
2025-01-08 16:40:12,754 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 5.0, num batches 1)
2025-01-08 16:40:12,754 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 7.0, num batches 1)
2025-01-08 16:40:12,755 INFO MainThread azul.azulclient: There are 6 bundle(s) with prefix '651' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:12,755 INFO MainThread azul.azulclient: Listing bundles with prefix '652' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:12,756 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:13,918 DEBUG MainThread azul.terra: Job info: {"job_id": "f927a6e5-483f-455e-86be-f29d9bf5d86b", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:14,069 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '652')\n "
2025-01-08 16:40:14,673 DEBUG MainThread azul.terra: Job info: {"job_id": "48a6aa00-fb38-4ac3-a74f-f4dc644a8d73", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '652')\n "}
2025-01-08 16:40:15,108 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '652' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:15,109 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'652\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'652\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:16,134 DEBUG MainThread azul.terra: Job info: {"job_id": "406d997f-3ffa-42e5-a26b-baca6114cb7c", "total_rows": 4, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '652')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '652')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:16,300 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 6.0, num batches 1)
2025-01-08 16:40:16,301 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 7.0, num batches 1)
2025-01-08 16:40:16,301 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 1.0, num batches 1)
2025-01-08 16:40:16,301 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:16,301 INFO MainThread azul.azulclient: There are 4 bundle(s) with prefix '652' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:16,301 INFO MainThread azul.azulclient: Listing bundles with prefix '653' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:16,302 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:17,064 DEBUG MainThread azul.terra: Job info: {"job_id": "3ecad737-f9d6-4752-a56a-b933d60cb970", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:17,233 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '653')\n "
2025-01-08 16:40:17,693 DEBUG MainThread azul.terra: Job info: {"job_id": "92392eca-3e48-4a04-976d-42e411eb02ad", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '653')\n "}
2025-01-08 16:40:18,128 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '653' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:18,129 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'653\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'653\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:18,927 DEBUG MainThread azul.terra: Job info: {"job_id": "6f141e00-5f03-4794-b65c-c5cc787c8176", "total_rows": 5, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '653')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '653')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:19,093 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 12.0, num batches 1)
2025-01-08 16:40:19,093 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:19,093 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 11.0, num batches 1)
2025-01-08 16:40:19,093 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 1.0, num batches 1)
2025-01-08 16:40:19,094 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 7.0, num batches 1)
2025-01-08 16:40:19,094 INFO MainThread azul.azulclient: There are 5 bundle(s) with prefix '653' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:19,095 INFO MainThread azul.azulclient: Listing bundles with prefix '654' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:19,095 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:19,767 DEBUG MainThread azul.terra: Job info: {"job_id": "6d234d44-fb5b-4467-8912-4baf391b3f3e", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:19,967 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '654')\n "
2025-01-08 16:40:20,622 DEBUG MainThread azul.terra: Job info: {"job_id": "6d82359f-874c-41f7-b2f5-c760b98e8d53", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '654')\n "}
2025-01-08 16:40:20,975 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '654' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:20,976 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'654\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'654\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:22,066 DEBUG MainThread azul.terra: Job info: {"job_id": "ed05d29f-7c11-4fd0-96a8-b751776f45d0", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '654')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '654')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:22,257 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 6.0, num batches 1)
2025-01-08 16:40:22,257 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:22,258 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:22,258 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 2.0, num batches 1)
2025-01-08 16:40:22,258 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 9.0, num batches 1)
2025-01-08 16:40:22,258 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 2.0, num batches 1)
2025-01-08 16:40:22,258 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 3.0, num batches 1)
2025-01-08 16:40:22,259 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '654' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:22,260 INFO MainThread azul.azulclient: Listing bundles with prefix '655' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:22,260 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:23,179 DEBUG MainThread azul.terra: Job info: {"job_id": "769d0f70-fac7-40d5-9b4d-aed4d11eb4f0", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:23,320 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '655')\n "
2025-01-08 16:40:24,003 DEBUG MainThread azul.terra: Job info: {"job_id": "f5e0f028-b442-4bc1-82c8-4eef8113e81a", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '655')\n "}
2025-01-08 16:40:24,317 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '655' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:24,317 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'655\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'655\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:25,726 DEBUG MainThread azul.terra: Job info: {"job_id": "c8e1b257-cbca-4ccf-bdcc-d3420bf86516", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '655')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '655')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 2.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 11.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 2.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 12.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 10.0, num batches 1)
2025-01-08 16:40:25,911 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '655' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:25,911 INFO MainThread azul.azulclient: Listing bundles with prefix '656' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:25,912 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:26,542 DEBUG MainThread azul.terra: Job info: {"job_id": "679c4a0c-a12c-4d2c-99db-86e3d8e3f346", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:26,692 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '656')\n "
2025-01-08 16:40:27,392 DEBUG MainThread azul.terra: Job info: {"job_id": "de424b3e-5d29-437b-be15-76c01d1b948c", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '656')\n "}
2025-01-08 16:40:27,723 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '656' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:27,724 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'656\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'656\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:28,798 DEBUG MainThread azul.terra: Job info: {"job_id": "c076821f-197b-4825-8415-7443646f2064", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '656')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '656')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 4.0, num batches 1)
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 8.0, num batches 1)
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 1.0, num batches 1)
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 6.0, num batches 1)
2025-01-08 16:40:28,965 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 3.0, num batches 1)
2025-01-08 16:40:28,966 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '656' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:28,966 INFO MainThread azul.azulclient: Listing bundles with prefix '657' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:28,966 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:29,662 DEBUG MainThread azul.terra: Job info: {"job_id": "1ef43952-e7ce-4f2b-83a0-63b6412d11c1", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:29,846 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '657')\n "
2025-01-08 16:40:30,622 DEBUG MainThread azul.terra: Job info: {"job_id": "c2bb7a19-50e1-4f0c-a4c2-f6f47e433697", "total_rows": 3, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '657')\n "}
2025-01-08 16:40:31,009 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '657' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:31,010 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'657\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'657\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:32,127 DEBUG MainThread azul.terra: Job info: {"job_id": "4e48a795-d378-4af9-8451-0e555a481289", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '657')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '657')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:32,363 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 2.0, num batches 1)
2025-01-08 16:40:32,363 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:32,364 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:32,364 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 1.0, num batches 1)
2025-01-08 16:40:32,364 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 11.0, num batches 1)
2025-01-08 16:40:32,364 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 1.0, num batches 1)
2025-01-08 16:40:32,364 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 10.0, num batches 1)
2025-01-08 16:40:32,365 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix '657' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:32,366 INFO MainThread azul.azulclient: Listing bundles with prefix '658' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:32,366 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:32,995 DEBUG MainThread azul.terra: Job info: {"job_id": "674fce4a-d420-4cdd-a546-5502a4301209", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:33,172 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '658')\n "
2025-01-08 16:40:33,839 DEBUG MainThread azul.terra: Job info: {"job_id": "12c61323-ce33-4914-8c91-9b3b60304d5e", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '658')\n "}
2025-01-08 16:40:34,153 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '658' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:34,154 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'658\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'658\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:35,099 DEBUG MainThread azul.terra: Job info: {"job_id": "26416088-274c-4324-aa93-6065b11a0ffb", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '658')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '658')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:35,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 4.0, num batches 1)
2025-01-08 16:40:35,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 7.0, num batches 1)
2025-01-08 16:40:35,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:35,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 3.0, num batches 1)
2025-01-08 16:40:35,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 1.0, num batches 1)
2025-01-08 16:40:35,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:40:35,242 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '658' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:35,243 INFO MainThread azul.azulclient: Listing bundles with prefix '659' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:35,243 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:36,041 DEBUG MainThread azul.terra: Job info: {"job_id": "cfaea5cb-bbc3-44d6-919b-603dd5d1f9d3", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:36,202 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '659')\n "
2025-01-08 16:40:37,010 DEBUG MainThread azul.terra: Job info: {"job_id": "44c85f9a-4720-4668-85ff-a99e96c85109", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '659')\n "}
2025-01-08 16:40:37,369 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '659' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:37,370 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'659\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'659\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:38,270 DEBUG MainThread azul.terra: Job info: {"job_id": "1b2b6be8-02a4-4aa8-b3c3-054809bae29f", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '659')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '659')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:38,429 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 6.0, num batches 1)
2025-01-08 16:40:38,429 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 2.0, num batches 1)
2025-01-08 16:40:38,430 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 2.0, num batches 1)
2025-01-08 16:40:38,430 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 3.0, num batches 1)
2025-01-08 16:40:38,430 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 8.0, num batches 1)
2025-01-08 16:40:38,430 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 2.0, num batches 1)
2025-01-08 16:40:38,430 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 4.0, num batches 1)
2025-01-08 16:40:38,431 INFO MainThread azul.azulclient: There are 9 bundle(s) with prefix '659' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:38,431 INFO MainThread azul.azulclient: Listing bundles with prefix '65a' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:38,431 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:39,180 DEBUG MainThread azul.terra: Job info: {"job_id": "c3200162-2b63-4dc9-8c53-2378db319afb", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:39,434 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65a')\n "
2025-01-08 16:40:40,137 DEBUG MainThread azul.terra: Job info: {"job_id": "c5f5f5a5-06f9-4f1e-bf31-0f8e49193497", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65a')\n "}
2025-01-08 16:40:40,460 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65a' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:40,460 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65a\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65a\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:41,626 DEBUG MainThread azul.terra: Job info: {"job_id": "f3d6bbb3-88df-4a8d-a285-dc45eb2e6975", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65a')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65a')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:41,758 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 2.0, num batches 1)
2025-01-08 16:40:41,758 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 9.0, num batches 1)
2025-01-08 16:40:41,758 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 2.0, num batches 1)
2025-01-08 16:40:41,758 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:41,758 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 2.0, num batches 1)
2025-01-08 16:40:41,758 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 2.0, num batches 1)
2025-01-08 16:40:41,759 INFO MainThread azul.azulclient: There are 6 bundle(s) with prefix '65a' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:41,759 INFO MainThread azul.azulclient: Listing bundles with prefix '65b' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:41,760 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:42,590 DEBUG MainThread azul.terra: Job info: {"job_id": "c7d28ae4-2a86-4a85-8fbc-e68d591b9cc6", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:42,809 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65b')\n "
2025-01-08 16:40:43,310 DEBUG MainThread azul.terra: Job info: {"job_id": "1bbddf5b-a8ca-4ab0-993a-1da29010de9a", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65b')\n "}
2025-01-08 16:40:43,636 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65b' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:43,636 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65b\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65b\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:44,670 DEBUG MainThread azul.terra: Job info: {"job_id": "4aad8f7c-fb2d-4ab4-8d92-fcaf363e2fda", "total_rows": 5, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65b')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65b')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:44,819 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 7.0, num batches 1)
2025-01-08 16:40:44,819 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:44,819 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:40:44,819 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 1.0, num batches 1)
2025-01-08 16:40:44,819 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 7.0, num batches 1)
2025-01-08 16:40:44,820 INFO MainThread azul.azulclient: There are 7 bundle(s) with prefix '65b' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:44,820 INFO MainThread azul.azulclient: Listing bundles with prefix '65c' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:44,821 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:45,572 DEBUG MainThread azul.terra: Job info: {"job_id": "656b2d45-f177-49d7-bce4-a0ca3f812acd", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:45,740 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65c')\n "
2025-01-08 16:40:46,353 DEBUG MainThread azul.terra: Job info: {"job_id": "05cb930a-8dcd-43ea-8c73-fa32aaf114b1", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65c')\n "}
2025-01-08 16:40:46,698 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65c' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:46,699 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65c\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65c\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:47,776 DEBUG MainThread azul.terra: Job info: {"job_id": "fadb9ebc-67fa-489e-8b2b-f23a5c3ee27e", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65c')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65c')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:47,924 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 6.0, num batches 1)
2025-01-08 16:40:47,925 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:47,925 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:47,925 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'sample' (average batch size 2.0, num batches 1)
2025-01-08 16:40:47,925 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 6.0, num batches 1)
2025-01-08 16:40:47,925 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:40:47,926 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '65c' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:47,926 INFO MainThread azul.azulclient: Listing bundles with prefix '65d' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:47,927 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:48,386 DEBUG MainThread azul.terra: Job info: {"job_id": "ea0ef052-2d9d-4c37-bf36-ea9247fe6aa5", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:48,644 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65d')\n "
2025-01-08 16:40:49,206 DEBUG MainThread azul.terra: Job info: {"job_id": "61c0f444-f376-4616-84ad-57382a543f60", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65d')\n "}
2025-01-08 16:40:49,541 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65d' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:49,541 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65d\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65d\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:50,629 DEBUG MainThread azul.terra: Job info: {"job_id": "0c33b3ef-dbab-4abe-a6c7-467406b6679e", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65d')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65d')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:50,794 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 10.0, num batches 1)
2025-01-08 16:40:50,795 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 10.0, num batches 1)
2025-01-08 16:40:50,795 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:50,795 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:50,795 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 7.0, num batches 1)
2025-01-08 16:40:50,795 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 2.0, num batches 1)
2025-01-08 16:40:50,796 INFO MainThread azul.azulclient: There are 7 bundle(s) with prefix '65d' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:50,796 INFO MainThread azul.azulclient: Listing bundles with prefix '65e' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:50,797 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:51,328 DEBUG MainThread azul.terra: Job info: {"job_id": "523b1ef8-1575-4cfe-b6bb-a9555c2e4533", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:51,473 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65e')\n "
2025-01-08 16:40:52,395 DEBUG MainThread azul.terra: Job info: {"job_id": "84c38f47-3371-4186-b968-93431cd75254", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65e')\n "}
2025-01-08 16:40:52,717 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65e' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:52,718 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65e\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65e\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:53,874 DEBUG MainThread azul.terra: Job info: {"job_id": "04d9f506-f64c-445f-94d4-f9ca628b9353", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65e')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65e')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:54,019 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 1.0, num batches 1)
2025-01-08 16:40:54,019 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 2.0, num batches 1)
2025-01-08 16:40:54,019 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:40:54,019 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:40:54,019 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:40:54,019 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 6.0, num batches 1)
2025-01-08 16:40:54,020 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '65e' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:54,020 INFO MainThread azul.azulclient: Listing bundles with prefix '65f' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:54,021 DEBUG MainThread azul.terra: Query (159 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n '
2025-01-08 16:40:54,772 DEBUG MainThread azul.terra: Job info: {"job_id": "af4ed447-647f-453e-bd60-519432b4192d", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset`\n "}
2025-01-08 16:40:54,926 DEBUG MainThread azul.terra: Query (203 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65f')\n "
2025-01-08 16:40:55,440 DEBUG MainThread azul.terra: Job info: {"job_id": "270d2690-e87c-4845-9a0a-021e732668a5", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '65f')\n "}
2025-01-08 16:40:55,915 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '65f' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:65/1
2025-01-08 16:40:55,916 DEBUG MainThread azul.terra: Query (17073 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'65f\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'65f\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n β¦'
2025-01-08 16:40:56,940 DEBUG MainThread azul.terra: Job info: {"job_id": "0512f0a7-8a30-4578-b034-2c3eea541c1a", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '65f')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 3 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 3 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 3 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 3 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 3 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '65f')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n \u2026"}
2025-01-08 16:40:57,086 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_activity' (average batch size 7.0, num batches 1)
2025-01-08 16:40:57,086 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'participant' (average batch size 2.0, num batches 1)
2025-01-08 16:40:57,086 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:40:57,086 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:40:57,086 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'qc_result_sample' (average batch size 2.0, num batches 1)
2025-01-08 16:40:57,086 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 3 for table 'pedigree' (average batch size 1.0, num batches 1)
2025-01-08 16:40:57,087 INFO MainThread azul.azulclient: There are 7 bundle(s) with prefix '65f' in source TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')).
2025-01-08 16:40:57,087 DEBUG MainThread azul.terra: Query (200 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:40:57,787 DEBUG MainThread azul.terra: Job info: {"job_id": "3576a5ef-5522-42e1-9736-52cd638f7a88", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:40:57,936 DEBUG MainThread azul.terra: Query (198 characters total): "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "
2025-01-08 16:40:58,603 DEBUG MainThread azul.terra: Job info: {"job_id": "09345d94-db5c-44a9-8684-469372edcd8f", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT COUNT(*) AS count\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n WHERE STARTS_WITH(datarepo_row_id, '')\n "}
2025-01-08 16:40:58,972 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:
2025-01-08 16:40:58,972 DEBUG MainThread azul.terra: Query (16969 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER β¦'
2025-01-08 16:41:00,056 DEBUG MainThread azul.terra: Job info: {"job_id": "9cc5e5b9-fe1c-497b-b719-2d0b334352c6", "total_rows": 10, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 0 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 0 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 0 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 0 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 0 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER \u2026"}
2025-01-08 16:41:00,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:41:00,241 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'file_inventory' (average batch size 50.0, num batches 1)
2025-01-08 16:41:00,242 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_sequencingactivity' (average batch size 50.0, num batches 1)
2025-01-08 16:41:00,242 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'subject' (average batch size 25.0, num batches 1)
2025-01-08 16:41:00,242 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_donor' (average batch size 25.0, num batches 1)
2025-01-08 16:41:00,242 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'workspace_attributes' (average batch size 37.0, num batches 1)
2025-01-08 16:41:00,242 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sequencing' (average batch size 50.0, num batches 1)
2025-01-08 16:41:00,243 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'sample' (average batch size 25.0, num batches 1)
2025-01-08 16:41:00,243 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'anvil_file' (average batch size 50.0, num batches 1)
2025-01-08 16:41:00,243 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 0 for table 'family' (average batch size 13.0, num batches 1)
2025-01-08 16:41:00,243 INFO MainThread azul.azulclient: Listing bundles with prefix '0' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:00,244 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:00,937 DEBUG MainThread azul.terra: Job info: {"job_id": "4f224279-1f94-4f7c-b460-3e9beace7112", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:01,074 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '0')\n "
2025-01-08 16:41:01,875 DEBUG MainThread azul.terra: Job info: {"job_id": "c4b9b094-61af-462a-a6a4-73153b1a8cbc", "total_rows": 5, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '0')\n "}
2025-01-08 16:41:02,220 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '0' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:02,221 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'0\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'0\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:03,324 DEBUG MainThread azul.terra: Job info: {"job_id": "531516f8-d39d-4421-919f-2bfde066f9bf", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '0')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '0')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 3.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 2.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 2.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 4.0, num batches 1)
2025-01-08 16:41:03,518 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 3.0, num batches 1)
2025-01-08 16:41:03,519 INFO MainThread azul.azulclient: There are 13 bundle(s) with prefix '0' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:03,519 INFO MainThread azul.azulclient: Listing bundles with prefix '1' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:03,519 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:04,198 DEBUG MainThread azul.terra: Job info: {"job_id": "28ef1bf7-3d66-4eee-9ab5-22a33f508299", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:05,279 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '1')\n "
2025-01-08 16:41:06,803 DEBUG MainThread azul.terra: Job info: {"job_id": "9928c0a8-028a-4caa-8e5a-09efe26230d0", "total_rows": 3, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '1')\n "}
2025-01-08 16:41:08,368 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '1' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:08,369 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'1\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'1\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:09,314 DEBUG MainThread azul.terra: Job info: {"job_id": "5c2da79d-dd8b-4bdb-9785-d3df893ccb40", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '1')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '1')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:10,283 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 5.0, num batches 1)
2025-01-08 16:41:10,283 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 6.0, num batches 1)
2025-01-08 16:41:10,283 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 2.0, num batches 1)
2025-01-08 16:41:10,283 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 2.0, num batches 1)
2025-01-08 16:41:10,283 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 6.0, num batches 1)
2025-01-08 16:41:10,283 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 4.0, num batches 1)
2025-01-08 16:41:10,284 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:41:10,285 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix '1' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:10,285 INFO MainThread azul.azulclient: Listing bundles with prefix '2' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:10,286 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:11,381 DEBUG MainThread azul.terra: Job info: {"job_id": "58d79ac3-8d85-46e7-b58d-331e4e70b7b7", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:11,538 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '2')\n "
2025-01-08 16:41:12,268 DEBUG MainThread azul.terra: Job info: {"job_id": "ca561327-ff2c-4caf-b282-2d1ee04ce95c", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '2')\n "}
2025-01-08 16:41:12,578 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '2' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:12,578 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'2\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'2\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:13,761 DEBUG MainThread azul.terra: Job info: {"job_id": "a45395fe-201e-4e39-ac05-2c0f69065925", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '2')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '2')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 5.0, num batches 1)
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 1.0, num batches 1)
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 2.0, num batches 1)
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 2.0, num batches 1)
2025-01-08 16:41:13,916 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 2.0, num batches 1)
2025-01-08 16:41:13,917 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '2' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:13,918 INFO MainThread azul.azulclient: Listing bundles with prefix '3' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:13,918 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:14,512 DEBUG MainThread azul.terra: Job info: {"job_id": "2812b5c1-e09b-4beb-8bc5-8c0eae59d254", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:14,685 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '3')\n "
2025-01-08 16:41:15,433 DEBUG MainThread azul.terra: Job info: {"job_id": "5eb25c50-f071-4db1-9955-5b1d8dd1c5df", "total_rows": 3, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '3')\n "}
2025-01-08 16:41:15,803 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '3' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:15,803 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'3\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'3\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:16,799 DEBUG MainThread azul.terra: Job info: {"job_id": "d78032fd-ad18-40cb-806b-93dc9535fd48", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '3')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '3')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:16,962 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 2.0, num batches 1)
2025-01-08 16:41:16,962 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 2.0, num batches 1)
2025-01-08 16:41:16,962 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:16,962 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 4.0, num batches 1)
2025-01-08 16:41:16,962 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 5.0, num batches 1)
2025-01-08 16:41:16,962 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 2.0, num batches 1)
2025-01-08 16:41:16,963 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 2.0, num batches 1)
2025-01-08 16:41:16,963 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 4.0, num batches 1)
2025-01-08 16:41:16,963 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:16,964 INFO MainThread azul.azulclient: There are 12 bundle(s) with prefix '3' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:16,964 INFO MainThread azul.azulclient: Listing bundles with prefix '4' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:16,965 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:17,806 DEBUG MainThread azul.terra: Job info: {"job_id": "f29d69ef-2034-46d8-9626-91318a46615c", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:17,941 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '4')\n "
2025-01-08 16:41:18,511 DEBUG MainThread azul.terra: Job info: {"job_id": "458f4d18-8111-4d34-b900-156d848ab031", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '4')\n "}
2025-01-08 16:41:18,838 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '4' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:18,838 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'4\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'4\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:19,705 DEBUG MainThread azul.terra: Job info: {"job_id": "dc9c821d-4403-4e76-ae08-8a76a84f08ab", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '4')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '4')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 4.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 2.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 4.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 1.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 2.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_project' (average batch size 1.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 4.0, num batches 1)
2025-01-08 16:41:19,869 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:41:19,870 INFO MainThread azul.azulclient: There are 9 bundle(s) with prefix '4' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:19,870 INFO MainThread azul.azulclient: Listing bundles with prefix '5' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:19,871 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:20,646 DEBUG MainThread azul.terra: Job info: {"job_id": "c877c070-7b34-4b84-b173-92eed8f27cbe", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:20,794 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '5')\n "
2025-01-08 16:41:21,529 DEBUG MainThread azul.terra: Job info: {"job_id": "ba719fb9-f442-42a4-9246-9797907888a9", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '5')\n "}
2025-01-08 16:41:21,863 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '5' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:21,864 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'5\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'5\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:22,708 DEBUG MainThread azul.terra: Job info: {"job_id": "d1fa9875-4cd6-452f-b799-e762d0478c09", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '5')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '5')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 1.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 3.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 1.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 3.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 7.0, num batches 1)
2025-01-08 16:41:22,867 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:41:22,868 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix '5' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:22,868 INFO MainThread azul.azulclient: Listing bundles with prefix '6' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:22,869 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:23,554 DEBUG MainThread azul.terra: Job info: {"job_id": "731982ef-3d4e-41b1-8bcd-670d3a3efdb0", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:23,725 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '6')\n "
2025-01-08 16:41:24,589 DEBUG MainThread azul.terra: Job info: {"job_id": "26b43ec2-ba5c-4936-8621-ba2e3719ed62", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '6')\n "}
2025-01-08 16:41:24,933 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '6' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:24,934 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'6\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'6\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:26,233 DEBUG MainThread azul.terra: Job info: {"job_id": "10f79781-534f-4d41-b7e9-a5af1fdd67c0", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '6')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '6')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:26,389 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 3.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 2.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 3.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 1.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 2.0, num batches 1)
2025-01-08 16:41:26,390 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 3.0, num batches 1)
2025-01-08 16:41:26,391 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:26,391 INFO MainThread azul.azulclient: There are 11 bundle(s) with prefix '6' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:26,391 INFO MainThread azul.azulclient: Listing bundles with prefix '7' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:26,392 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:27,048 DEBUG MainThread azul.terra: Job info: {"job_id": "3bda0146-db92-4228-bce7-0396c11415f4", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:27,211 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '7')\n "
2025-01-08 16:41:27,848 DEBUG MainThread azul.terra: Job info: {"job_id": "d28c8327-a656-4ad5-a849-a4693d29f142", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '7')\n "}
2025-01-08 16:41:28,164 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '7' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:28,165 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'7\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'7\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:29,222 DEBUG MainThread azul.terra: Job info: {"job_id": "399ef6c1-29fa-4fa4-863f-893f7c7b09ab", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '7')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '7')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 2.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 3.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 2.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 2.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 3.0, num batches 1)
2025-01-08 16:41:29,401 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 6.0, num batches 1)
2025-01-08 16:41:29,402 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix '7' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:29,402 INFO MainThread azul.azulclient: Listing bundles with prefix '8' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:29,403 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:30,039 DEBUG MainThread azul.terra: Job info: {"job_id": "0e759ce1-2e31-4b9c-9465-f1223c0d982c", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:30,208 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '8')\n "
2025-01-08 16:41:30,977 DEBUG MainThread azul.terra: Job info: {"job_id": "f405f815-788c-4694-af4a-46c17854e785", "total_rows": 0, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '8')\n "}
2025-01-08 16:41:31,305 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '8' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:31,305 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'8\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'8\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:32,285 DEBUG MainThread azul.terra: Job info: {"job_id": "18ec20fa-b892-4b50-986c-2cf271576f9a", "total_rows": 6, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '8')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '8')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:32,434 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:41:32,434 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 2.0, num batches 1)
2025-01-08 16:41:32,434 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 2.0, num batches 1)
2025-01-08 16:41:32,434 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 2.0, num batches 1)
2025-01-08 16:41:32,434 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 3.0, num batches 1)
2025-01-08 16:41:32,434 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 5.0, num batches 1)
2025-01-08 16:41:32,435 INFO MainThread azul.azulclient: There are 6 bundle(s) with prefix '8' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:32,435 INFO MainThread azul.azulclient: Listing bundles with prefix '9' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:32,436 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:33,095 DEBUG MainThread azul.terra: Job info: {"job_id": "2ec7abfe-cec1-4e3a-8a74-e5d37788b840", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:33,264 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '9')\n "
2025-01-08 16:41:33,740 DEBUG MainThread azul.terra: Job info: {"job_id": "c6386add-cfd8-43bd-a0fd-74489370f69a", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, '9')\n "}
2025-01-08 16:41:34,084 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition '9' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:34,085 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'9\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'9\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:35,277 DEBUG MainThread azul.terra: Job info: {"job_id": "7f659629-4b7f-430f-9bbe-27bbba382bf7", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, '9')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, '9')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:35,546 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 2.0, num batches 1)
2025-01-08 16:41:35,546 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 5.0, num batches 1)
2025-01-08 16:41:35,546 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:41:35,546 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:35,547 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:35,547 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:41:35,547 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 1.0, num batches 1)
2025-01-08 16:41:35,547 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 2.0, num batches 1)
2025-01-08 16:41:35,548 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix '9' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:35,548 INFO MainThread azul.azulclient: Listing bundles with prefix 'a' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:35,549 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:36,273 DEBUG MainThread azul.terra: Job info: {"job_id": "b0dd71ca-bb32-492e-81fc-30023aaa0403", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:36,433 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'a')\n "
2025-01-08 16:41:36,983 DEBUG MainThread azul.terra: Job info: {"job_id": "f351f563-17cc-47c0-955d-a4922518ec0e", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'a')\n "}
2025-01-08 16:41:37,338 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition 'a' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:37,339 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'a\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'a\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:38,253 DEBUG MainThread azul.terra: Job info: {"job_id": "830c882a-3185-4b52-b473-bccb15c61021", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, 'a')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, 'a')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 3.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 5.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 1.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 3.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 4.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 2.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 4.0, num batches 1)
2025-01-08 16:41:38,395 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:41:38,396 INFO MainThread azul.azulclient: There are 9 bundle(s) with prefix 'a' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:38,396 INFO MainThread azul.azulclient: Listing bundles with prefix 'b' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:38,396 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:39,131 DEBUG MainThread azul.terra: Job info: {"job_id": "a48c596d-cd94-4be0-8c29-efd8ae4bc7db", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:39,303 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'b')\n "
2025-01-08 16:41:39,931 DEBUG MainThread azul.terra: Job info: {"job_id": "d255e18e-2cc9-494a-a614-c2ac4968a7cc", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'b')\n "}
2025-01-08 16:41:40,253 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition 'b' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:40,254 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'b\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'b\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:41,368 DEBUG MainThread azul.terra: Job info: {"job_id": "de8fad4e-2a87-4c80-b6a9-b0202403fd67", "total_rows": 7, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, 'b')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, 'b')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:41,542 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:41,542 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 1.0, num batches 1)
2025-01-08 16:41:41,542 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 1.0, num batches 1)
2025-01-08 16:41:41,543 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 1.0, num batches 1)
2025-01-08 16:41:41,543 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 3.0, num batches 1)
2025-01-08 16:41:41,543 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 3.0, num batches 1)
2025-01-08 16:41:41,543 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:41,544 INFO MainThread azul.azulclient: There are 8 bundle(s) with prefix 'b' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:41,544 INFO MainThread azul.azulclient: Listing bundles with prefix 'c' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:41,545 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:42,268 DEBUG MainThread azul.terra: Job info: {"job_id": "5607cd5d-72be-4298-ac09-b2ed2e141c8b", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:42,487 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'c')\n "
2025-01-08 16:41:43,124 DEBUG MainThread azul.terra: Job info: {"job_id": "04093530-5c90-4cb6-935c-d6db8918be4a", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'c')\n "}
2025-01-08 16:41:43,535 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition 'c' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:43,535 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'c\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'c\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:44,435 DEBUG MainThread azul.terra: Job info: {"job_id": "195451f4-e41e-4efa-95c1-e8280bca122e", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, 'c')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, 'c')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:44,614 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 3.0, num batches 1)
2025-01-08 16:41:44,614 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 4.0, num batches 1)
2025-01-08 16:41:44,615 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 2.0, num batches 1)
2025-01-08 16:41:44,615 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 1.0, num batches 1)
2025-01-08 16:41:44,615 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 5.0, num batches 1)
2025-01-08 16:41:44,615 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 5.0, num batches 1)
2025-01-08 16:41:44,615 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:41:44,615 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 2.0, num batches 1)
2025-01-08 16:41:44,616 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix 'c' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:44,616 INFO MainThread azul.azulclient: Listing bundles with prefix 'd' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:44,617 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:45,206 DEBUG MainThread azul.terra: Job info: {"job_id": "f0ebc773-4f6a-45da-b90f-6b5d76d6e0a5", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:45,391 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'd')\n "
2025-01-08 16:41:46,051 DEBUG MainThread azul.terra: Job info: {"job_id": "eca2fc75-833b-4367-beec-3f3b550a8408", "total_rows": 2, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'd')\n "}
2025-01-08 16:41:46,399 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition 'd' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:46,400 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'d\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'d\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:47,184 DEBUG MainThread azul.terra: Job info: {"job_id": "9023d380-fe14-48b6-8023-d94ec11335a5", "total_rows": 8, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, 'd')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, 'd')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:47,321 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 6.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 2.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 2.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 3.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 6.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 7.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:47,322 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 5.0, num batches 1)
2025-01-08 16:41:47,323 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix 'd' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:47,324 INFO MainThread azul.azulclient: Listing bundles with prefix 'e' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:47,325 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:47,763 DEBUG MainThread azul.terra: Job info: {"job_id": "6eb681e8-d38c-47b9-a353-299803482a6e", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:47,921 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'e')\n "
2025-01-08 16:41:48,388 DEBUG MainThread azul.terra: Job info: {"job_id": "14858e75-a43b-41a2-b527-bdd531386878", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'e')\n "}
2025-01-08 16:41:48,731 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition 'e' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:48,731 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'e\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'e\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:49,709 DEBUG MainThread azul.terra: Job info: {"job_id": "5acb866b-a50c-477c-b78d-73c7fa675e4f", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, 'e')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, 'e')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 2.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 1.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 3.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 4.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 3.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 1.0, num batches 1)
2025-01-08 16:41:49,863 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 3.0, num batches 1)
2025-01-08 16:41:49,864 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix 'e' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:49,864 INFO MainThread azul.azulclient: Listing bundles with prefix 'f' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:49,864 DEBUG MainThread azul.terra: Query (157 characters total): '\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n '
2025-01-08 16:41:50,500 DEBUG MainThread azul.terra: Job info: {"job_id": "7f024640-b76d-4508-bd98-27b491c17f95", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_dataset`\n "}
2025-01-08 16:41:50,666 DEBUG MainThread azul.terra: Query (199 characters total): "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'f')\n "
2025-01-08 16:41:51,325 DEBUG MainThread azul.terra: Job info: {"job_id": "2985aace-dd0f-469f-8dd0-65e6dce1c1d1", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT datarepo_row_id\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_biosample`\n WHERE STARTS_WITH(datarepo_row_id, 'f')\n "}
2025-01-08 16:41:51,639 INFO MainThread azul.plugins.repository.tdr_anvil: Calculating batch prefix lengths for partition 'f' of 16 tables in source tdr:bigquery:gcp:datarepo-dev-97ad270b:ANVIL_CMG_Sample_1_20230225_ANV5_202302281509:/1
2025-01-08 16:41:51,639 DEBUG MainThread azul.terra: Query (16985 characters total): '(\n SELECT * FROM (\n SELECT\n \'anvil_activity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, \'f\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n \'anvil_alignmentactivity\' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, ""), IFNULL(p2, ""), IFNULL(p3, ""), IFNULL(p4, "")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, \'f\')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDEβ¦'
2025-01-08 16:41:52,662 DEBUG MainThread azul.terra: Job info: {"job_id": "9cebd1be-8025-42e7-91ab-e0bcb1917275", "total_rows": 9, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "(\n SELECT * FROM (\n SELECT\n 'anvil_activity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_activity`\n WHERE STARTS_WITH(datarepo_row_id, 'f')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDER BY ABS(256 - average_batch_size)\n LIMIT 1\n )\n ) UNION ALL (\n SELECT * FROM (\n SELECT\n 'anvil_alignmentactivity' AS table_name,\n 1 + LENGTH(CONCAT(\n IFNULL(p1, \"\"), IFNULL(p2, \"\"), IFNULL(p3, \"\"), IFNULL(p4, \"\")\n )) AS batch_prefix_length,\n AVG(num_rows) AS average_batch_size,\n COUNT(*) AS num_batches\n FROM (\n SELECT\n SUBSTR(datarepo_row_id, 1 + 1, 1) AS p1, SUBSTR(datarepo_row_id, 1 + 2, 1) AS p2, SUBSTR(datarepo_row_id, 1 + 3, 1) AS p3, SUBSTR(datarepo_row_id, 1 + 4, 1) AS p4,\n COUNT(*) AS num_rows\n FROM `datarepo-dev-97ad270b.ANVIL_CMG_Sample_1_20230225_ANV5_202302281509.anvil_alignmentactivity`\n WHERE STARTS_WITH(datarepo_row_id, 'f')\n GROUP BY ROLLUP (p1, p2, p3, p4)\n )\n GROUP BY batch_prefix_length\n ORDE\u2026"}
2025-01-08 16:41:52,819 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'file_inventory' (average batch size 4.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'family' (average batch size 2.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sequencing' (average batch size 4.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_sequencingactivity' (average batch size 5.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_donor' (average batch size 1.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'anvil_file' (average batch size 2.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'sample' (average batch size 2.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'workspace_attributes' (average batch size 2.0, num batches 1)
2025-01-08 16:41:52,820 INFO MainThread azul.plugins.repository.tdr_anvil: Selected batch prefix length 1 for table 'subject' (average batch size 1.0, num batches 1)
2025-01-08 16:41:52,821 INFO MainThread azul.azulclient: There are 10 bundle(s) with prefix 'f' in source TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')).
2025-01-08 16:41:55,242 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:41:55,333 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:41:55,420 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:41:55,506 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:41:55,506 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:41:55,506 INFO MainThread azul.queues: Message count history (most recent first) is [0].
2025-01-08 16:41:55,506 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:42:05,597 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:05,686 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:05,776 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:05,863 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:05,863 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:42:05,863 INFO MainThread azul.queues: Message count history (most recent first) is [0, 0].
2025-01-08 16:42:05,863 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:42:15,955 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 184 message(s) (176 available, 8 in flight and 0 delayed).
2025-01-08 16:42:16,043 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:16,132 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:16,220 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:16,220 INFO MainThread azul.queues: Counting 184 messages in 4 queues.
2025-01-08 16:42:16,220 INFO MainThread azul.queues: Message count history (most recent first) is [184, 0, 0].
2025-01-08 16:42:16,220 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:42:26,315 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 371 message(s) (364 available, 7 in flight and 0 delayed).
2025-01-08 16:42:26,402 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:26,494 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 23 message(s) (0 available, 0 in flight and 23 delayed).
2025-01-08 16:42:26,579 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:26,579 INFO MainThread azul.queues: Counting 394 messages in 4 queues.
2025-01-08 16:42:26,579 INFO MainThread azul.queues: Message count history (most recent first) is [394, 184, 0, 0].
2025-01-08 16:42:26,579 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:42:36,672 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 337 message(s) (329 available, 8 in flight and 0 delayed).
2025-01-08 16:42:36,759 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:36,850 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 64 message(s) (23 available, 13 in flight and 28 delayed).
2025-01-08 16:42:36,937 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:36,937 INFO MainThread azul.queues: Counting 401 messages in 4 queues.
2025-01-08 16:42:36,937 INFO MainThread azul.queues: Message count history (most recent first) is [401, 394, 184, 0, 0].
2025-01-08 16:42:36,937 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:42:47,028 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 165 message(s) (163 available, 2 in flight and 0 delayed).
2025-01-08 16:42:47,116 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:47,208 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 80 message(s) (0 available, 1 in flight and 79 delayed).
2025-01-08 16:42:47,295 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:47,295 INFO MainThread azul.queues: Counting 245 messages in 4 queues.
2025-01-08 16:42:47,295 INFO MainThread azul.queues: Message count history (most recent first) is [245, 401, 394, 184, 0, 0].
2025-01-08 16:42:47,295 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:42:57,384 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 320 message(s) (312 available, 8 in flight and 0 delayed).
2025-01-08 16:42:57,472 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:57,563 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 60 message(s) (0 available, 0 in flight and 60 delayed).
2025-01-08 16:42:57,649 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:42:57,649 INFO MainThread azul.queues: Counting 380 messages in 4 queues.
2025-01-08 16:42:57,649 INFO MainThread azul.queues: Message count history (most recent first) is [380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:42:57,649 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:43:07,741 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 292 message(s) (287 available, 5 in flight and 0 delayed).
2025-01-08 16:43:07,829 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:07,920 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 53 message(s) (0 available, 0 in flight and 53 delayed).
2025-01-08 16:43:08,008 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:08,008 INFO MainThread azul.queues: Counting 345 messages in 4 queues.
2025-01-08 16:43:08,008 INFO MainThread azul.queues: Message count history (most recent first) is [345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:43:08,008 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:43:18,098 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 371 message(s) (364 available, 7 in flight and 0 delayed).
2025-01-08 16:43:18,189 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:18,280 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 66 message(s) (0 available, 2 in flight and 64 delayed).
2025-01-08 16:43:18,367 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:18,367 INFO MainThread azul.queues: Counting 437 messages in 4 queues.
2025-01-08 16:43:18,367 INFO MainThread azul.queues: Message count history (most recent first) is [437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:43:18,368 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:43:28,462 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 235 message(s) (228 available, 7 in flight and 0 delayed).
2025-01-08 16:43:28,549 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:28,639 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 75 message(s) (0 available, 0 in flight and 75 delayed).
2025-01-08 16:43:28,727 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:28,727 INFO MainThread azul.queues: Counting 310 messages in 4 queues.
2025-01-08 16:43:28,727 INFO MainThread azul.queues: Message count history (most recent first) is [310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:43:28,727 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:43:38,823 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 206 message(s) (199 available, 7 in flight and 0 delayed).
2025-01-08 16:43:38,912 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:39,009 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 61 message(s) (0 available, 0 in flight and 61 delayed).
2025-01-08 16:43:39,096 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:39,096 INFO MainThread azul.queues: Counting 267 messages in 4 queues.
2025-01-08 16:43:39,096 INFO MainThread azul.queues: Message count history (most recent first) is [267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:43:39,096 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:43:49,191 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 152 message(s) (146 available, 6 in flight and 0 delayed).
2025-01-08 16:43:49,282 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:49,376 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 51 message(s) (0 available, 0 in flight and 51 delayed).
2025-01-08 16:43:49,463 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:49,463 INFO MainThread azul.queues: Counting 203 messages in 4 queues.
2025-01-08 16:43:49,463 INFO MainThread azul.queues: Message count history (most recent first) is [203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:43:49,463 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:43:59,557 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 179 message(s) (171 available, 8 in flight and 0 delayed).
2025-01-08 16:43:59,646 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:59,734 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 85 message(s) (0 available, 0 in flight and 85 delayed).
2025-01-08 16:43:59,825 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:43:59,825 INFO MainThread azul.queues: Counting 264 messages in 4 queues.
2025-01-08 16:43:59,825 INFO MainThread azul.queues: Message count history (most recent first) is [264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:43:59,825 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:44:09,969 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 152 message(s) (146 available, 6 in flight and 0 delayed).
2025-01-08 16:44:10,055 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:10,144 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 72 message(s) (0 available, 2 in flight and 70 delayed).
2025-01-08 16:44:10,230 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:10,230 INFO MainThread azul.queues: Counting 224 messages in 4 queues.
2025-01-08 16:44:10,230 INFO MainThread azul.queues: Message count history (most recent first) is [224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:44:10,230 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:44:20,323 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 153 message(s) (146 available, 7 in flight and 0 delayed).
2025-01-08 16:44:20,413 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:20,805 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 61 message(s) (0 available, 8 in flight and 53 delayed).
2025-01-08 16:44:20,902 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:20,902 INFO MainThread azul.queues: Counting 214 messages in 4 queues.
2025-01-08 16:44:20,902 INFO MainThread azul.queues: Message count history (most recent first) is [214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:44:20,902 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:44:31,003 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 36 message(s) (29 available, 7 in flight and 0 delayed).
2025-01-08 16:44:31,096 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:31,183 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 80 message(s) (0 available, 20 in flight and 60 delayed).
2025-01-08 16:44:31,268 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:31,269 INFO MainThread azul.queues: Counting 116 messages in 4 queues.
2025-01-08 16:44:31,269 INFO MainThread azul.queues: Message count history (most recent first) is [116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:44:31,269 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:44:41,362 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 4 message(s) (0 available, 4 in flight and 0 delayed).
2025-01-08 16:44:41,450 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:41,541 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 37 message(s) (0 available, 0 in flight and 37 delayed).
2025-01-08 16:44:41,630 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:41,630 INFO MainThread azul.queues: Counting 41 messages in 4 queues.
2025-01-08 16:44:41,630 INFO MainThread azul.queues: Message count history (most recent first) is [41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:44:41,631 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:44:51,726 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 56 message(s) (51 available, 5 in flight and 0 delayed).
2025-01-08 16:44:51,815 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:51,919 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 24 message(s) (0 available, 0 in flight and 24 delayed).
2025-01-08 16:44:52,010 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:44:52,011 INFO MainThread azul.queues: Counting 80 messages in 4 queues.
2025-01-08 16:44:52,011 INFO MainThread azul.queues: Message count history (most recent first) is [80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:44:52,011 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:45:02,109 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 4 message(s) (0 available, 4 in flight and 0 delayed).
2025-01-08 16:45:02,200 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:02,291 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:02,381 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:02,381 INFO MainThread azul.queues: Counting 4 messages in 4 queues.
2025-01-08 16:45:02,381 INFO MainThread azul.queues: Message count history (most recent first) is [4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:45:02,381 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:45:12,470 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:12,604 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:12,690 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:12,775 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:12,775 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:45:12,775 INFO MainThread azul.queues: Message count history (most recent first) is [0, 4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:45:12,775 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:45:22,871 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:22,961 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:23,050 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:23,138 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:23,138 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:45:23,138 INFO MainThread azul.queues: Message count history (most recent first) is [0, 0, 4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:45:23,138 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:45:33,230 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:33,320 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:33,407 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:33,496 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:33,496 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:45:33,496 INFO MainThread azul.queues: Message count history (most recent first) is [0, 0, 0, 4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:45:33,496 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:45:43,589 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:43,681 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:43,769 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:43,857 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:43,857 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:45:43,857 INFO MainThread azul.queues: Message count history (most recent first) is [0, 0, 0, 0, 4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:45:43,858 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:45:53,951 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:54,039 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:54,126 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:54,212 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:45:54,213 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:45:54,213 INFO MainThread azul.queues: Message count history (most recent first) is [0, 0, 0, 0, 0, 4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:45:54,213 INFO MainThread azul.queues: Waiting for 4 queue(s) to stabilize ...
2025-01-08 16:46:04,306 DEBUG MainThread azul.queues: Queue azul-notifications-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:46:04,393 DEBUG MainThread azul.queues: Queue azul-notifications_retry-daniel2 has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:46:04,478 DEBUG MainThread azul.queues: Queue azul-tallies-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:46:04,566 DEBUG MainThread azul.queues: Queue azul-tallies_retry-daniel2.fifo has 0 message(s) (0 available, 0 in flight and 0 delayed).
2025-01-08 16:46:04,567 INFO MainThread azul.queues: Counting 0 messages in 4 queues.
2025-01-08 16:46:04,567 INFO MainThread azul.queues: Message count history (most recent first) is [0, 0, 0, 0, 0, 0, 4, 80, 41, 116, 214, 224, 264, 203, 267, 310, 437, 345, 380, 245, 401, 394, 184, 0, 0].
2025-01-08 16:46:04,567 INFO MainThread azul.queues: The queues have emptied.
2025-01-08 16:46:04,567 INFO MainThread test.integration_test: Beginning sub-test [catalog_complete] {'catalog': 'anvil-it'}
2025-01-08 16:46:04,568 INFO MainThread test.integration_test: Ignoring replica bundles {TDRAnvilBundleFQID(uuid='56dc1d51-08ed-aec4-9127-4947fba536d9', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_activity', batch_prefix='659'), TDRAnvilBundleFQID(uuid='b2518a34-9a7f-acca-9bf1-5dc65c90fa5c', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='file_inventory', batch_prefix='650'), TDRAnvilBundleFQID(uuid='85780270-6151-aeda-b0ab-61bee1e772d4', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='7'), TDRAnvilBundleFQID(uuid='0c49c6d1-d2a7-ab97-9b65-819774b73201', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='file_inventory', batch_prefix='657'), TDRAnvilBundleFQID(uuid='f925cf5b-898c-a9a0-8315-14d895550569', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='participant', batch_prefix='65f'), TDRAnvilBundleFQID(uuid='e39f9662-0a1e-a40d-94df-ebde5d6f243e', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_activity', batch_prefix='65d'), TDRAnvilBundleFQID(uuid='dc35e287-40bd-a53b-9031-715d3415683a', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='workspace_attributes', batch_prefix='f'), TDRAnvilBundleFQID(uuid='7c1b09c9-6731-a98d-8f4b-2110e176033b', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sequencing', batch_prefix='5'), TDRAnvilBundleFQID(uuid='c4e6458f-d6a9-a5f2-8f92-a5a8cc9b0ad3', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_activity', batch_prefix='654'), TDRAnvilBundleFQID(uuid='066a2707-03c2-ae99-bffd-999f4650a252', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='file_inventory', batch_prefix='b'), TDRAnvilBundleFQID(uuid='d78ed2c7-d784-a59e-a9ae-46e47856ac22', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='family', batch_prefix='7'), TDRAnvilBundleFQID(uuid='6de02232-ef88-a76e-9763-18af94e57b53', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='pedigree', batch_prefix='656'), TDRAnvilBundleFQID(uuid='1e5be681-dc5e-a818-9462-ea5eb6aba626', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='sample', batch_prefix='654'), TDRAnvilBundleFQID(uuid='5f64f8f2-de68-afbd-9f86-251ba15f5c06', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='participant', batch_prefix='65b'), TDRAnvilBundleFQID(uuid='09c46b12-4b0c-a9d3-9778-77378ccff785', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_donor', batch_prefix='3'), TDRAnvilBundleFQID(uuid='a3c31d1f-e00e-a190-97db-1e14850d6a6c', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='workspace_attributes', batch_prefix='8'), TDRAnvilBundleFQID(uuid='a04f5482-4ef0-a711-87e2-b4b16cd000cd', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='5'), TDRAnvilBundleFQID(uuid='da4a2978-b2f3-aac5-a5ef-3b74c9948184', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='a'), TDRAnvilBundleFQID(uuid='5092af2e-2187-aa16-9690-8d3724d59948', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='sample', batch_prefix='651'), TDRAnvilBundleFQID(uuid='c47e1e97-3b87-a261-a5b7-14bc7b7ac77f', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sequencing', batch_prefix='1'), TDRAnvilBundleFQID(uuid='71ab4a75-8d61-a289-933d-154123ef15a7', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='file_inventory', batch_prefix='7'), TDRAnvilBundleFQID(uuid='572d0b8b-98d5-a442-a97f-7b1248d5210f', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='file_inventory', batch_prefix='65f'), TDRAnvilBundleFQID(uuid='2b053d15-aeac-aa08-9ab5-9cbdc8c648fa', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sequencing', batch_prefix='e'), TDRAnvilBundleFQID(uuid='b39624e9-7e5a-adcc-bd7b-3353756175ed', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_activity', batch_prefix='657'), TDRAnvilBundleFQID(uuid='4a2b1e50-82bd-aac2-9c40-d745aac7996d', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='subject', batch_prefix='9'), TDRAnvilBundleFQID(uuid='ce4f2dd7-80d3-a7d4-8fff-c9807f23a066', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='participant', batch_prefix='654'), TDRAnvilBundleFQID(uuid='4375f664-4114-a63e-b76c-89c22a6e5fe8', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='subject', batch_prefix='3'), TDRAnvilBundleFQID(uuid='8ce249fc-a1e3-a50c-b079-84afd5fa91c3', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_donor', batch_prefix='4'), TDRAnvilBundleFQID(uuid='b541f6f5-339f-ad77-99dc-fd340832d87c', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='workspace_attributes', batch_prefix='2'), TDRAnvilBundleFQID(uuid='68798973-3d1a-a158-a3cf-b3cfa3f68563', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='e'), TDRAnvilBundleFQID(uuid='0d40682d-9da1-aea9-998c-4a00a7d62fc5', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='qc_result_sample', batch_prefix='657'), TDRAnvilBundleFQID(uuid='b1b308c8-1264-aeb4-ae19-88c201ff59f3', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='workspace_attributes', batch_prefix='c'), TDRAnvilBundleFQID(uuid='1e52af6f-f2c0-a679-8e28-1685a225cd8d', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='8'), TDRAnvilBundleFQID(uuid='1dfc8876-c8a6-ace0-902d-53d543bfe599', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='file_inventory', batch_prefix='654'), TDRAnvilBundleFQID(uuid='5042d311-d1b7-a23f-8abb-662ad7a89aef', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sample', batch_prefix='d'), TDRAnvilBundleFQID(uuid='5915c7c7-cb8e-af45-be6b-faae037da0f1', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='qc_result_sample', batch_prefix='652'), TDRAnvilBundleFQID(uuid='339a53e4-bc2a-a77d-a18f-77cbedcef36b', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_donor', batch_prefix='2'), TDRAnvilBundleFQID(uuid='64267182-8a65-a134-b7d3-2692e8c3ac0a', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='file_inventory', batch_prefix='656'), TDRAnvilBundleFQID(uuid='98461546-b12f-af50-8f29-3ecb09607951', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='sample', batch_prefix='65a'), TDRAnvilBundleFQID(uuid='ff66ed4b-a104-a882-8f2d-f84e6621cfad', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='workspace_attributes', batch_prefix='6'), TDRAnvilBundleFQID(uuid='af18d83b-c0cc-a3a1-a6aa-3cce691a9a6f', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_donor', batch_prefix='e'), TDRAnvilBundleFQID(uuid='1c20f31d-a1ea-ad5c-9f68-451f363b8c16', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sequencing', batch_prefix='2'), TDRAnvilBundleFQID(uuid='cf0ec6b6-7827-ab15-9faa-7aaecab70b15', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sample', batch_prefix='3'), TDRAnvilBundleFQID(uuid='aa400ebf-c522-a672-99c4-f769360dee45', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_activity', batch_prefix='65c'), TDRAnvilBundleFQID(uuid='5cd96ab3-2052-a41d-837b-38a14ab13a41', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='file_inventory', batch_prefix='651'), TDRAnvilBundleFQID(uuid='b4f00071-ce45-a918-9da4-bb382b033d1c', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='workspace_attributes', batch_prefix='4'), TDRAnvilBundleFQID(uuid='d0f6d994-8bf0-a87d-bc6f-d98ac472daf7', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='f'), TDRAnvilBundleFQID(uuid='0651c958-9d83-afe0-9674-f9cdc35d0320', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='family', batch_prefix='4'), TDRAnvilBundleFQID(uuid='2b2adaf3-5728-aa01-aafe-16dcee738cf2', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='family', batch_prefix='5'), TDRAnvilBundleFQID(uuid='43679b83-16b9-a7eb-8c27-076bc5be0f90', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_sequencingactivity', batch_prefix='4'), TDRAnvilBundleFQID(uuid='2331fcad-c05b-a292-ba81-7f11cf2bc296', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='qc_result_sample', batch_prefix='65b'), TDRAnvilBundleFQID(uuid='1e34bea5-0606-a4a9-aa2e-48d8c4ad80d7', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='sample', batch_prefix='65c'), TDRAnvilBundleFQID(uuid='d85e8a6a-af43-a978-b9e8-8a613ab139cf', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='file_inventory', batch_prefix='d'), TDRAnvilBundleFQID(uuid='97b0971b-b80b-aab8-bbf9-2192e4ee5756', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_donor', batch_prefix='655'), TDRAnvilBundleFQID(uuid='0bdaa429-159b-a8a9-be89-962a16f68e53', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='file_inventory', batch_prefix='1'), TDRAnvilBundleFQID(uuid='cca9ee85-8f5b-ae6d-b45f-6b806c325edf', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='sequencing', batch_prefix='a'), TDRAnvilBundleFQID(uuid='29c138eb-3879-adce-a507-e5282a9c0649', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='790795c4-49b1-4ac8-a060-207b92ea08c5', spec=TDRSourceSpec(prefix=Prefix(common='65', partition=1), type=, domain=, subdomain='datarepo-dev-e53e74aa', name='ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732')), table_name='anvil_activity', batch_prefix='65e'), TDRAnvilBundleFQID(uuid='01dff31d-bbbc-a1e8-803d-73b928f77a88', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=, subdomain='datarepo-dev-97ad270b', name='ANVIL_CMG_Sample_1_20230225_ANV5_202302281509')), table_name='anvil_donor', batch_prefix='b'), TDRAnvilBundleFQID(uuid='35d8287a-e4c4-a1b0-bf85-2828ffcd35ed', version='2022-06-01T00:00:00.000000Z', source=TDRSourceRef(id='7d632e2a-54f1-4021-81fd-dbaf893715cd', spec=TDRSourceSpec(prefix=Prefix(common='', partition=1), type=, domain=