What level of (Terraform) parallelism is a good idea?

[AaronCoquet-Easypark]

We use a Makefile to run Terraform. Our plan actions take this form:
terraform plan -var-file (...) -parallelism=250 -out tfplan -no-color 2>err.txt | tee plan.txt;

These plans typically run fairly fast, but I realized that Snowflake Warehouses are typically set to max_concurrency_level = 8, meaning that only 8 queries can run at a time.

I started thinking about this because I've had several plans recently just ... hang. The output will be flying by and then suddenly stops and gives this:

snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Reading...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (10s) ...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (20s) ...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (30s) ...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (40s) ...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (50s) ...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (1m) ...
...
snowflake_tables.all_tables["<DB>.<SCHEMA>"]: Still reading (2m 30s) ...

It usually catches up after 2.5 to 3 minutes, and there are usually several resources that get stuck all at once. But! Sometimes it doesn't catch up, and I get a raft of "Unexpected EOF" errors.

I tried similar plans without the -parallelism=250 argument, and it's MUCH slower... and doesn't seem to resolve the issue.

So I guess I have 3 (ish) questions:

1. What causes that weird lag? (Or: Does anyone else experience this?)
2. What level of -parallelism strikes a balance between "Fast" and "Stable"? (Or: What's a "good" -parallelism value?)
3. What Warehouse configuration(s) do y'all use / suggest?

[sfc-gh-jmichalak]

Hi @AaronCoquet-Easypark 👋

1. IMO, setting p=250 might be overkill and may cause network issues. Why is the parallelism value so big? What's the size of your TF deployment? In general, we suggest having smaller deployments of 100s resources instead of 1000s. This is also recommended by HashiCorp itself.
2. It may depend on your environment, we don't have a specific number here. Recently, we performed some benchmarks (we'll publish the results soon), and there's a small difference between 40 and 60, so the gain probably diminishes further.
3. We haven't explored the max_concurrency_level field. It's limited by the MAX_CONCURRENCY_LEVEL parameter, but it can be increased (see Snowflake docs). If you think that's the bottleneck here, have you tried setting bigger values?
4. Reading tables shouldn't take that long. It's hard to know what's causing this: please provide more logs with TF_LOG=DEBUG for at least the initial tables.

[AaronCoquet-Easypark]

Hey! Thanks for the quick response!

1:
p=250 is almost certainly overkill... it's legacy from before I joined, though, so I don't know the reasons.

I'm currently working to reduce the size of our states. We had 2 states per instance that grew organically ("tables" and "users"). In our second-largest instance, tables has 1236 and users has 1494 resources. After I finish remodeling, we will have 15 states, each covering a different configuration area (Admin setups, Instance parameters, Human users, service accounts, databases, ...). The biggest states (humans and masking policies) are going to have resource counts in the 500 to 700 range. I'm afraid of the largest instance counts...

2:
I'll check the benchmarks when you post them! Do we know what the default is (when the parameter isn't set)?

3:
I don't know if it's a bottleneck. For a lot of resources, the warehouse isn't used at all; in those cases it wouldn't have any impact. I haven't tinkered with the WH settings, because (currently) Terraform doesn't have a dedicated warehouse.

4:
I'll post logs once i have them. It's REALLY inconsistent behavior, though.