CLOUDFLARE: adopt ZoneCache

[das7pad]

This PR is adopting the new ZoneCache for the CLOUDFLARE driver.

The integration tests are passing (CF_REDIRECT tests skipped). I've done some extra manual testing on the zone creation plus provisioning a zone in a single pass, see below.

(#3330 plus the zone cache changes as linked in #3337)

Manual testing

Preview, can create zone and preview new records

var REG_NONE = NewRegistrar('none')
var DSP = NewDnsProvider('CLOUDFLAREAPI')

D('testing-2025-01-15-0.dev', REG_NONE, DnsProvider(DSP),
    A('@', '127.0.0.1')
)

$ dnscontrol preview
******************** Domain: testing-2025-01-15-0.dev
1 correction (CLOUDFLAREAPI)
#1: Ensuring zone "testing-2025-01-15-0.dev" exists in "CLOUDFLAREAPI"
Added zone for testing-2025-01-15-0.dev to Cloudflare account: xxx
SUCCESS!
CONCURRENTLY gathering 1 zone(s)
SERIALLY gathering 0 zone(s)
Waiting for concurrent gathering(s) to complete...DONE
******************** Domain: testing-2025-01-15-0.dev
1 correction (CLOUDFLAREAPI)
#1: + CREATE testing-2025-01-15-0.dev A 127.0.0.1 proxy=false ttl=1
Done. 2 corrections.

Push, can add record to existing zone and create and add records in one go

var REG_NONE = NewRegistrar('none')
var DSP = NewDnsProvider('CLOUDFLAREAPI')

D('testing-2025-01-15-0.dev', REG_NONE, DnsProvider(DSP),
    A('@', '127.0.0.1')
)

D('testing-2025-01-15-1.dev', REG_NONE, DnsProvider(DSP),
    A('@', '127.0.0.1')
)

$ dnscontrol push
******************** Domain: testing-2025-01-15-1.dev
1 correction (CLOUDFLAREAPI)
#1: Ensuring zone "testing-2025-01-15-1.dev" exists in "CLOUDFLAREAPI"
Added zone for testing-2025-01-15-1.dev to Cloudflare account: xxx
SUCCESS!
CONCURRENTLY gathering 2 zone(s)
SERIALLY gathering 0 zone(s)
Waiting for concurrent gathering(s) to complete...DONE
******************** Domain: testing-2025-01-15-0.dev
1 correction (CLOUDFLAREAPI)
#1: + CREATE testing-2025-01-15-0.dev A 127.0.0.1 proxy=false ttl=1
SUCCESS!
******************** Domain: testing-2025-01-15-1.dev
1 correction (CLOUDFLAREAPI)
#1: + CREATE testing-2025-01-15-1.dev A 127.0.0.1 proxy=false ttl=1
SUCCESS!
Done. 3 corrections.

// no corrections on retry
$ dnscontrol push
CONCURRENTLY gathering 2 zone(s)
SERIALLY gathering 0 zone(s)
Waiting for concurrent gathering(s) to complete...DONE
******************** Domain: testing-2025-01-15-0.dev
******************** Domain: testing-2025-01-15-1.dev
Done. 0 corrections.

[das7pad]

The integration tests are passing

@tlimoncelli How long does it take to run the integration tests for CLOUDFLARE on your end? It takes about 30min on my end, which I find quite slow. The HETZNER tests run in about a third of that, which is still slow. Do you usually run a subset of the tests during development?

[tlimoncelli]

The integration tests are passing

@tlimoncelli How long does it take to run the integration tests for CLOUDFLARE on your end? It takes about 30min on my end, which I find quite slow. The HETZNER tests run in about a third of that, which is still slow. Do you usually run a subset of the tests during development?

It takes about 14 minutes. Cloudflare is one of the slower providers.

You can see the timings here:
https://github.com/StackExchange/dnscontrol/actions/runs/12797268789
(click on "Matrix: integration-tests")

I would love it if they all took less than 5 minutes.

Tom

[tlimoncelli]

Do you usually run a subset of the tests during development?

Yes, I use the -start and -end flags to just run specific tests. However I don't merge to main until they all run cleanly, and that can take a long while.

I'm open to suggestions. Here are some ideas I've had:

1. Have a "fast test" version that skips all but the most crucial tests.
2. Work with each maintainer to speed up the code (like I did with TransIP recently)
3. Ask the maintainer to request a rate limit exception from the provider (some have done that!)
4. Use 2 domains and run half the tests on each.
5. Use a bigger VM for the tests. GHA's free tier runs these tests much, much, slower than on my desktop

I could use a volunteer to manage the relationships with the maintainers. First job would be to find them all (some emails now bounce), second job would be to get more providers into the automated testing, third would be to get the tests to run a lot faster.

Tom

[das7pad]

Thanks for the insights. I can throw in another idea for improving the wall-time of the tests:

Run independent tests concurrently:

• give each TestGroup a "namespace"/prefix: e.g. "complex TXT" gets prefix "0-", record "foo0" turns into "0-foo0"; "testByLabel" gets "1-", record "foo" turns into "1-foo"
• bundle the corrections of each TestCase step together and apply them
• provide each TestCase with their filtered records after applying

-> time ->

Before: TestGroup1
        `-> TestCase1 "foo" is "1.1.1.1" -> TestCase2 "foo" is "1.1.1.2"
                                                                        TestGroup2
                                                                        `-> TestCase1' "foo" is "2.2.2.2" -> TestCase2' "foo" is "2.2.2.3"

---

After: TestGroup1 -> TestCase1  "0-foo" is "1.1.1.1" -> TestCase2  "0-foo" is "1.1.1.2"
       TestGroup2 -> TestCase1' "1-foo" is "2.2.2.2" -> TestCase2' "1-foo" is "2.2.2.3"
                                                                                       TestGroup3 -- opting out of concurrency
                                                                                       `-> TestCase1'' -> TestCase2''

Some TestGroups might need to opt-out of this as they do something special, e.g. need to control the exact record name. Some drivers might hit limits on the number of records in a zone.

As many drivers can batch together multipler create/update operations in a single API calls, this should cut the wall-time from reducing the waterfall of requests. The number of requests will also drop, easing the pressure on rate-limits.

In the above example:

• Before: list, create 1, list, update 1, list; delete 1; list, create 1 , list, update 1, list, delete 1
• After: list, create 2, list, update 2, list, delete 2

[tlimoncelli]

@das7pad I agree about grouping updates. I think all (most?) providers do that if its an option.

Sadly the way the testcases work, we can run them concurrently unless we have a separate domain for each concurrent group. Not an impossible request, but it will require reworking the github action.

[das7pad]

Me two days ago: It takes about 30min on my end [to run the CLOUDFLARE integration tests], which I find quite slow.

Perhaps I should have compared the timings with main 🤦 #3394 brings them back to 11min.