Random timeout issues

[Franklin89]

Checklist

• I have looked into the README and have not found a suitable solution or answer.
• I have looked into the documentation and have not found a suitable solution or answer.
• I have searched the issues and have not found a suitable solution or answer.
• I have upgraded to the latest version of this tool and the issue still persists.
• I have searched the Auth0 Community forums and have not found a suitable solution or answer.
• I agree to the terms within the Auth0 Code of Conduct.

Description

When running the tool with either Export or Import I am randomly getting Timeout errors. I am running the export command which worked a couple of times and then it did not work a few times now it is working again.

a0deploy export --config_file Sandbox.json --format yaml --output_folder ./Sandbox

This error appears at random stages.

APIError: connect ETIMEDOUT ***.**.***.**:443

I switched internet connect, from WLAN to LAN to mobile LTE. Same behavior.

The following is happening when I run it from GitHub Actions:

Expectation

The command to work!

Reproduction

1. Run the command in the description

Deploy CLI version

7.15.1

Node version

v16.13.0

[r1m]

Same in windows/wsl with cli in version 7.15.1 or 7.14.3.
export is always failing at branding data
a0deploy export --config_file config.json --format directory --output_folder export --debug

It's a Development Tenant

[Franklin89]

I updated Node to the latest version. So, Node Version v19.0.0. Which is not LTS as far as I know but for now this seems to have fixed the issue on my side. @r1m maybe worth trying on your side too?

[willvedd]

This has popped up before: #562. I would expect this to be an intermittent occurrence; upgrading to Node 19 is not likely to be a permanent solve. There are several reasons why this issue could occur but I initially suspect one of:

• Port constraints on hardware or VM. Deploy CLI tends to be greedy with network requests, not unheard of to hit up against limits.
• Auth0 server-side error. As the message suggests, the error is originating when making a request to the API. If multiple users are experiencing this at the same time, a brief outage might be to blame.

export is always failing at branding data

@r1m I'd test your theory by configuring the AUTH0_EXCLUDED configuration property with something like the following to see if it resolves the issue:

{
  "AUTH0_EXCLUDED": ["branding"]
}

I'd also recommend excluding "prompts" too as that could be an expensive resource to manage.

[r1m]

I managed to get a full export working after several tries. Branding is most of the time the failing step but it is not 100% reproducible.
I will use partial export to workaround this. Thanks for the tip.

[ewanharris]

@Franklin89, Node.js 19 could possibly alleviate some of the pressure from the amount of requests as it enables keepAlive by default. That should allow HTTPS connections to be reused by multiple requests rather than opening a new connection every time.

For folks that hit this type of issue it might be worthwhile testing out Node.js 19 to see if it is more stable

[willvedd]

Closing this one out. While the Deploy CLI can certainly be resource-hungry at times, it does so for performance reasons. Perhaps there is room for optimization in the future, but until then, there is clear guidance on how to "fix".

[burck1]

This is still an issue for us. We are regularly seeing the error message APIError: connect ETIMEDOUT <IP>:443 when running the deploy CLI, but the error doesn't always occur for the same resources:

Error: Problem updating roles ...
APIError: connect ETIMEDOUT 104.16.216.241:443
    at node_modules\auth0-deploy-cli\lib\tools\auth0\handlers\roles.js:159:27 
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { 
  type: 'roles', 
  stage: 'processChanges' 
} 

Error: Problem updating databases ...
APIError: connect ETIMEDOUT 104.16.217.241:443 
    at node_modules\auth0-deploy-cli\lib\tools\auth0\handlers\default.js:231:31 
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { 
  type: 'databases', 
  stage: 'processChanges' 
}

@willvedd, you mentioned "there is clear guidance on how to fix", but it is not clear to me. We have a policy to only use Active LTS Node.js versions in our CI environment (we're on the latest v18) so upgrading to Node 19 is not an option for us. What guidance would you recommend? Our CI environment (based on the docs) is essentially failing on every deployment. Every once and a while one will fully succeed the deployment, but more often than not it fails.

I see that keep alive support was recently merged into the node-auth0 v3.7.0 library for this exact error, "We experienced a lot of ETIMEDOUT errors ...". Would it be possible to update the auth0-deploy-cli to use that keep alive configuration or at least allow a pass-through configuration setting for keep alive?