[Issue] Problem with two APIs

[Bundy01]

Describe the bug

• The GitHub API is very slow (see [GITHUB] Very slow GitHub API #1393)
• Hunter's API gives me an error with the -v option:

[WRN] Could not run source hunter: 令牌过期 (trad: expired token)

I recreated the tokens, without success.

I've contacted GitHub and Hunter and it looks like it's from SubFinder.

Are you observing the same thing?

Subfinder version
Current Version: v2.6.6

Complete command you used to reproduce this
subfinder -nW -dL FILE -s github_or_hunter -v

Config file
The uncommented lines:

sources: [alienvault,anubis,binaryedge,chaos,crtsh,digitorus,dnsdumpster,github,hackertarget,netlas,rapiddns,securitytrails,virustotal,waybackarchive]
rate-limits: ["github=83/m", "hackertarget=2/s", "netlas=1/s", "securitytrails=1/s", "sitedossier=8/m", "virustotal=4/m", "waybackarchive=15/m"]
t: 15
ip: false
config: /home/***/.config/subfinder/config.yaml
provider-config: /home/***/.config/subfinder/provider-config.yaml
r: [149.112.112.10,1.0.0.1,77.88.8.8]
exclude-ip: true
stats: true
timeout: 30
max-time: 30

[dogancanbakir]

I replied to the discussion for GitHub. Looking for hunter API key, we can speed up the debugging process by you reaching out over Discord - @dogancanbakir is my handle.

[Bundy01]

@dogancanbakir for GitHub API, it's possible that it's due to the limitation of the latter. I've put the token's limitations in the discussion.

[dogancanbakir]

I’ve sent a fix for GitHub. Could you test it and let me know the results? #1421

[Bundy01]

@dogancanbakir

I'm seeing the same time savings as in the PR example.

However, Chaos API for example, is faster for this domain. Perhaps there's still some progress to be made, or is it due to Github API?
Tomorrow or Monday at the latest, I need to run a scan with 122 domains, which took me about 12 hours with Github API included.

I'll tell you the time difference then.
Thanks.

[Bundy01]

I'm back to say that the scan took me 4h35 instead of about 12h.
Despite adding the commoncrawl,leakix and subdomaincenter sources. I also set goroutines back from 15 to 10.

I'd like to take this opportunity to ask a question about this option. Should the value be equal to the number of sources or could it create errors in sources that are limited in requests if the value is too high?

The commit is really good, but I've noticed that three times, the github stats had negative values.

[dogancanbakir]

The -t flag is used for subdomain resolution rather than controlling the go routines in sources. And for the negative values, could you create another issue for me to investigate? Thanks.

[Bundy01]

I noticed that when playing with the -t option, the result with the -nW option changed enormously.
Is this normal?

[dogancanbakir]

Yes, as I said before the -t flag is used for subdomain resolution (active) rather than controlling the go routines in sources.