Athena Data Catalogs doesn't work and impacts fetching resources

[Chan9390]

Describe the bug

I tried fetching all resources in a bunch of AWS accounts but it always got the error rpc error: code = Unavailable desc = error reading from server: EOF

Detailed logs from cloudquery.log:

{"level":"error","instance_id":"221042b7-ed4f-407b-8ebe-befd1b7822fc","provider":"aws","alias":"aws","error":"rpc error: code = Unavailable desc = error reading from server: EOF","time":"2022-07-28T11:46:08Z","message":"received unexpected provider fetch error"}
{"level":"error","instance_id":"221042b7-ed4f-407b-8ebe-befd1b7822fc","error":"exit status 2","path":".cq/providers/cloudquery/aws/v0.13.2-linux_amd64","pid":942674,"time":"2022-07-28T11:46:08Z","message":"plugin process exited"}
{"level":"error","instance_id":"221042b7-ed4f-407b-8ebe-befd1b7822fc","type":"Internal","resource":[],"details":"","time":"2022-07-28T11:46:09Z","message":"rpc error: code = Unavailable desc = error reading from server: EOF"}

Then I tried to see if cloudquery didn't work in individual accounts. Surprisingly it worked in all accounts except few. Then I tried debugging the resources that caused this error in those failed accounts.

Finally found athena.data_catalogs didn't work.

Expected Behavior

Athena Data Catalogs option is deprecated in aws cli is now available in Glue. Not sure what should be the expected behavior.

Steps to Reproduce

Sample policy to reproduce the issue:

providers:
  - name: aws
    configuration:
      accounts:
        - account_name: dev-account
          local_profile: dev-account
      regions:
        - us-east-1
        - ap-south-1
    resources:
        - athena.data_catalogs

Possible Solution

Not sure about the possible solution.

But I do have some suggestions based on my experiments with Cloudquery so far. Having the following behavior would have saved me a lot of time :)

• If one AWS account in multiple accounts fail, then fail safe for that account only along with a warning or error in logs for the failed account. Let the other accounts continue as expected. (In case I use resources: * so future assets get added automatically, this default behavior would fail the entire setup because of one failing resource check)
• If fetching a particular resource fails, explicitly mention which resource failed instead of exiting with non-zero status code and having a generic error statement. It would be easier to debug the issue.

Provider and CloudQuery version

Version: 0.32.1 Commit: d324b37050836c6533a7d208d8e54960fb04e386 Date: 2022-07-27T10:53:35Z

Additional Context

[bbernays]

@Chan9390 - Thank you for the bug report as well as the suggestions on how to improve! We have opened PR to fix (what we think) is the issue and will continue to look into the suggestions you made