Use HeadObject to retrieve object metadata

[JonMerlevede]

This PR addresses #2311 by using HeadObject instead of GetObject to retrieve object metadata such as sse_kms_key_id. This severely speeds up queries requiring object metadata but not object contents.

Notes:

• Although the AWS documentation of GetObjectAttributes mentions it combining the functionality of HeadObject and ListParts, many metadata (Expires, ServerSideEncryption, ...) seem to appear only in the results of HeadObject. I did not manage to retrieve them as an object attribute.
• Calls to GetObjectAttributes possibly could be replaced by calls to ListParts.
• If GetObject needs to be called, there is never a need also to call HeadObject; I do not know if this is easily expressible in the Steampipe framework.
• I don't know how I should run the tests, and there appear to be no tests yet for aws_s3_object, but I would be happy to look into this if someone can point me in the right direction.

[ParthaI]

Hello @JonMerlevede,

The rationale behind the proposed changes looks great! I agree that users may not always require the object content when querying the table, so using HeadBucket to populate the column values seems like a good approach.

Could you kindly test these changes with larger objects and share the query execution times (with and without the changes in this PR)?

Here are the steps to measure the query timing:

1. Install the latest AWS plugin build (v0.147.0): steampipe plugin install [email protected]
2. Build the code changes in this PR from the JonMerlevede:feature/s3-head-object branch by running the command make.
3. Create two separate connections in your aws.spc file for the two different AWS plugin versions, like so:

   connection "aws_local_build" {
     plugin = "aws"
     regions = ["*"]
     profile = "default"
   }
   
   connection "aws_current" {
     plugin = "[email protected]"
     regions = ["*"]
     profile = "default"
   }

4. Run steampipe query.
5. Enable query timing by running .timing on.
6. Execute the following queries for both connections:
   • Local build: select key, version_id from aws_local_build.aws_s3_object where bucket_name = '<name of the bucket containing larger objects>'
   • AWS v0.147.0: select key, version_id from aws_current.aws_s3_object where bucket_name = '<name of the bucket containing larger objects>'

Please compare the query execution times and share the differences.

Thanks so much for giving this a try and sharing your observations!