Define basics for collecting Iceberg metadata for the current snapshot

[phet]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1707 - "Add Iceberg support to DistCp"

Description

• Here are some details about my PR, including screenshots (if applicable):

Toward eventual full DistCp Iceberg support: walk an iceberg's metadata to determine all physical file paths for both metadata and actual data. At present these are the full contents; subsequently we'll calculate a delta against previously copied data.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

I tested manually, and working now to create a test iceberg that I could check into src/test/resources, which has no PII. that would be a fast follow along w/ the test, if I'm unable to generate that before committing this (to share code and unblock collaborators).

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[Will-Lo]

If these objects are logically linked and order enforced, would it make sense to make it a list of a new class, say IcebergManifestFiles objects? And each object would contain a manifest file path and a manifest list. and this object can have an interface to fetch all the files for getAllPaths()

[phet]

makes sense. I added a helper POJO class

[codecov-commenter]

Codecov Report

Merging #3559 (47d5f7f) into master (d0e530a) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #3559      +/-   ##
============================================
- Coverage     46.81%   46.78%   -0.03%     
  Complexity    10514    10514              
============================================
  Files          2095     2099       +4     
  Lines         81945    81985      +40     
  Branches       9129     9131       +2     
============================================
- Hits          38361    38357       -4     
- Misses        40040    40086      +46     
+ Partials       3544     3542       -2     

Impacted Files	Coverage Δ
...management/copy/iceberg/IcebergCatalogFactory.java	0.00% <0.00%> (ø)
...ta/management/copy/iceberg/IcebergHiveCatalog.java	0.00% <0.00%> (ø)
...a/management/copy/iceberg/IcebergSnapshotInfo.java	0.00% <0.00%> (ø)
...lin/data/management/copy/iceberg/IcebergTable.java	0.00% <0.00%> (ø)
...e/gobblin/service/monitoring/GitConfigMonitor.java	95.23% <0.00%> (-4.77%)	⬇️
...a/org/apache/gobblin/cluster/GobblinHelixTask.java	81.72% <0.00%> (-2.16%)	⬇️
.../modules/scheduler/GobblinServiceJobScheduler.java	64.11% <0.00%> (-1.44%)	⬇️
...main/java/org/apache/gobblin/util/HadoopUtils.java	29.54% <0.00%> (-0.33%)	⬇️
...anagement/copy/replication/ConfigBasedDataset.java	68.87% <0.00%> (ø)
.../apache/gobblin/runtime/api/JobExecutionState.java	80.37% <0.00%> (+0.93%)	⬆️
... and 1 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more