feat: Add AWS SQS publisher and example job file.

[Wonong]

Summary of Changes

• AWS SQS publisher takes two folders of csv files(nodes and relations) and publish it to AWS SQS queue
• Common expected usage is running job on client-side cluster(environment of DBMS) and push extracted metadata to message queue
• Its message format is like below.

{
  "nodes": [
    {
      "LABEL": "...",
      "KEY": "...",
      "description_source": "...",
      "description": "..."
    }, ...
  ],
  "relations": [ 
    {
      "START_KEY": "...",
      "START_LABEL": "...",
      "END_KEY": "...",
      "END_LABEL": "...",
      "TYPE": "...",
      "REVERSE_TYPE": "..."
    }, ...
  ]
}

Tests

Test it using container executing sample job periodically using crontab and it works.
test repo: https://github.com/Wonong/ab-metadata-pusher

Documentation

https://github.com/Wonong/ab-metadata-pusher

CheckList

Make sure you have checked all steps below to ensure a timely review.

• PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• PR includes a summary of changes.
• PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
• PR passes make test

[feng-tao]

besides unit test, is the pr ready for review? exciting about supporting sqs publisher in databuilder etl pattern!

[Wonong]

This pr is ready. but, I want you to review both this pr and the other pr which implement extractor which extracts metadata from sqs. I'm currently working on extractor PR.
So, I will reopen this PR when I finish extractor PR and also add unit tests of this PR as you said :)

[feng-tao]

sg, thanks @Wonong

[feng-tao]

@Wonong just to check, have you had the chance to continue the pr?

[Wonong]

@feng-tao sorry for no progress... I had been busy these days. but, I can continue the pr soon if you don't mind

[feng-tao]

@Wonong that would be great!

[Wonong]

@feng-tao
I just wonder your opinion.
How about update job class to be able to create a non-task job(only publisher)?
I think it is easy to update. but original concept of job(described in readme) say publisher is only optional.
Below is simple implementation of none-task available job class. I think implementing a NoopTask like NoopPublisher works too.
https://github.com/Wonong/amundsendatabuilder/blob/non-task-job/databuilder/job/job.py#L64

[dorianj]

Hey @Wonong, I'd suggest using a NoopPublisher would work fine here.

We're now in a code-freeze period as part of the monorepo transition. If you can get this landed in the next few days, I'll cleanly transfer it over to the new repo. If not, it'll need to be manually re-opened against the new repo.

[Wonong]

Hey @Wonong, I'd suggest using a NoopPublisher would work fine here.

We're now in a code-freeze period as part of the monorepo transition. If you can get this landed in the next few days, I'll cleanly transfer it over to the new repo. If not, it'll need to be manually re-opened against the new repo.

sorry for late answer. I think i need more time. so, i will manually open it if needed. thank you for inform :)

[dorianj]

OK, happy to review on the monorepo :)