Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add vector tile generation docker resources #15

Merged
merged 4 commits into from
Jul 14, 2022
Merged

Add vector tile generation docker resources #15

merged 4 commits into from
Jul 14, 2022

Conversation

junqiu-lei
Copy link
Member

@junqiu-lei junqiu-lei commented Jun 16, 2022

Signed-off-by: Junqiu Lei [email protected]

Description

  1. Add vector tiles generation docker resources;
  2. Add the docker resources to CDK pipeline to make it executable on ECS;

There are there main parts on the container task workflow:

  1. Use plantiler to generate mbtiles(a file format for storing tilesets.)
  2. Use mbutils to extract mbtiles to pbf(Protocolbuffer Binary Format) files.
  3. Use s5cmd to upload pbf files to s3 bucket.

The ECS task execution workflow: here

Issues Resolved

#12

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@junqiu-lei junqiu-lei requested a review from a team June 16, 2022 01:18
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

if [ "$1" == "generate" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add help command to display supported options ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I plan to create a document file to explain usages

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not include this inside script only?

Copy link
Member Author

@junqiu-lei junqiu-lei Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

// Copyright OpenSearch Contributors
// SPDX-License-Identifier: Apache-2.0

// In order to support add metadata 'Content-Encoding=gzip' and 'Content-Type=application/x-protobuf' on copy command,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try extending from library instead of copying complete src code?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Offline discussed with Vijay, it can't be used by extending from library due to method not public. And I will keep the existing method that replacing some codebase and build them in docker. I will also create a github issue on s5cmd repo to support the feature.

cd /mbutil-0.3.0 && \
python3 setup.py install

RUN wget -O s5cmd.tar.gz https://github.com/peak/s5cmd/archive/refs/tags/v2.0.0-beta.2.tar.gz && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to depend on beta version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since 2.0 has some bug fixes and improvements, in our case it's stable enough to use the copy operation, and before 2.0 beta, the 1.4 was released last year Sep 21, 2021. https://github.com/peak/s5cmd/releases

Signed-off-by: Junqiu Lei <[email protected]>
Copy link
Collaborator

@navneet1v navneet1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per my understanding we were thinking to automate this whole setup rather than doing it via scripts and docker images.

@junqiu-lei
Copy link
Member Author

@navneet1v yes, adding the docker resources is the first step, then I will add the docker resources to ECS CDK resources to make it a one stop executable task on ECS. Example here

@junqiu-lei junqiu-lei requested a review from a team July 12, 2022 21:31
VijayanB
VijayanB previously approved these changes Jul 12, 2022
@junqiu-lei
Copy link
Member Author

Since we changed the CDK stack name, the github actions test CDK will be failed, and I have updated the actions workflow in this PR as well.

tiles-generation/cdk/bin/cdk.ts Outdated Show resolved Hide resolved
tiles-generation/cdk/bin/cdk.ts Outdated Show resolved Hide resolved
tiles-generation/cdk/bin/cdk.ts Outdated Show resolved Hide resolved
tiles-generation/cdk/bin/cdk.ts Outdated Show resolved Hide resolved
tiles-generation/cdk/lambda/event-process.js Show resolved Hide resolved
@junqiu-lei junqiu-lei requested a review from navneet1v July 13, 2022 17:11
@junqiu-lei junqiu-lei merged commit 3890fb8 into opensearch-project:main Jul 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants