Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ ooil executable in a docker image #3458

Merged
merged 24 commits into from
Oct 25, 2022

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Oct 19, 2022

What do these changes do?

service-integration executable ooil is now released as an image. NOTE that this is NOT a new stack service!

  • build specs are added in services/docker-compose-build.yml so that the image is published and tagged analogous to the rest of the service stack images
  • ♻️ some fixes reported by type checker ( i.e. make mypy)
  • 🔨 adds python's .gitginore as part of .dockerignore
  • 🔨 VCS_REF takes now entire commit SHA
  • 🔨 minor update on sample/sleepers.py

Related issue/s

How to test

cd packages/service-integration
make build
make inspect
docker run -it local/service-integration:production --version

@codecov
Copy link

codecov bot commented Oct 19, 2022

Codecov Report

Merging #3458 (5c8b065) into master (97a99c0) will decrease coverage by 4.9%.
The diff coverage is 80.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #3458     +/-   ##
========================================
- Coverage    83.4%   78.4%   -5.0%     
========================================
  Files         828     792     -36     
  Lines       35183   33460   -1723     
  Branches      740     741      +1     
========================================
- Hits        29363   26257   -3106     
- Misses       5632    7026   +1394     
+ Partials      188     177     -11     
Flag Coverage Δ
integrationtests 50.5% <ø> (-17.2%) ⬇️
unittests 77.7% <80.0%> (-2.7%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...vice_integration/pytest_plugin/folder_structure.py 0.0% <0.0%> (ø)
...rvice_integration/pytest_plugin/validation_data.py 0.0% <0.0%> (ø)
...tion/src/service_integration/osparc_image_specs.py 94.1% <66.6%> (-5.9%) ⬇️
...e_integration/_compose_spec_model_autogenerated.py 100.0% <100.0%> (ø)
...service-integration/src/service_integration/cli.py 84.0% <100.0%> (ø)
...egration/src/service_integration/oci_image_spec.py 93.1% <100.0%> (ø)
...tegration/src/service_integration/osparc_config.py 88.4% <100.0%> (+0.2%) ⬆️
...-v2/src/simcore_service_director_v2/cli/_client.py 0.0% <0.0%> (-100.0%) ⬇️
...v2/src/simcore_service_director_v2/cli/__init__.py 0.0% <0.0%> (-100.0%) ⬇️
...service_director_v2/cli/_close_and_save_service.py 0.0% <0.0%> (-100.0%) ⬇️
... and 134 more

@pcrespov pcrespov self-assigned this Oct 19, 2022
@pcrespov pcrespov force-pushed the is3418/ooil-fsgenerator branch from 75b934c to 604cb82 Compare October 24, 2022 19:32
@pcrespov pcrespov changed the title WIP: ✨ is3418/ooil-fsgenerator ✨ ooil executable in a docker image Oct 24, 2022
@pcrespov pcrespov marked this pull request as ready for review October 24, 2022 22:10
@pcrespov pcrespov added the a:ooil integration-library or ooil label Oct 24, 2022
@pcrespov
Copy link
Member Author

pcrespov commented Oct 24, 2022

@GitHK @sanderegg any hints on how to solve any of these mypy issues appreciated :-)

src/service_integration/compose_spec_model.py:8:1: error: Module "service_integration._compose_spec_model_autogenerated" has no attribute "SCHEMA_VERSION"
src/service_integration/compose_spec_model.py:8:1: error: Module "service_integration._compose_spec_model_autogenerated" has no attribute "BuildItem"
src/service_integration/compose_spec_model.py:8:1: error: Module "service_integration._compose_spec_model_autogenerated" has no attribute "ComposeSpecification"
src/service_integration/compose_spec_model.py:8:1: error: Module "service_integration._compose_spec_model_autogenerated" has no attribute "Service"
src/service_integration/compose_spec_model.py:8:1: error: Module "service_integration._compose_spec_model_autogenerated" has no attribute "Volume1"
src/service_integration/versioning.py:48:14: error: Variable "service_integration.versioning.SemanticVersionStr" is not valid as a type
src/service_integration/versioning.py:48:14: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
src/service_integration/versioning.py:65:14: error: Variable "service_integration.versioning.SemanticVersionStr" is not valid as a type
src/service_integration/versioning.py:65:14: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
src/service_integration/versioning.py:66:26: error: Variable "service_integration.versioning.SemanticVersionStr" is not valid as a type
src/service_integration/versioning.py:66:26: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
src/service_integration/oci_image_spec.py:143:22: error: Argument 1 to "LabelSchemaAnnotations" has incompatible type "**Dict[str, str]"; expected "datetime"
src/service_integration/oci_image_spec.py:143:22: error: Argument 1 to "LabelSchemaAnnotations" has incompatible type "**Dict[str, str]"; expected "AnyUrl"
src/service_integration/pytest_plugin/validation_data.py:106:28: error: Argument 2 to "isinstance" has incompatible type "object"; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]"
src/service_integration/pytest_plugin/docker_integration.py:41:5: error: Returning Any from function declared to return "str"
src/service_integration/pytest_plugin/docker_integration.py:81:9: error: Returning Any from function declared to return "Dict[Any, Any]"
src/service_integration/pytest_plugin/docker_integration.py:88:9: error: Returning Any from function declared to return "Dict[Any, Any]"
src/service_integration/osparc_config.py:72:9: error: Returning Any from function declared to return "DockerComposeOverwriteCfg"
src/service_integration/osparc_config.py:88:13: error: Returning Any from function declared to return "DockerComposeOverwriteCfg"
src/service_integration/osparc_config.py:110:9: error: Returning Any from function declared to return "MetaConfig"
src/service_integration/osparc_config.py:117:9: error: Returning Any from function declared to return "MetaConfig"
src/service_integration/commands/config.py:50:25: error: Name "labels" already defined on line 47
src/service_integration/cli.py:51:34: error: Argument 1 to "AppSettings" has incompatible type "**Dict[str, str]"; expected "Dict[str, Registry]"
Found 20 errors in 8 files (checked 23 source files)

Copy link
Member

@mrnicegyu11 mrnicegyu11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed the code, looking good, did not fully go through all codeflows tbh :) Thanks a lot for this!

packages/service-integration/Makefile Outdated Show resolved Hide resolved
services/docker-compose-build.yml Show resolved Hide resolved
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a few questions:

  1. We already have ci-service-integration-library which is published to GitHub that already contains ooil.
  2. Why is this an image that is not part of simcore released with simcore?
  3. Where it will be pushed, how it will be versioned?
  4. How would this be used?

Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, a few comments

.github/workflows/ci-testing-deploy.yml Outdated Show resolved Hide resolved
packages/service-integration/Dockerfile Outdated Show resolved Hide resolved
Comment on lines +23 to +25
org.opencontainers.image.created: "${BUILD_DATE}"
org.opencontainers.image.source: "${VCS_URL}"
org.opencontainers.image.revision: "${VCS_REF}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that the new standard? shall we create an issue and change them all?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. I will update the others in a separate maintenance PR. There are other entries I want to add.

@@ -31,3 +31,172 @@ dask*-space/

# mypy cache
**/.mypy_cache/


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a thought but: the idea with dockerignore files I think are:

  • ignore everything *
  • then only add what is necessary as this is used in the docker build to set up the context and might make the build magnitudes faster. I guess I will review this one of these Fridays

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a few questions:

  1. We already have ci-service-integration-library which is published to GitHub that already contains ooil.
  2. Why is this an image that is not part of simcore released with simcore?
  3. Where it will be pushed, how it will be versioned?
  4. How would this be used?

discussed offline

Copy link
Member Author

@pcrespov pcrespov Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a thought but: the idea with dockerignore files I think are:

  • ignore everything *
  • then only add what is necessary as this is used in the docker build to set up the context and might make the build magnitudes faster. I guess I will review this one of these Fridays

In principle I agree with this approach but IMO there is a flaw. Say that you want to include packages/servicelib. Using your approach you exclude everything except whatever is in that folder. The problem is, what happens e.g. with all the artifacts produced for python development (i.e. __cache__, eggs, etc) inside of that folder? Shouldn't it make sense that the .dockerignore includes as well the .gitignore?

@pcrespov pcrespov requested a review from GitHK October 25, 2022 10:50
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions below

packages/service-integration/Dockerfile Outdated Show resolved Hide resolved
packages/service-integration/Dockerfile Outdated Show resolved Hide resolved
@@ -2,10 +2,11 @@
# filename: https://raw.githubusercontent.com/compose-spec/compose-spec/master/schema/compose-spec.json
# timestamp: 2021-11-19T10:40:07+00:00

from __future__ import annotations
# type: ignore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this at the beginning of the file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to ignore the entire file from mypy. Thisi s auto-generated

@pcrespov pcrespov added this to the Katherine Switzer milestone Oct 25, 2022
@pcrespov pcrespov force-pushed the is3418/ooil-fsgenerator branch from 1b51983 to 5c8b065 Compare October 25, 2022 13:34
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@pcrespov pcrespov merged commit 86b0800 into ITISFoundation:master Oct 25, 2022
@pcrespov pcrespov deleted the is3418/ooil-fsgenerator branch October 25, 2022 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:ooil integration-library or ooil
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants