⬆️ Maintenance/test and tools [week 19]

[pcrespov]

What do these changes do?

• ⬆️ Periodic update of tests and tool libraries .
• ⬆️ Fixed missing dependency downstream in dask-task-models-library: when jsonschema was introduced in models_library, we forgot to pip-compile downstream.
• ⬆️ Fixed storage tests requirements: Any change in _test.in (add/remove/modify) requires pip-compile it.
• ✨ Improved reporting scripts to scan for packages affected by the changes

Summary

This is a summary of the libraries updated in this PRs:

• #packages before: 50
• #packages after : 53

where 🔧=tools and 🧪= test requirements.

#	name	before	after	upgrade	count	packages
1	aio-pika	7.1.2	7.2.0	minor	1	swarm-deploy🧪
2	aioboto3	9.4.0, 9.5.0	9.6.0	minor	2	director-v2🧪 simcore-sdk🧪
3	aiobotocore	2.1.2, 2.2.0	2.3.0	minor	2	director-v2🧪 simcore-sdk🧪
4	astroid	2.11.2, 2.11.3	2.11.5		15	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🔧 models-library🧪 postgres-database🧪 service-integration🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage🧪 web🧪
5	bcrypt	3.2.0	3.2.2		6	api-server🧪 catalog🧪 director-v2🧪 postgres-database🧪 service-library🧪 web🧪
6	black	22.1.0	22.3.0	minor	1	service-integration🔧
7	boto3	1.20.24	1.21.21	minor	1	director-v2🧪
8	botocore	1.23.24	1.24.21	minor	1	director-v2🧪
9	click	8.0.4, 8.1.2	8.1.3		16	dask-task-models-library🧪🔧 models-library🔧 postgres-database🧪🔧 public-api🧪 service-integration🧪🔧 service-library🔧 settings-library🧪🔧 simcore-sdk🧪🧪🔧 swarm-deploy🧪🧪
10	cryptography	36.0.2	37.0.2	MAJOR	5	catalog🧪 dask-sidecar🧪 director-v2🧪 postgres-database🧪 service-library🧪
11	dask	2022.4.0	2022.5.0	minor	1	dask-task-models-library🧪
12	dask-gateway-server	0.9.0	2022.4.0	MAJOR	1	director-v2🧪
13	distributed	2022.4.0	2022.5.0	minor	1	dask-task-models-library🧪
14	email-validator	1.1.3	1.2.1	minor	5	dask-task-models-library🧪 models-library🧪 service-integration🧪 simcore-sdk🧪 swarm-deploy🧪
15	faker	13.3.3, 13.3.4, 13.3.5	13.7.0	minor	14	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🧪 models-library🧪 postgres-database🧪 public-api🧪 service-library🧪 simcore-sdk🧪 storage🧪 web🧪
16	fastapi	0.75.1, 0.75.0	0.76.0	minor	2	helpers🧪 service-library🧪
17	hypothesis	6.39.6	6.46.2	minor	1	web🧪
18	icdiff	2.0.4	2.0.5		7	dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 models-library🧪 storage🧪 web🧪
19	identify	2.4.12	2.5.0	minor	17	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 simcore-sdk🔧 storage🔧 swarm-deploy🧪 web🔧
20	jinja2	3.1.1	3.1.2		1	dask-task-models-library🧪
21	jmespath	0.10.0	1.0.0	MAJOR	1	director-v2🧪
22	jsonschema	4.4.0	4.5.1	minor	4	models-library🧪 public-api🧪 service-integration🧪 tests🧪
23	locket	0.2.1	1.0.0	MAJOR	1	dask-task-models-library🧪
24	more-itertools	8.12.0	8.13.0	minor	1	tests🧪
25	mypy	0.942	0.950	minor	1	dynamic-sidecar🔧
26	pandas	1.4.1	1.4.2		1	storage🧪
27	paramiko	2.10.3	2.10.4		6	api-server🧪 catalog🧪 director-v2🧪 postgres-database🧪 service-library🧪 web🧪
28	pillow	9.0.1	9.1.0	minor	1	director-v2🧪
29	pint	0.19.1, 0.18	0.19.2		4	dask-task-models-library🧪 models-library🧪 simcore-sdk🧪 swarm-deploy🧪
30	pip-tools	6.5.1	6.6.0	minor	16	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 storage🔧 swarm-deploy🧪 web🔧
31	platformdirs	2.5.1	2.5.2		29	api-server🧪🔧 catalog🧪🔧 dask-sidecar🧪🔧 dask-task-models-library🧪🔧 datcore-adapter🧪🔧 director-v2🧪🔧 dynamic-sidecar🔧 models-library🧪🔧 postgres-database🧪🔧 public-api🧪 service-integration🧪🔧 service-library🧪🔧 settings-library🧪🔧 storage🧪🔧 swarm-deploy🧪 web🧪🔧
32	pre-commit	2.17.0, 2.18.1	2.19.0	minor	17	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 simcore-sdk🔧 storage🔧 swarm-deploy🧪 web🔧
33	ptvsd	4.3.2	🗑️ removed		1	storage🧪
34	pylint	2.13.2, 2.13.7, 2.13.4	2.13.8		15	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🔧 models-library🧪 postgres-database🧪 service-integration🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage🧪 web🧪
35	pyparsing	3.0.7	3.0.8		14	dask-task-models-library🧪🧪 datcore-adapter🧪 environment-setup🧪 models-library🧪 postgres-database🧪 public-api🧪 service-integration🧪🧪 service-library🧪 settings-library🧪 storage🧪 swarm-deploy🧪 tests🧪
36	pytest	7.1.1	7.1.2		20	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🧪 environment-setup🧪 models-library🧪 postgres-database🧪 public-api🧪 service-integration🧪🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage🧪 swarm-deploy🧪 tests🧪 web🧪
37	pytest-docker	0.11.0	0.12.0	minor	6	api-server🧪 catalog🧪 director-v2🧪 postgres-database🧪 service-library🧪 web🧪
38	sqlalchemy	1.4.32, 1.4.35	1.4.36		7	models-library🧪 postgres-database🧪🧪🧪 simcore-sdk🧪🧪 swarm-deploy🧪
39	starlette	0.17.1	0.18.0	minor	2	helpers🧪 service-library🧪
40	thrift	0.15.0	0.16.0	minor	1	service-library🧪
41	tqdm	4.63.1	4.64.0	minor	1	swarm-deploy🧪
42	types-aiofiles	0.8.5	0.8.8		1	dynamic-sidecar🧪
43	types-pyyaml	6.0.5	6.0.7		1	dynamic-sidecar🧪
44	typing-extensions	4.1.1	4.2.0	minor	23	dask-task-models-library🧪🧪🔧 environment-setup🧪 helpers🧪 models-library🧪🧪🔧 postgres-database🧪🔧 public-api🧪 service-integration🧪🧪🔧 service-library🧪🧪🧪🔧 settings-library🧪🧪🔧 swarm-deploy🧪🧪
45	virtualenv	20.14.0	20.14.1		16	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 storage🔧 swarm-deploy🧪 web🔧
46	websocket-client	0.59.0, 1.3.1	1.3.2		3	public-api🧪 service-integration🧪 storage🧪
47	websockets	10.2	10.3	minor	1	web🧪
48	werkzeug	2.0.3	2.1.2	minor	1	tests🧪
49	wrapt	1.14.0	1.14.1		11	api-server🧪 catalog🧪 dask-task-models-library🧪 director-v2🧪 dynamic-sidecar🔧 models-library🧪 postgres-database🧪 service-integration🧪 service-library🧪 settings-library🧪 simcore-sdk🧪
50	zict	2.1.0	2.2.0	minor	1	dask-task-models-library🧪

Next steps

After some weeks without updates, I noticed that some of the libraries are used in different versions of the same dependency ( we denote this "dependency dispersion"). This will be corrected separately per service/library in following PRs.

Related issue/s

• periodic maintenance of test and tooling libraries

How to test

Checklist

• Add links to major changes
• color affected packages
• Openapi changes? make openapi-specs, git commit ... and then make version-*)
• Database migration script? cd packages/postgres-database, make setup-commit, sc-pg review -m "my changes"
• Unit tests for the changes exist
• Runs in the swarm
• Documentation reflects the changes
• New module? Add your github username to .github/CODEOWNERS

[codecov]

Codecov Report

Merging #3031 (456f132) into master (25178f5) will increase coverage by 0.9%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #3031     +/-   ##
========================================
+ Coverage    78.8%   79.7%   +0.9%     
========================================
  Files         698     698             
  Lines       29226   29226             
  Branches     3758    3758             
========================================
+ Hits        23043   23321    +278     
+ Misses       5357    5066    -291     
- Partials      826     839     +13     

Flag	Coverage Δ
integrationtests	66.1% <ø> (+1.4%)	⬆️
unittests	75.5% <ø> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ore_service_director_v2/utils/client_decorators.py	73.3% <0.0%> (-3.4%)	⬇️
...e_service_director_v2/modules/dask_clients_pool.py	92.7% <0.0%> (-1.5%)	⬇️
..._director_v2/modules/dynamic_sidecar/docker_api.py	87.1% <0.0%> (-0.9%)	⬇️
...simcore_service_director_v2/modules/dask_client.py	91.9% <0.0%> (-0.7%)	⬇️
.../director/src/simcore_service_director/producer.py	62.7% <0.0%> (+0.8%)	⬆️
.../simcore_service_catalog/services/access_rights.py	82.5% <0.0%> (+1.2%)	⬆️
...ector_v2/modules/dynamic_sidecar/scheduler/task.py	83.2% <0.0%> (+1.8%)	⬆️
...imcore_service_webserver/garbage_collector_core.py	70.1% <0.0%> (+1.8%)	⬆️
...core-sdk/src/simcore_sdk/node_data/data_manager.py	94.1% <0.0%> (+1.9%)	⬆️
...core-sdk/src/simcore_sdk/node_ports/_items_list.py	95.9% <0.0%> (+4.0%)	⬆️
... and 14 more

[KZzizzle]

as code owner, I approve this request

[sanderegg]

looking good. I'm a little bit worried about the mega update in dask-gateway. but let's see

[GitHK]

:+1 nice

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.6% Duplication