Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify ports for database, BucketFS, and SSH #354

Merged
merged 57 commits into from
Jul 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
b9ca0fe
initial implementation
ckunki Jun 22, 2023
975f8ac
Annotated properties of class Ports with references to usages
ckunki Jun 22, 2023
c1b5d18
Updated changes_2.0.0.md
ckunki Jun 22, 2023
482219a
Replaced ports in EXAConf by template variables
ckunki Jun 22, 2023
a52f516
fixed errors in pytest_plugin_itde_test.py
ckunki Jun 23, 2023
4b166de
Removed Exposed ports from files EXAConf
ckunki Jun 23, 2023
a4bbce0
reviewed and cleaned up references to class Port
ckunki Jun 23, 2023
344f515
added ssh_port to all EXAConf templates and enhanced template substit…
ckunki Jun 23, 2023
37a2f59
Fixed review findings
ckunki Jun 26, 2023
fcd251a
Added forwarded ports to DatabaseInfo
ckunki Jun 26, 2023
adbfd09
Removed port from ShInfo
ckunki Jun 26, 2023
82861fb
fixed Test
ckunki Jun 26, 2023
7f8ad18
fixed test/integration/pytest_itde_test.py
ckunki Jun 27, 2023
8fb0b08
Fixed review findings
ckunki Jun 27, 2023
282b979
Fixed tests
ckunki Jun 27, 2023
4b4679f
made moved forwarded port an attribute of class SpawnTestDockerDatabase
ckunki Jun 27, 2023
ec773f1
[run all tests]
ckunki Jun 27, 2023
5415844
Use EXASOL_VERSION in pytest_itde_test.py [run all tests]
tkilias Jun 28, 2023
c6d4243
set log level for debugging purposes
ckunki Jun 29, 2023
a121b61
[run all tests]
ckunki Jun 29, 2023
35103c8
added workflow for manual experiments
ckunki Jun 29, 2023
09b11ba
added on: workflow_dispatch:
ckunki Jun 29, 2023
d0699d4
fixed workflow
ckunki Jun 29, 2023
f438803
fixed id of workflow
ckunki Jun 29, 2023
dc3c302
simplified workflow
ckunki Jun 29, 2023
93c22a3
added --log-debug=DEBUG
ckunki Jun 29, 2023
5e75517
reverted test workflow file
ckunki Jun 29, 2023
42727c8
x
ckunki Jun 29, 2023
8a6a956
x
ckunki Jun 29, 2023
e274d75
y
ckunki Jun 29, 2023
835c7b6
different approach to skip pytest_itde_tests for non-default version …
ckunki Jun 29, 2023
28da851
x
ckunki Jun 29, 2023
e8cafd4
refactored test skips
ckunki Jun 29, 2023
9bfd3af
changed noxfile
ckunki Jun 29, 2023
ab3a5df
fixed noxfile.py
ckunki Jun 29, 2023
ab1fcf8
CLeaned up tests
ckunki Jun 29, 2023
d7e6c9d
removed experimental test
ckunki Jun 29, 2023
cf67beb
[run all tests]
ckunki Jun 30, 2023
820ce1b
added env to noxfile
ckunki Jun 30, 2023
5d0c96a
experimental
ckunki Jun 30, 2023
6c64352
added unit tests and old tests based on python lib "unittest"
ckunki Jun 30, 2023
84ffc38
finalized noxfile
ckunki Jun 30, 2023
9fb77b0
[run all tests]
ckunki Jun 30, 2023
eac0c59
Update test/integration/pytest_itde_test.py
ckunki Jul 3, 2023
07575e5
removed commented lines in 8.18.1/EXAConf
ckunki Jul 3, 2023
2c33cee
[run all tests]
ckunki Jul 3, 2023
20f4898
re-activated new integration tests
ckunki Jul 3, 2023
66442a7
added build script for experimental tests
ckunki Jul 3, 2023
30808e9
fixed build script
ckunki Jul 3, 2023
de76d4d
added chroot for 8.17.0/EXAConf as well
ckunki Jul 3, 2023
180a9c6
changed COS version in path to sbin/nsexec_chroot to match ConfVersion
ckunki Jul 3, 2023
1082e82
copied EXAConf template to exasol_integration_test_docker_environment…
ckunki Jul 3, 2023
e863e24
[run all tests]
ckunki Jul 3, 2023
27f0c34
removed experimental workflow
ckunki Jul 3, 2023
1424cc1
[run all tests]
ckunki Jul 3, 2023
f66f715
Fixed build matrix
ckunki Jul 4, 2023
b024bdc
[run all tests]
ckunki Jul 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/changes/changes_2.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ If you need further versions, please open an issue.

* #329: Added CLI option `--ssh-port-forward` to forward SSH port
* #343: Added SshInfo to DatabaseInfo containing user, port and path to SSH key file
* #308: Unified ports for database, BucketFS, and SSH
4 changes: 2 additions & 2 deletions doc/user_guide/user_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,8 @@ The following config files are available:

# Database IP in environment docker network
export ITDE_DATABASE_HOST=172.21.0.2
export ITDE_DATABASE_DB_PORT=8888
export ITDE_DATABASE_BUCKETFS_PORT=6583
export ITDE_DATABASE_DB_PORT=8563
export ITDE_DATABASE_BUCKETFS_PORT=2580
export ITDE_DATABASE_SSH_PORT=22
export ITDE_DATABASE_CONTAINER_NAME=db_container_test
export ITDE_DATABASE_CONTAINER_NETWORK_ALIASES="exasol_test_database db_container_test"
Expand Down
8 changes: 3 additions & 5 deletions docker_db_config_template/7.0.0/EXAConf
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Platform = Docker
LicenseFile = /exa/etc/license.xml
CoredPort = 10001
SSHPort = 22
SSHPort = {{ ssh_port }}
XMLRPCPort = 443
AuthenticationToken = WG9JQkZYb0R5T1pIcWZPWTpyUHFsR3V6QUFkV1FQbHZmdm91cXJDYmNQTFl4eE1vRlZ6VnhxamdxYnpxUGRkZEhtR2NJaVVGbEpsb2NnbnRZ
# List of networks for this cluster: 'private' is mandatory, 'public' is optional.
Expand Down Expand Up @@ -74,8 +74,6 @@
Name = n11
UUID = A5F8F92113A34BA4B0A48D5397423BBA5CF95161
DockerVolume = n11
# Ports to be exposed (container : host)
ExposedPorts = 8888:8899, 6583:6594
[[Disk : disk1]]
Component = exastorage
Devices = dev.1
Expand Down Expand Up @@ -121,7 +119,7 @@
Version = {{ db_version }}
# Memory size over all nodes (e. g. '1 TiB').
MemSize = {{ mem_size }}
Port = 8888
Port = {{ db_port }}
# User and group IDs that own this database (e. g. '1000:1005').
Owner = 500 : 500
# Comma-separated list of node IDs for this DB (put reserve nodes at the end, if any).
Expand Down Expand Up @@ -155,7 +153,7 @@
[BucketFS : bfsdefault]
Owner = 500:500
# HTTP port number (0 = disabled)
HttpPort = 6583
HttpPort = {{ bucketfs_port }}
# HTTPS port number (0 = disabled)
HttpsPort = 0
SyncKey = OGhvcmxrTnluTlFrZTV3RVBTWG5Idlc4bkVMQUNxcWs=
Expand Down
8 changes: 3 additions & 5 deletions docker_db_config_template/7.1.0/EXAConf
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Platform = Docker
LicenseFile = /exa/etc/license.xml
CoredPort = 10001
SSHPort = 22
SSHPort = {{ ssh_port }}
XMLRPCPort = 443
WebUIBackendPort = 4444
WebUIDomain = exacluster.local
Expand Down Expand Up @@ -88,8 +88,6 @@
Affinity = 11
UUID = 02F4E2AF9C294CABA0A119A89AC60B44EC95D19C
DockerVolume = n11
# Ports to be exposed (container : host)
ExposedPorts = 8888:8899, 6583:6594
[[Disk : disk1]]
Component = exastorage
Devices = dev.1
Expand Down Expand Up @@ -135,7 +133,7 @@
Version = {{ db_version }}
# Memory size over all nodes (e. g. '1 TiB').
MemSize = {{ mem_size }}
Port = 8888
Port = {{ db_port }}
# User and group IDs that own this database (e. g. '1000:1005').
Owner = 500 : 500
# Comma-separated list of node IDs for this DB (put reserve nodes at the end, if any).
Expand Down Expand Up @@ -172,7 +170,7 @@
[BucketFS : bfsdefault]
Owner = 500 : 500
# HTTP port number (0 = disabled)
HttpPort = 6583
HttpPort = {{ bucketfs_port }}
# HTTPS port number (0 = disabled)
HttpsPort = 0
SyncKey = c3RxWjRxWUFpZUFQblEyc0p2djZUM0VZamVZa1M0bUs=
Expand Down
12 changes: 6 additions & 6 deletions docker_db_config_template/8.17.0/EXAConf
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Platform = Docker
LicenseFile = /exa/etc/license.exasol_license
CoredPort = 325
SSHPort = 22
SSHPort = {{ ssh_port }}
IssueTracker = Jira
XMLRPCPort = 443
WebUIBackendPort = 4444
Expand Down Expand Up @@ -104,8 +104,6 @@
Affinity = 11
UUID = F0EA64D47F374A00B0530772981C559EE1C59086
DockerVolume = n11
# Ports to be exposed (container : host)
ExposedPorts = 8888:8899, 6583:6594
[[Disk : disk1]]
Component = exastorage
Devices = dev.1
Expand Down Expand Up @@ -159,7 +157,7 @@
Version = {{ db_version }}
# Memory size over all nodes (e. g. '1 TiB').
MemSize = {{ mem_size }}
Port = 8888
Port = {{ db_port }}
# User and group IDs that own this database (e. g. '1000:1005').
Owner = 500 : 500
# Comma-separated list of node IDs for this DB (put reserve nodes at the end, if any).
Expand All @@ -172,7 +170,9 @@
BuiltinScriptLanguageName = slc-6.0.0-c4-5-standard-EXASOL-8.0.0/ScriptLanguages-standard-EXASOL-8.0.0-slc-v6.0.0-PB5EHDLN
AutoStart = True
{% if additional_db_parameters %}
Params = {{ additional_db_parameters }}
Params = -sandboxCHROOT=/opt/exasol/cos-8.28.0/sbin/nsexec_chroot {{ additional_db_parameters }}
{% else %}
Params = -sandboxCHROOT=/opt/exasol/cos-8.28.0/sbin/nsexec_chroot
{% endif %}

# JDBC driver configuration
Expand All @@ -198,7 +198,7 @@
[BucketFS : bfsdefault]
Owner = 500 : 500
# HTTP port number (0 = disabled)
HttpPort = 6583
HttpPort = {{ bucketfs_port }}
# HTTPS port number (0 = disabled)
HttpsPort = 0
SyncKey = bDRYa3pCNXRwaW9OR3k2NjJ2TGpyQ3YweXZEOVUyanQ=
Expand Down
8 changes: 3 additions & 5 deletions docker_db_config_template/8.18.1/EXAConf
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Platform = Docker
LicenseFile = /exa/etc/license.exasol_license
CoredPort = 325
SSHPort = 22
SSHPort = {{ ssh_port }}
IssueTracker = Jira
XMLRPCPort = 443
WebUIBackendPort = 4444
Expand Down Expand Up @@ -104,8 +104,6 @@
Affinity = 11
UUID = F0EA64D47F374A00B0530772981C559EE1C59086
DockerVolume = n11
# Ports to be exposed (container : host)
ExposedPorts = 8888:8899, 6583:6594
[[Disk : disk1]]
Component = exastorage
Devices = dev.1
Expand Down Expand Up @@ -159,7 +157,7 @@
Version = {{ db_version }}
# Memory size over all nodes (e. g. '1 TiB').
MemSize = {{ mem_size }}
Port = 8888
Port = {{ db_port }}
# User and group IDs that own this database (e. g. '1000:1005').
Owner = 500 : 500
# Comma-separated list of node IDs for this DB (put reserve nodes at the end, if any).
Expand Down Expand Up @@ -200,7 +198,7 @@
[BucketFS : bfsdefault]
Owner = 500 : 500
# HTTP port number (0 = disabled)
HttpPort = 6583
HttpPort = {{ bucketfs_port }}
# HTTPS port number (0 = disabled)
HttpsPort = 0
SyncKey = bDRYa3pCNXRwaW9OR3k2NjJ2TGpyQ3YweXZEOVUyanQ=
Expand Down
2 changes: 1 addition & 1 deletion exasol_integration_test_docker_environment/cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def cli():
Spawn an ITDE test environment:

$ itde spawn-test-environment --environment-name test \\
--database-port-forward 8888 --bucketfs-port-forward 6666 \\
--database-port-forward 8563 --bucketfs-port-forward 2580 \\
--docker-db-image-version 7.1.9 --db-mem-size 4GB
"""
pass

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading