Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] epicli init --full -p azure generates not correct region value #2497

Closed
6 of 10 tasks
plirglo opened this issue Aug 11, 2021 · 6 comments
Closed
6 of 10 tasks

[BUG] epicli init --full -p azure generates not correct region value #2497

plirglo opened this issue Aug 11, 2021 · 6 comments
Assignees
@przemyslavic @sbbroot
Labels
priority/low Task with low priority provider/any python Pull requests that update Python code type/bug type/low-hanging-fruit Good, nice, simple task

Comments

@plirglo
Copy link
Contributor

plirglo commented Aug 11, 2021 •
edited by mkyc
Loading

Describe the bug
Epicli apply --full -p azure generates not correct region value - it generates eu-west-2 which is value for AWS.

How to reproduce
Steps to reproduce the behavior:

  1. execute `epicli init --full -p azure
  2. see config file
  3. execution with default region value will fail

Expected behavior
Region value should be compatible with Azure if we generate file for Azure

Environment

  • Cloud provider: Azure
  • OS: all

epicli version: all

Additional context
Probably default value in file https://github.com/epiphany-platform/epiphany/blob/develop/core/src/epicli/data/common/defaults/epiphany-cluster.yml should be changed.

DoD checklist

  • Changelog updated (if affected version was released)
  • COMPONENTS.md updated / doesn't need to be updated
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
  • Case covered by automated test (if possible)
  • Idempotency tested
  • Documentation updated / doesn't need to be updated
  • All conversations in PR resolved
  • Backport tasks created / doesn't need to be backported
@erzetpe erzetpe added the python Pull requests that update Python code label Sep 6, 2021
@erzetpe erzetpe added the type/low-hanging-fruit Good, nice, simple task label Sep 6, 2021
@sbbroot sbbroot self-assigned this Sep 14, 2021
@sbbroot
Copy link
Contributor

sbbroot commented Sep 20, 2021

Bug is in this function:

def load_all_yaml_objs(file_type, provider, kind):
    script_dir = os.path.dirname(__file__)
    path_to_file = os.path.join(script_dir, DATA_FOLDER_PATH, provider, file_type, kind+'.yml')
    return load_file_from_path(script_dir, path_to_file, file_type, kind)

path_to_file will always have a value of:
"epiphany/core/src/epicli/cli/helpers/../../data/azure/defaults/epiphany-cluster.yml"
such file does not exist.

Therefore in load_file_from_path function else statement will always be executed.

def load_file_from_path(script_dir, path_to_file, file_type, kind):
    if os.path.isfile(path_to_file):
        return load_yamls_file(path_to_file)
    else:
        path_to_file = os.path.join(script_dir, DATA_FOLDER_PATH, 'common', file_type, kind + '.yml')
        return load_yamls_file(path_to_file)

This leads to getting values from the:
"epiphany/core/src/epicli/cli/helpers/../../data/common/defaults/epiphany-cluster.yml"
which is the default file for all providers that contains aws values.

sbbroot added a commit to sbbroot/epiphany that referenced this issue Sep 21, 2021
@sbbroot
Fix epicli apply --full region values (hitachienergy#2497)
9ad6c34 
* Add separate epiphany-cluster.yml for Azure

* Move epiphany-cluster.yml from common to aws
sbbroot added a commit to sbbroot/epiphany that referenced this issue Sep 23, 2021
@sbbroot
Fix epicli apply --full region values (hitachienergy#2497)
828ada1 
* Add separate epiphany-cluster.yml for Azure

* Move epiphany-cluster.yml from common to aws
seriva added a commit that referenced this issue Sep 30, 2021
@sbbroot @seriva
Fix epicli apply --full region values (#2497) (#2599)
1f9fb7f 
* Fix epicli apply --full region values (#2497)

* Add separate epiphany-cluster.yml for Azure

* Move epiphany-cluster.yml from common to aws

* add CHANGELOG

* * add `any` default cluster configuration

* remove second subnets from the default azure configuration

* remove availability_zone from the default azure configuration

* * remove `cloud` from the default `any` configuration

* remove `use_service_principal` from the default aws configuration

* remove `credentials` from the default Azure configuration

* * remove `subscription_name` from the default aws configuration

Co-authored-by: Luuk van Venrooij <[email protected]>
sbbroot added a commit to sbbroot/epiphany that referenced this issue Oct 1, 2021
@sbbroot
Fix upgrade mode (hitachienergy#2497) (hitachienergy#2599)
efb5f0f 
* Fix bug introduced in 1f9fb7f
@przemyslavic przemyslavic mentioned this issue Oct 8, 2021
Closed
@przemyslavic
Copy link
Collaborator

PR #2652 is still in progress so this is definitely not ready for testing.
Additionally during testing another task I found an issue when deploying cluster for any provider.

To reproduce:

  1. Execute epicli init -p any -n anytest
  2. Execute epicli apply -f anytest.yml
epiuser@e72902465e4d:/shared$ epicli apply -f /shared/build/anydev/anydev.yml
Provide password to encrypt vault:
11:28:52 INFO cli.engine.schema.DefaultMerger - Starting step
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: epiphany-cluster name: default
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-repository
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-k8s-master1
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-k8s-node1
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-k8s-node2
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-logging
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-monitoring
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-kafka1
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-kafka2
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-postgresql
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-loadbalancer
11:28:52 INFO cli.engine.schema.DefaultMerger - Merging: infrastructure/machine name: default-rabbitmq
11:28:52 INFO cli.engine.schema.DefaultMerger - Step finished in: 0.05s
11:28:52 INFO cli.engine.schema.SchemaValidator - Starting step
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: epiphany-cluster - anydev
11:28:52 INFO cli.engine.schema.SchemaValidator - Step finished in: 0.05s
11:28:52 INFO cli.engine.providers.any.InfrastructureBuilder - Starting step
11:28:52 INFO cli.engine.providers.any.InfrastructureBuilder - Step finished in: 0.00s
11:28:52 INFO cli.engine.schema.SchemaValidator - Starting step
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: infrastructure/machine
11:28:52 INFO cli.engine.schema.SchemaValidator - Step finished in: 0.07s
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Starting step
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/feature-mapping
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/shared-config
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/kubernetes-master
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/helm
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/applications
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/node-exporter
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/filebeat
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/firewall
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/vault
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/kubernetes-node
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/logging
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/kibana
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/prometheus
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/grafana
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/zookeeper
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/jmx-exporter
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/kafka
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/kafka-exporter
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/postgresql
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/postgres-exporter
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/haproxy
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/haproxy-exporter
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/rabbitmq
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/repository
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Adding: configuration/image-registry
11:28:52 INFO cli.engine.schema.ConfigurationAppender - Step finished in: 0.36s
11:28:52 INFO cli.engine.schema.SchemaValidator - Starting step
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/feature-mapping
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/shared-config
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: epiphany-cluster - anydev
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/kubernetes-master
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/helm
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/applications
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/node-exporter
11:28:52 INFO cli.engine.schema.SchemaValidator - Validating: configuration/filebeat
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/firewall
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/vault
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/kubernetes-node
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/logging
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/kibana
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/prometheus
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/grafana
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/zookeeper
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/jmx-exporter
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/kafka
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/kafka-exporter
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/postgresql
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/postgres-exporter
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/haproxy
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/haproxy-exporter
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/rabbitmq
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/repository
11:28:53 INFO cli.engine.schema.SchemaValidator - Validating: configuration/image-registry
11:28:53 INFO cli.engine.schema.SchemaValidator - Step finished in: 0.76s
11:28:53 INFO cli.engine.providers.any.InfrastructureConfigCollector - Starting step
11:28:53 INFO cli.engine.ansible.AnsibleRunner - Starting step
11:28:53 INFO cli.engine.ansible.AnsibleRunner - Copying Ansible resources
11:29:01 INFO cli.engine.ansible.AnsibleInventoryCreator - Creating Ansible inventory
11:29:11 INFO cli.engine.ansible.AnsibleConfigFileCreator - Creating ansible.cfg
11:29:11 INFO cli.engine.ansible.AnsibleVarsGenerator - Generate Ansible vars
11:29:11 WARNING cli.engine.ansible.AnsibleVarsGenerator - No config document for enabled role: opendistro-for-elasticsearch
11:29:11 WARNING cli.engine.ansible.AnsibleVarsGenerator - No config document for enabled role: elasticsearch-curator
11:29:11 WARNING cli.engine.ansible.AnsibleVarsGenerator - No config document for enabled role: logstash
11:29:12 WARNING cli.engine.ansible.AnsibleVarsGenerator - No config document for enabled role: ignite
11:29:12 ERROR epicli - No such attribute: cloud

Moving back to TODO.

@przemyslavic przemyslavic self-assigned this Oct 8, 2021
@przemyslavic przemyslavic mentioned this issue Oct 8, 2021
Closed
@seriva seriva mentioned this issue Oct 11, 2021
Closed
@seriva
Copy link
Collaborator

seriva commented Oct 11, 2021 •
edited
Loading

Added a fix so we can move forward with testing: #2677

@seriva seriva mentioned this issue Oct 11, 2021
Merged
@przemyslavic
Copy link
Collaborator

The default user for AWS should be Ubuntu.
https://github.com/epiphany-platform/epiphany/blob/develop/schema/aws/defaults/epiphany-cluster.yml#L9

https://github.com/epiphany-platform/epiphany/blob/develop/schema/aws/defaults/configuration/minimal-cluster-config.yml#L9

And which user should be default for any provider?

@seriva
Copy link
Collaborator

seriva commented Oct 14, 2021

Updated pull for #2677 with ubuntu user for AWS.

seriva added a commit that referenced this issue Oct 14, 2021
@seriva
Removed legacy code and some minor refactoring (#2677)
bfd1414 
- Removed legacy code for Epiphany < 0.3.
- Refactored build_saver.py and renamed to build_io.py as we do not only save.
- Updated tests to reflect all changes.
- Also added missing changelogs for #2643, #2666
- Added fixes for #2497 which where outstanding in #2652 and reported in: #2497 (comment)
@przemyslavic
Copy link
Collaborator

✔️ Fixed. Now a different default configuration is used for each provider.

@mkyc mkyc changed the title [BUG] Epicli apply --full -p azure generates not correct region value [BUG] epicli init --full -p azure generates not correct region value Oct 19, 2021
@mkyc mkyc closed this as completed Oct 19, 2021
@przemyslavic przemyslavic mentioned this issue Nov 10, 2021
Closed
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@przemyslavic przemyslavic

@sbbroot sbbroot

Labels
priority/low Task with low priority provider/any python Pull requests that update Python code type/bug type/low-hanging-fruit Good, nice, simple task
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@erzetpe @seriva @mkyc @przemyslavic @plirglo @sbbroot