platforms/aix.yml makes assumptions it should not be making, e.g., that everything is in /usr/bin #1581

aixtools · 2020-09-30T12:59:25Z

Just ran the playbook on an as generic (vanilla) system as possible.

Delta from fresh install:

/var/ssl/cacerts.pem
python3.6 installed
git installed (to clone the infrastructure site)
wget and curl installed (to fetch the above - and below)
ansible-2.10 installed

Running the aix.yml playbook gives:

(py36) root@p9-aix1-ojdk06:[/root/openjdk-infrastructure/ansible]ansible-playbook -i "localhost," -C --skip-tags adoptopenjdk,jenkins playbooks/aix.yml
/opt/lib/python3.6/site-packages/cryptography/hazmat/bindings/openssl/binding.py:177: CryptographyDeprecationWarning: OpenSSL version 1.0.2 is no longer supported by the OpenSSL project, please upgrade. The next version of cryptography will drop support for it.
  utils.CryptographyDeprecationWarning,
[DEPRECATION WARNING]: DEFAULT_SQUASH_ACTIONS option, Loop squashing is deprecated and this configuration will no longer be used , use a list directly with the module argument instead. This
feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [all] **************************************************************************************************************************************************************************************

TASK [Load AdoptOpenJDKs variable file] *********************************************************************************************************************************************************
ok: [localhost]

TASK [Check if bash is installed] ***************************************************************************************************************************************************************
[WARNING]: No python interpreters found for host localhost (tried ['/usr/bin/python', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python',
'/usr/bin/python3', 'python'])
fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "module_stderr": "Shared connection to localhost closed.\r\n", "module_stdout": "/bin/sh: /usr/bin/python:  not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}
...ignoring

TASK [install bash] *****************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": ["Could not detect which major revision of yum is in use, which is required to determine module backend.", "You should manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})"]}

PLAY RECAP **************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1

Issues:

as I do not know ansible (yet) - what is causing the depreciation warning re Loop Squashing. If this is part of aix.yml, I'll need help understanding where/what to change.
OpenSSL - have to figure out where that is coming from - perhaps the Python build and the OpenSSL library it calls. In any case, not, imho, and aixyml issue
No python interpreters - needs improved checking - rather than looking at file-system should (also) look at the installed software database(s). Assuming it is installed in /usr/bin is wrong. It may have been there in the past - but it will not be there in the future.
yum: this is a separate discussion - why even bother. The OSS packages needed by nodejs are not random - they can all be downloaded individually and installed by either rpm and/or installp. afaik yum4 (dnf) is not available for AIX.
bash check: that bash is wanted is not an issue - should not depend on yum to install it - imho.

The text was updated successfully, but these errors were encountered:

aixtools · 2020-09-30T13:25:46Z

OK. I have done some massageing of the test system - such as add a symbolic link python3.6 in /usr/bin. (Where it points is not relevant atm - only that this block fails:
a) both bash and python are installed
b) I tried modifying the PATH
c) after adding symbolic link /usr/bin/python3.6 ==> /some/where/else/python3.6 -- I get the following message(s)


TASK [Load AdoptOpenJDKs variable file] *********************************************************************************************************************************************************
ok: [localhost]

TASK [Check if bash is installed] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [install bash] *****************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'does_bash_exist.rc != 0' failed. The error was: error while evaluating conditional (does_bash_exist.rc != 0): 'dict object' has no attribute 'rc'\n\nThe error appears to be in '/root/openjdk-infrastructure/ansible/playbooks/aix.yml': line 28, column 11, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n          ignore_errors: yes\n        - name: install bash\n          ^ here\n"}

PLAY RECAP **************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

aixtools · 2020-10-01T06:34:18Z

OK. So, no comments yet.

My humble opinion:

The aix.yml should be a sort of process management flow.

Step 1: verify that licensed requirements are installed - including - especially! things such as XLC and other items that come from IBM AIX BOS (aka, the DVD's).
edit: check for latest/minimal OpenSSL also belongs here. Not installing it - verifying it.

Step 2: if/when needed - verify that known issues have been addressed.

IMHO: key in these first two steps is that no changes are made - other than perhaps copying a script for execution as is currently done for the "filesystems" config script. ** speaking of ** the part of the file systems script that checks on the hdisk minimum size would be moved to step 2 (above). The disk, better volume group size is, or it is not sufficient.

The purpose: these are all elements that are, or should be, the responsibility, of an "AIX admin" aka system owner. Perhaps 'node.js and/or AdoptOpenJDK' have been 'owners', but to be truely supporting as an OSS project open to others you should protect yourself and not be installing things requiring a license AND - initially verifying a system is "ready to go" for the project (nodejs/AdoptOpenJDK).

In following steps - the strategy I am considering now is to write scripts that get executed as a 'task' - again, taking the filesystems approach as a template:


      ################################
      # AIX filesystem configuration #
      ################################

        - name: Download AIX filesystem configuration script
          get_url:
            url: https://raw.githubusercontent.com/AdoptOpenJDK/openjdk-infrastructure/master/ansible/playbooks/scripts/AIX_filesystem_config.sh
            dest: /tmp/AIX_filesystem_config.sh
            mode: 0775
            validate_certs: no
          tags: filesystem

        - name: Execute AIX filesystem configuration script
          command: /tmp/AIX_filesystem_config.sh
          tags: filesystem

        - name: Remove AIX filesystem configuration script
          file:
            state: absent
            path: /tmp/AIX_filesystem_config.sh
          tags: filesystem

I suspect, as I go through more of aix.yml, that this is already the approach.

_FYI: The need for a ramdisk has been resolved (by me). The issue is when lots (10's and 100's of thousands of temporary files are created) the lag introduced by the journal logging creates a lag that cannot be parallelized. Taking OSU as an example - as it uses vscsi only - the top speed for jfs(2) logging seems to be between 8 and 10Mbs.

The 'ramdisk' is not faster than a regular filesystem - once jfs(2) logging is switched off. So the actual solution is to ensure that the project work directory is not being "JFS2" logged, e.g., /home is mounted with attribute log=NULL)._

Comments welcome!! actually, needed!!

aixtools · 2020-10-01T07:00:30Z

Question: are these being installed to support 'yum', or are they actually needed by node.js/AdoptOpenJDK.

If the answer is yes - yum needs these (as it does have some - imho - extraordinary pre-requisites (and yum.sh also has pre-requisites) -- I have a second question:

which of these OSS packages are actually needed - explicitly - by node.js/AdoptOpenJDK?

        - name: Install yum package support
          yum: name={{ item }} state=present update_cache=yes
          with_items:
            - autoconf
            - bc
            - bison
            - coreutils
            - cpio
            - cups-devel
            - cups-libs
            - expect
            - flex
            - freetype2-devel
            - fontconfig-devel
            - gawk
            - git
            - grep
            - libXrender-devel
            - libffi-devel
            - make
            - m4
            - pcre
            - pkg-config
            - popt
            - sed
            - sudo
            - tar
            - tcl
            - tk
            - unzip
            - wget
            - xz-libs
            - zip
            - zsh
          tags: yum

aixtools · 2020-10-01T07:14:48Z

re: syslog

      #########################
      # Configure system logs #
      #########################
        - name: Ensure /etc/syslog.conf entries are present
          blockinfile:
            backup: yes
            dest: /etc/syslog.conf
            block: |
              *.debug;*.emerg;*.alert;*.crit;*.warning /var/log/messages rotate size 1000k files 4
              syslog.debug /var/log/syslog rotate size 1000k files 4
              *.err;kern.debug;daemon.notice;mail.crit /var/adm/messages rotate size 1000k files 4
          tags: syslog

        - name: Ensure log files exist
          file:
            path: "{{ item }}"
            state: touch
            mode: 0644
          with_items:
            - /var/log/syslog
            - /var/adm/messages
            - /var/log/messages
          changed_when: False
          tags: syslog

At customers I advised I never saw syslog entries in /var/adm. Nothing 'wrong', just unusual at AIX customers I have advised.
rather than a file /var/log/syslog - that is often a directory with files for the different streams, e.g., /var/log/syslog/mail.log and /var/log/syslog/auth.log. Further, /var/log/syslog is often a separate filesystem to prevent it blocking /var (in particular, /var/tmp).

Questions: are these names chosen because that is what GNU-Linix uses? Are there project scripts that scan these filenames?

aixtools · 2020-10-01T07:49:07Z

So, now finished reading through aix.yml, and imho, it is in need of 'major' restructuring.

Will start reading #1391 , #710 , #1104 and perhaps others - to determine if they are all related to this issue - and determine an approach to getting them all resolved (aka closed). edit: some are already closed - my bad!

aahlenst · 2020-10-01T09:40:59Z

You bring up many issues. While that is great, it's a lot of work to unpack for me. Maybe there's a way to make things more focused.

Step 1: verify that licensed requirements are installed - including - especially! things such as XLC and other items that come from IBM AIX BOS (aka, the DVD's). edit: check for latest/minimal OpenSSL also belongs here. Not installing it - verifying it.

Ansible is a system management/configuration tool. As such, it's within Ansible's scope to install or re-configure things. The question is where you draw the line. For the sake of reproducibility, I try to move as many things into the domain of Ansible as possible. On Windows, autounattend.xml installs a minimal system with a known user account and activates WinRM. The rest is done by Ansible. On Debian/Ubuntu, d-i/subiquity install a minimal system including OpenSSH before Ansible takes over. In the same vein, it would be Ansible's responsibility to install XLC if that's possible with a download. As soon as you have to juggle DVDs, that's something that the admin has to do and should ideally happen before Ansible runs.

which of these OSS packages are actually needed - explicitly - by node.js/AdoptOpenJDK?

Requirements for building OpenJDK are listed on https://openjdk.java.net/groups/build/doc/building.html. OpenJ9 needs some additional things. The list above looks accurate.

Why do you mention node.js?

In following steps - the strategy I am considering now is to write scripts that get executed as a 'task' - again, taking the filesystems approach as a template:

Using scripts is the last resort when using Ansible. If possible, everything should happen in Ansible. If you use scripts (ksh, ...) you forego all advantages of Ansible.

aixtools · 2020-10-01T10:06:11Z

You bring up many issues. While that is great, it's a lot of work to unpack for me. Maybe there's a way to make things more focused.
Not quite sure what you mean by unpack - See #1583 as an example of what I am willing to work on, and how.

Step 1: verify that licensed requirements are installed - including - especially! things such as XLC and other items that come from IBM AIX BOS (aka, the DVD's). edit: check for latest/minimal OpenSSL also belongs here. Not installing it - verifying it.

Ansible is a system management/configuration tool. As such, it's within Ansible's scope to install or re-configure things. The question is where you draw the line. For the sake of reproducibility, I try to move as many things into the domain of Ansible as possible. On Windows, autounattend.xml installs a minimal system with a known user account and activates WinRM. The rest is done by Ansible. On Debian/Ubuntu, d-i/subiquity install a minimal system including OpenSSH before Ansible takes over. In the same vein, it would be Ansible's responsibility to install XLC if that's possible with a download. As soon as you have to juggle DVDs, that's something that the admin has to do and should ideally happen before Ansible runs.

which of these OSS packages are actually needed - explicitly - by node.js/AdoptOpenJDK?

Requirements for building OpenJDK are listed on https://openjdk.java.net/groups/build/doc/building.html. OpenJ9 needs some additional things. The list above looks accurate.
OK. I'll look there - part of the learning curve - which is steep.

Why do you mention node.js?
Because Ash Cripps is involved in both node.js and openjdk - my mistake for combining them. And, because I am the volunteer admin for AIX at OSU - where I am currently trying to prep new systems for both node.js and OJDK.

In following steps - the strategy I am considering now is to write scripts that get executed as a 'task' - again, taking the filesystems approach as a template:

Using scripts is the last resort when using Ansible. If possible, everything should happen in Ansible. If you use scripts (ksh, ...) you forego all advantages of Ansible.

Well, I do not know ansible well enough to know what I forego. I did package ansible-2.10 using py36 and wheels - so ansible can be something local on the system. From the looks of things - the assumption has been than ansible is not available on AIX - and maybe that is why there is a dependency, if not preference, for using scripts on AIX.

Note: learning curve applies. I am likely to make many assumptions that are incorrect - so ASSuME - aka ASS + u + (especially) ME. :)

aahlenst · 2020-10-01T10:14:23Z

It's odd to install Ansible on the target system. You only do that if you have no other choice (for example provisioning Linux machines on Windows because Ansible does not run on Windows). Ansible is usually present on the sysadmin's machine (or some jump host). You run Ansible locally on that system. Ansible then connects over SSH (or WinRM) to the target system and performs its work there.

To get an idea what Ansible can do for you:
https://www.jeffgeerling.com/blog/2020/ansible-101-jeff-geerling-youtube-streaming-series
https://www.ansiblefordevops.com/

I found the book very approachable when I started learning Ansible.

aixtools · 2020-10-01T11:08:24Z

OK. But I need ansible (locally) to test. And, again, for testing - I can install it on the NIM system I have for installation and test how that behaves with targets. Might even make provisioning new lpars (and wpars) easier - separate from any projects.

Probably in your references (thx!) above - but I will need to work out how the Ansible communicates (i.e., executes commands) on the target. Something such as how the hosts line works (all versus a specified system).

So, along those lines #1583 becomes, imho, more relevant as far as managing the Ansible process goes and issues such as this one are more content related.

aixtools · 2020-10-01T16:15:33Z

FYI: I bought the book @aahlenst recommended, and have installed ansible on another system - so I can reinstall the test server and use ansible for final setup (outside of scope here, will be a side project).

Comment: although added to October 2020 milestone - I cannot guarantee I have it ready "on time". But I'll do my best.

sxa · 2020-10-05T16:02:28Z

The "October 2020" mileston is to indicate that it is being worked on during that timeframe and not a guarantee that we'll hold you to that it will be completed in there :-)

sxa · 2020-10-05T16:05:03Z

Ref installing ansible on a remote system from the target, if you think the section at https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/FAQ.md#running-the-ansible-scripts-remotely-on-another-machine can be improved feel free to submit a PR :-)

aixtools · 2020-10-06T08:14:22Z

Let me get a bit more personal experience - and then I'll write.

aixtools · 2020-10-30T06:46:33Z

You bring up many issues. While that is great, it's a lot of work to unpack for me. Maybe there's a way to make things more focused.

I know more, eg, Ansible is agentless, where/why warning messages come from.

Step 1: verify that licensed requirements are installed - including - especially! things such as XLC and other items that come from IBM AIX BOS (aka, the DVD's). edit: check for latest/minimal OpenSSL also belongs here. Not installing it - verifying it.

Ansible is a system management/configuration tool. As such, it's within Ansible's scope to install or re-configure things. The question is where you draw the line. For the sake of reproducibility, I try to move as many things into the domain of Ansible as possible. On Windows, autounattend.xml installs a minimal system with a known user account and activates WinRM. The rest is done by Ansible. On Debian/Ubuntu, d-i/subiquity install a minimal system including OpenSSH before Ansible takes over. In the same vein, it would be Ansible's responsibility to install XLC if that's possible with a download. As soon as you have to juggle DVDs, that's something that the admin has to do and should ideally happen before Ansible runs.

If the environment is prepared, so that file copy is prepared, then that can work. My comment then, is that the playbook assumes this, and that the location is xxx.

This can be enhanced using one of group|host_vars, depending on how licensed resources are made available, or pre-installed.

which of these OSS packages are actually needed - explicitly - by node.js/AdoptOpenJDK?

Requirements for building OpenJDK are listed on https://openjdk.java.net/groups/build/doc/building.html. OpenJ9 needs some additional things. The list above looks accurate.

Why do you mention node.js?

In following steps - the strategy I am considering now is to write scripts that get executed as a 'task' - again, taking the filesystems approach as a template:

Using scripts is the last resort when using Ansible. If possible, everything should happen in Ansible. If you use scripts (ksh, ...) you forego all advantages of Ansible.

I am waiting for review and merge of #1633 before i start work on rewriting the current script as ansible tasks.

aixtools · 2021-04-08T07:57:13Z

As this was originally about the flat playbook aix.yml - I am closing this as no longer relevant. We are working from a new playbook structure and issues need to be addressed in those files.

aixtools mentioned this issue Oct 1, 2020

AIX playbook should not put /opt/freeware/bin at start of /etc/environment #1544

Open

This was referenced Oct 1, 2020

AIX playbook requires an update #1391

Closed

AIX_filesystem_config.sh #1547

Closed

karianna added os:aix bug labels Oct 1, 2020

karianna assigned aixtools Oct 1, 2020

karianna added this to the October 2020 milestone Oct 1, 2020

Haroon-Khel modified the milestones: October 2020, November 2020 Nov 9, 2020

Haroon-Khel modified the milestones: November 2020, December 2020 Dec 10, 2020

Haroon-Khel modified the milestones: December 2020, January 2021 Jan 4, 2021

Haroon-Khel removed this from the January 2021 milestone Feb 1, 2021

Haroon-Khel added this to the February 2021 milestone Feb 1, 2021

Haroon-Khel modified the milestones: February 2021, March 2021 Mar 2, 2021

Haroon-Khel modified the milestones: March 2021, April 2021 Apr 6, 2021

aixtools closed this as completed Apr 8, 2021

karianna added the wontfix label Apr 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

platforms/aix.yml makes assumptions it should not be making, e.g., that everything is in /usr/bin #1581

platforms/aix.yml makes assumptions it should not be making, e.g., that everything is in /usr/bin #1581

aixtools commented Sep 30, 2020

aixtools commented Sep 30, 2020

aixtools commented Oct 1, 2020 •

edited

Loading

aixtools commented Oct 1, 2020

aixtools commented Oct 1, 2020

aixtools commented Oct 1, 2020 •

edited

Loading

aahlenst commented Oct 1, 2020 •

edited

Loading

aixtools commented Oct 1, 2020

aahlenst commented Oct 1, 2020

aixtools commented Oct 1, 2020

aixtools commented Oct 1, 2020

sxa commented Oct 5, 2020

sxa commented Oct 5, 2020

aixtools commented Oct 6, 2020

aixtools commented Oct 30, 2020

aixtools commented Apr 8, 2021

platforms/aix.yml makes assumptions it should not be making, e.g., that everything is in /usr/bin #1581

platforms/aix.yml makes assumptions it should not be making, e.g., that everything is in /usr/bin #1581

Comments

aixtools commented Sep 30, 2020

aixtools commented Sep 30, 2020

aixtools commented Oct 1, 2020 • edited Loading

aixtools commented Oct 1, 2020

aixtools commented Oct 1, 2020

aixtools commented Oct 1, 2020 • edited Loading

aahlenst commented Oct 1, 2020 • edited Loading

aixtools commented Oct 1, 2020

aahlenst commented Oct 1, 2020

aixtools commented Oct 1, 2020

aixtools commented Oct 1, 2020

sxa commented Oct 5, 2020

sxa commented Oct 5, 2020

aixtools commented Oct 6, 2020

aixtools commented Oct 30, 2020

aixtools commented Apr 8, 2021

aixtools commented Oct 1, 2020 •

edited

Loading

aixtools commented Oct 1, 2020 •

edited

Loading

aahlenst commented Oct 1, 2020 •

edited

Loading