AIX_filesystem_config.sh

[aixtools]

This does not look like it was written by an an AIX admin.

It checks for bash being installed, but it is #! (has bang) started by /bin/bash (which iirc is not where bash should be - maybe packages have changed).

As far as I can see, everything in the script would work fine using ksh. (And then checking for bash would be okay, but what does that have to do with filesystems?).

The is a 'constant' being used to increase the default number of logical partition segments from 512 to 2048? Why? This doesn't hurt anything - but changes should not be made on a whim. What is the goal? To be able to assign 2048 * 1G for every filesystem. While the defaults - one a single disk system do not provide more than 1016 partitions on the disk.

To an AIX admin, imho, it looks sloppy and pointless.

Further, without going into a lot of detail - the idea of trying to assign all the space available in the disk to filesystems goes contrary to all IBM AIX education principles. AIX storage administration is 'on demand' - grow as you need it. This looks more like how UNIX and Xenix was setup 30+ years ago when you had to use a program like divvy to partition the disk prior to creating file systems. Everyone uses logical volume managers so that this kind of pre-sizing is outdated - at least on AIX.

So, are you willing to consider an alternative approach to setting up AIX?

[aixtools]

For reference: here is a new system. Not knowing better I assigned a 40G disk.

All is used, and /home is very small (nothing in it atm, but that will fail later I expect). Yet, /, /usr, /var, and /opt have about 25G free (over 50% of the storage provided). Also note - /tmp is still only 256 MB!

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           5.50      4.56   18%     4482     1% /
/dev/hd2          10.75      6.63   39%    54238     4% /usr
/dev/hd9var        5.62      5.04   11%     4781     1% /var
/dev/hd3           0.25      0.25    2%       91     1% /tmp
/dev/hd1           2.12      2.12    1%       18     1% /home
/dev/hd11admin      0.12      0.12    1%        9     1% /admin
/proc                 -         -    -        -      - /proc
/dev/hd10opt      10.38      8.96   14%    23785     2% /opt
/dev/livedump      0.25      0.25    1%        4     1% /var/adm/ras/livedump
/dev/lvBESC        2.00      1.93    4%      782     1% /var/opt/BESClient
/dev/fslv00        0.12      0.06   51%        9     1% /audit
/dev/ramdisk0      4.00      3.87    4%       17     1% /ramdisk
root@p9-aix1-ojdk05:[/root]lsfs /home
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/hd1        --         /home                  jfs2  4456448 rw,log=NULL yes  no
root@p9-aix1-ojdk05:[/root]lspv hdisk0
PHYSICAL VOLUME:    hdisk0                   VOLUME GROUP:     rootvg
PV IDENTIFIER:      00fac25ffb892afd VG IDENTIFIER     00fac25f00004b0000000174204f579b
PV STATE:           active
STALE PARTITIONS:   0                        ALLOCATABLE:      yes
PP SIZE:            128 megabyte(s)          LOGICAL VOLUMES:  14
TOTAL PPs:          319 (40832 megabytes)    VG DESCRIPTORS:   2
FREE PPs:           0 (0 megabytes)          HOT SPARE:        no
USED PPs:           319 (40832 megabytes)    MAX REQUEST:      256 kilobytes
FREE DISTRIBUTION:  00..00..00..00..00
USED DISTRIBUTION:  64..64..63..64..64

From experience, 1G is usually enough for / - yet this new system is already over that, by a bit. /usr shows more than a regular AIX system (rarely above 4G for 'normal systems, and regardless of size, usually between 95 and 99% full - as it grows on demand when using installp). It is not supposed to include 'data' files that grow over time.

/var at 5G is fine - /var/tmp may need more - and syslog and errpt logs go here as well.

/tmp - 256MB - bit small

/home - well, it got the rest didn't it. Noone using it (yet), and it probably is not enough when they do (git clone ...).

Looking into what is on / atm - foregoing that my proposal is to have

/ 2G
/usr 4G
/var 5G
/tmp 5G
/opt 4G

/home 20G (assuming that fits - not done the math).

The other change I'll make - as this is all ansible driven - is to reinstall with 60G of storage - and use the numbers above - MAYBE increase /home to 30G.

Comments!?

[aixtools]

p.s. If you give me the permission and authority to kickoff the ansible process on (this) AIX host - I can rinse and repeat - to get the process perfected.

[aahlenst]

I do not know anything about AIX, so only general remarks: If you think something can be improved, please do so and open a PR with explanations and justifications. This info also helps in the future if somebody wants to know why certain things are the way they are.

Everything's bash because the build needs bash. However, /bin/bash is never good, not even on Linux (should be /usr/bin/env bash). Regarding disk sizes keep in mind that those scripts are for provisioning our build and test machines. The work directories need appropriately sized disks (40 GB sounds about right).

[Willsparker]

ping @sxa

[aixtools]

I can ping @sxa again, but shall I put up a PR? This concerns both process and content (e.g., number of LP in a logical partition), and minimum size of harddisk - and on AIX, that should actually be the volume group, not a single physical volume.

[aixtools]

I do not know anything about AIX, so only general remarks: If you think something can be improved, please do so and open a PR with explanations and justifications. This info also helps in the future if somebody wants to know why certain things are the way they are.

Everything's bash because the build needs bash. However, /bin/bash is never good, not even on Linux (should be /usr/bin/env bash). Regarding disk sizes keep in mind that those scripts are for provisioning our build and test machines. The work directories need appropriately sized disks (40 GB sounds about right).

Right. I'll open a PR then.

That bash is 'needed' later on is a good reason to have it installed. However, imho, for a configuration script - that can run without it (better, can run with the system shell) it is better to run with the system shell.

I have no issues with bash - but this is 'process'. Needing bash for something later is 'content' - and should be dealt with separate from configuring storage.

So - assuming 40G - how much is 'work/project' that should probably be specified for /home.

As to /, /usr and /opt - where applications and application data are installed - the AIX way is to grow the filesystems, if/when needed. For RPM related packages - installed via rpm and/or yum - a check for free space could be made, and increased when not enough. (while isntallp can do this automatically).

This idea is a template I could work on, or perhaps better - write a script that can be installed as part of the base config - and then give an RPM as argument and let it install. I hope that yum also has a preview mode - so it could report how much space it needs - and increase the filesystem on demand.

However, first the basics - PR coming up (with comments).

[sxa]

I feel @sej-jackson would be in favour of these changes too!

(As would I, just to be clear!)

FYI, the people who initially wrote the AIX scripts are no longer actively working on the project so they are in need of some maintenance

[aixtools]

My first assignment :p !!

…

On 18/09/2020 12:28, Martijn Verburg wrote: Assigned #1547 <#1547> to @aixtools <https://github.com/aixtools>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSZR5PMZE4CJUUCGETB4YLSGMY4ZANCNFSM4ROVTJGQ>.

[aixtools]

A bit more serious - as AIX is not included in the CI of these files - what do I need to do to 'emulate' that. Is it possible to add a test server (that I can re-install for the rinse and repeat part) - or use one of the servers I already manage/setup.

Ultimately - if not by default - I'll want to follow the complete chain (new system, configure, run odjk and/or nodejs (jenkins?) job.

And, I suppose - along the path - I'll be able to update the AIX part of the documentation.

[aahlenst]

I have no issues with bash - but this is 'process'. Needing bash for something later is 'content' - and should be dealt with separate from configuring storage.

You have to keep one thing in mind: Because we're using Bash everywhere, we have more people that are able to review changes and the appropriate tooling is already in place. If we were to switch to ksh for AIX, we'd need at least a replacement for shellcheck.

Is it possible to add a test server (that I can re-install for the rinse and repeat part) - or use one of the servers I already manage/setup.

If you want to see whether your changes result in a working build, the easiest way is probably to provision a test machine that you have access to, run the playbooks on it afterwards and do a manual build by following the instructions in https://github.com/adoptopenjdk/openjdk-build#tldr-i-want-to-build-a-jdk-now.

[aixtools]

On 19/09/2020 11:43, Andreas Ahlenstorf wrote: I have no issues with bash - but this is 'process'. Needing bash for something later is 'content' - and should be dealt with separate from configuring storage. You have to keep one thing in mind: Because we're using Bash everywhere, we have more people that are able to review changes and the appropriate tooling is already in place. If we were to switch to ksh for AIX, we'd need at least a replacement for shellcheck.

Agreed. Bash needs to be available - and apparently it was installed (in the wrong way, imho should not expect #!/bin/bash) before this script gets run. FYI: Their was nothing in the previous script that needed bash. My idea is to find where it is being installed (using your hint below, thx), and find, where it (should) be documented. In short, use system (ksh) for system config stuff; ensure (elsewhere), that bash is available for the scripts that depend on it.

Is it possible to add a test server (that I can re-install for the rinse and repeat part) - or use one of the servers I already manage/setup. If you want to see whether your changes result in a working build, the easiest way is probably to provision a test machine that you have access to, run the playbooks on it afterwards and do a manual build by following the instructions in https://github.com/adoptopenjdk/openjdk-build#tldr-i-want-to-build-a-jdk-now.

Thx. I'll chat with @sxa to steal one that has been setup, but is not yet configured. And, I'll try a scratch server as well.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSZR5OBXRKBZNDFVJMOZUTSGR4KZANCNFSM4ROVTJGQ>.

[aixtools]

Update: On the scratch server - the "I want to build...now" starting point is, imho, after the script(s) we are discussing here.

On a scratch server (no FS config, no OSS tools installed, no Java installed - I started the script - after cloning - as ./makejdk-any-platform.sh jdk11u.

This gave various warnings: I needed to install Java8_64 and set JDK8_BOOT_DIR to get past some warnings and cannot find errors:

But now it dies with:

BUILD_CONFIG[WORKING_DIR]="./build/"
BUILD_CONFIG[WORKSPACE_DIR]="/root/openjdk-build/workspace"
Running gradle with /usr/java8_64 at /root/openjdk-build/workspace/.gradle
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.5/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing
> Task :clean
> Task :compileJava NO-SOURCE
> Task :compileGroovy
> Task :processResources NO-SOURCE
> Task :classes
> Task :shadowJar

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.5/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 42s
3 actionable tasks: 3 executed
Exception in thread "main" java.lang.IllegalArgumentException: No group with name <pre>
        at java.util.regex.Matcher.getMatchedGroupIndex(Matcher.java:1327)
        at java.util.regex.Matcher.group(Matcher.java:583)
        at java_util_regex_Matcher$group$0.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
        at common.VersionInfo.match223(VersionInfo.groovy:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:193)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:61)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
        at common.VersionInfo.parse(VersionInfo.groovy:23)
        at common.VersionInfo$parse.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
        at ParseVersion.parseFromStdIn(ParseVersion.groovy:45)
        at ParseVersion.main(ParseVersion.groovy:14)
root@osutest:[/root/openjdk-build]

All very interesting - as I can really check/find perquisites needed for AIX, BUT(!) I still not not how to manually "ci" this repository.

Michael

[aahlenst]

You need a JDK 10 as boot JDK to build 11 and pass that with -j to ./makejdk-any-platform.sh. https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/aix.yml should install everything you need.

[aixtools]

So - how to bootstrap when I do not have any JDK? Or do I need to start with jdk9?

This is bootstrap from "I currently have nothing".

Edit: Update: answer No: jdk9 as 'target' fails in the same way.

Edit2: jdk8u as target also fails. So, question: how to start 'from nothing'.

[aixtools]

You need a JDK 10 as boot JDK to build 11 and pass that with -j to ./makejdk-any-platform.sh. https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/aix.yml should install everything you need.

Do you mean by above that I should call ./makejdk-any-platform.sh URL to ...playbooks/aix.yml?

Going to try that, in any case. (update: fails as above, but you probably knew it would).

Edit add: I assume nothing - Blinders (as in horse pulling trailer) on, view on 'Infinity'. (I have no big issue with falling all over missing assumptions - as that is the best way to document them all). 'Rinse and Repeat' til the wash is perfect!

[aahlenst]

You really have to run https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/aix.yml before ./makejdk-any-platform.sh.

And that happens by ansible-playbook -i "localhost," --skip-tags adoptopenjdk,jenkins playbooks/playbooks/aix.yml if you run it on the AIX box. If you have a local machine that isn't Windows and want to provision the AIX box from there, run ansible-playbook -i "<ip of aix box>," --skip-tags adoptopenjdk,jenkins playbooks/playbooks/aix.yml. You might need -u <your login user> and --ask-pass if you login with a password.

[aixtools]

:) Thx. One step back - then more steps forward.

Edit: and where do I find 'ansible-playbook'? By 'installing' ansible? (on AIX, or Windows, or Linux, etc..)?

[aahlenst]

See README in https://github.com/AdoptOpenJDK/openjdk-infrastructure/tree/master/ansible.

[If you want help via chat, there's https://adoptopenjdk.net/slack.html. It's very quiet on weekends.]

[aixtools]

I have ansible for AIX. Will try tomorrow if I have time, otherwise on Monday.

[aixtools]

Took a bit longer than expected to test local ansible: but it is working, even if the playbook is failing.

e.g., I have python3(.6) installed, but it is not looking at the right location.

So, I'll have to open a new PR for aix.yml I guess.

(py36) root@p9-aix1-ojdk06:[/root/openjdk-infrastructure/ansible]ansible-playbook -i "localhost," -C --skip-tags adoptopenjdk,jenkins playbooks/aix.yml
/opt/lib/python3.6/site-packages/cryptography/hazmat/bindings/openssl/binding.py:177: CryptographyDeprecationWarning: OpenSSL version 1.0.2 is no longer supported by the OpenSSL project, please upgrade. The next version of cryptography will drop support for it.
  utils.CryptographyDeprecationWarning,
[DEPRECATION WARNING]: DEFAULT_SQUASH_ACTIONS option, Loop squashing is deprecated and this configuration will no longer be used , use a list directly with the module argument instead. This
feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [all] **************************************************************************************************************************************************************************************

TASK [Load AdoptOpenJDKs variable file] *********************************************************************************************************************************************************
ok: [localhost]

TASK [Check if bash is installed] ***************************************************************************************************************************************************************
[WARNING]: No python interpreters found for host localhost (tried ['/usr/bin/python', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python',
'/usr/bin/python3', 'python'])
fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "module_stderr": "Shared connection to localhost closed.\r\n", "module_stdout": "/bin/sh: /usr/bin/python:  not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}
...ignoring

TASK [install bash] *****************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": ["Could not detect which major revision of yum is in use, which is required to determine module backend.", "You should manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})"]}

PLAY RECAP **************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1

[aixtools]

I see this as (an example of) a sub-project of #1581 - which needs a better title, more like the title for #1391

[sxa]

As an FYI we hit an issue with a new server last week which had a relatively small (I think 40G) hdisk0 and so gave up without taking into account the fact we had a 400GB hdisk0 that was allocated to /home/jenkins

[aixtools]

I guess I can try and move this off the back burner.

[Haroon-Khel]

@aixtools Any movement on this?

[aixtools]

This is a good 'ping', I have some work on CIS AIX Security Benchmark to finish up, plus issues from a power outage as OSU - then I can start on this again.

[aixtools]

Just reviewed - and the last work on this was before the current playbook structure was used (using many role definitions as sub-tasks).

The goal now is to remove this file and get everything into a playbook. Will open a new issue for that.

[aixtools]

Closing this issue. New issue is #2336