Error deploying OracleRealApplicationClusters

[pushpi55]

I am following the steps mentioned in the github page to create racnode
https://github.com/oracle/docker-images/tree/main/OracleDatabase/RAC/OracleRealApplicationClusters
I installed all the prerequisite conatiners, i.e DNS and Storage containers for NFS volumes.
docker start racnoded1 is giving the following error

07-03-2024 07:55:07 UTC : : Generating DB Responsefile Running DB creation
07-03-2024 07:55:07 UTC : : Running DB creation
07-03-2024 07:55:07 UTC : : Workaround for Bug 32449232 : Removing /u01/app/grid/kfod
07-03-2024 07:55:13 UTC : : Checking DB status
07-03-2024 07:55:13 UTC : : ORCLCDB is not up and running on racnoded1
07-03-2024 07:55:13 UTC : : Error has occurred in Grid Setup, Please verify

Upon delving more in the installation logs, found that the img files generated by Storage container, does not sufficient permissions and they are not part of proper group and user.

Access Control List check ...FAILED
PRVF-9991 : Owner of device "/oradata/asm_disk01.img" did not match the
expected owner. [Expected = "grid"; Found = "oracle"] on nodes: [racnoded1]
PRVF-9991 : Owner of device "/oradata/asm_disk02.img" did not match the
expected owner. [Expected = "grid"; Found = "oracle"] on nodes: [racnoded1]
PRVF-9991 : Owner of device "/oradata/asm_disk03.img" did not match the
expected owner. [Expected = "grid"; Found = "oracle"] on nodes: [racnoded1]
PRVF-9991 : Owner of device "/oradata/asm_disk05.img" did not match the
expected owner. [Expected = "grid"; Found = "oracle"] on nodes: [racnoded1]
PRVF-9991 : Owner of device "/oradata/asm_disk04.img" did not match the
expected owner. [Expected = "grid"; Found = "oracle"] on nodes: [racnoded1]
PRVF-9992 : Group of device "/oradata/asm_disk01.img" did not match the
expected group. [Expected = "asmadmin"; Found = "oinstall"] on nodes:
[racnoded1]
PRVF-9992 : Group of device "/oradata/asm_disk02.img" did not match the
expected group. [Expected = "asmadmin"; Found = "oinstall"] on nodes:
[racnoded1]
PRVF-9992 : Group of device "/oradata/asm_disk03.img" did not match the
expected group. [Expected = "asmadmin"; Found = "oinstall"] on nodes:
[racnoded1]
PRVF-9992 : Group of device "/oradata/asm_disk05.img" did not match the
expected group. [Expected = "asmadmin"; Found = "oinstall"] on nodes:
[racnoded1]
PRVF-9992 : Group of device "/oradata/asm_disk04.img" did not match the
expected group. [Expected = "asmadmin"; Found = "oinstall"] on nodes:
[racnoded1]

There are other failures as well. Pasting it here

Pre-check for cluster services setup was unsuccessful on all the nodes.

Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Swap Size ...FAILED
racnoded1: PRVF-7573 : Sufficient swap size is not available on node
"racnoded1" [Required = 16GB (1.6777216E7KB) ; Found = 0.0 bytes]

Device Checks for ASM ...FAILED
ASM device sharedness check ...FAILED
Shared Storage
Accessibility:/oradata/asm_disk01.img,/oradata/asm_disk02.img,/oradata/asm_d
isk03.img,/oradata/asm_disk05.img,/oradata/asm_disk04.img ...FAILED
racnoded1: PRVG-0809 : Mount options for file system
"dbdevfssmnt-prod01vcn.com:/hom
es/prod/test/docker_volumes/asm_vol/ORCLCDB" mounted on path
"/oradata/asm_disk01.img" at node "racnoded1" did not meet the
requirements for this platform [Expected =
"rw|!ro,hard|!soft,rsize>=32768,wsize>=32768,tcp|proto=tcp,v3|nfs
v3|vers=3|nfsvers=3|v4|nfsv4|vers=4|nfsvers=4|vers=4.0|vers=4.1|n
fsvers=4.1,timeo>=600,acregmin=0&acregmax=0&acdirmin=0&acdirmax=0
|actimeo=0" ; Found =
"rw,nosuid,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255
,hard,noacl,proto=tcp,timeo=600,retrans=3,sec=sys,mountaddr=100.7
0.122.116,mountvers=3,mountport=2048,mountproto=tcp,local_lock=no
ne,addr=x.y.z.a"]

created the NFS storage using the container, command used.

          docker run -d -t --hostname racnode-storage --dns-search=example.com  --cap-add SYS_ADMIN --cap-add AUDIT_WRITE --volume /home/test/docker_volumes/asm_vol/$ORACLE_DBNAME:/oradata --init --network=rac_priv1_nw --ip=192.168.17.80 --tmpfs=/run  --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --name racnode-storage oracle/rac-storage-server:19.3.0

Used the following command to create RACnode

docker create -t -i --hostname racnoded1 --volume /boot:/boot:ro --volume /dev/shm --tmpfs /dev/shm:rw,exec,size=4G --volume /opt/containers/rac_host_file:/etc/hosts --volume /opt/.secrets:/run/secrets:ro --dns=172.16.1.25 --dns-search=example.com --privileged=false --volume /home/test/docker_volumes/asm_vol/ORCLCDB:/oradata --cap-add=SYS_NICE --cap-add=SYS_RESOURCE --cap-add=NET_ADMIN -e DNS_SERVERS="172.16.1.25" -e NODE_VIP=172.16.1.130 -e VIP_HOSTNAME=racnoded1-vip -e PRIV_IP=192.168.17.100 -e PRIV_HOSTNAME=racnoded1-priv -e PUBLIC_IP=172.16.1.100 -e PUBLIC_HOSTNAME=racnoded1 -e SCAN_NAME=racnodedc1-scan -e OP_TYPE=INSTALL -e DOMAIN=example.com -e ASM_DISCOVERY_DIR=/oradata -e ASM_DEVICE_LIST=/oradata/asm_disk01.img,/oradata/asm_disk02.img,/oradata/asm_disk03.img,/oradata/asm_disk04.img,/oradata/asm_disk05.img -e CMAN_HOSTNAME=racnodedc1-cman -e CMAN_IP=172.16.1.164 -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc -e PWD_KEY=pwd.key -e RESET_FAILED_SYSTEMD="true" --restart=always --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --cpu-rt-runtime=95000 --ulimit rtprio=99 --name racnoded1 oracle/database-rac:21.3.0

Let me know if you need any other info
Docker info

docker info
Client:
Debug Mode: false
Server:
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 142
Server Version: 19.03.11-ol
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
runc version: 51d5e94
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.4.17-2136.330.7.1.el7uek.x86_64
Operating System: Oracle Linux Server 7.9
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.06GiB
Name: phoenix526501
ID: JHPS:YBTM:3KDE:TXA5:JBGL:WYYL:FJVT:5EPW:LGEH:DG2Z:7MME:XCSN
Docker Root Dir: /scratch/docker
Debug Mode: false
HTTP Proxy: http://internalproxy
HTTPS Proxy: http://internalproxy
No Proxy: localhost,127.0.0.1,
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries:

[saurabhuja]

Please give the output of below-

1. docker logs -f rac-dnsserver
2. docker logs racnode-storage
3. docker ps -a
4. docker network ls
5. docker volume ls
6. docker volume inspect racstorage
7. ls -lart /docker_volumes/asm_vol/$ORACLE_DBNAME/

[saurabhuja]

Looks like u r not following document , why do u have -volume /home/test/docker_volumes/asm_vol/ORCLCDB:/oradata , it has to come from docker volume racstorage. Can you try using it?

[saurabhuja]

I followed below commands and it went fine in OL7.9

docker network create --driver=bridge --subnet=172.16.1.0/24 rac_pub1_nw
docker network create --driver=bridge --subnet=192.168.17.0/24 rac_pzriv1_nw
docker create --hostname racdns --dns-search=example.com --cap-add=AUDIT_WRITE -e DOMAIN_NAME="example.com" -e WEBMIN_ENABLED=false -e RAC_NODE_NAME_PREFIXD="racnoded" -e SETUP_DNS_CONFIG_FILES="setup_true" --privileged=false --name rac-dnsserver oracle/rac-dnsserver:latest
docker network disconnect bridge rac-dnsserver
docker network connect rac_pub1_nw --ip 172.16.1.25 rac-dnsserver
docker network connect rac_pzriv1_nw --ip 192.168.17.25 rac-dnsserver
docker start rac-dnsserver
docker logs rac-dnsserver
export ORACLE_DBNAME=ORCLCDB
mkdir -p /docker_volumes/asm_vol/$ORACLE_DBNAME
rm -rf /docker_volumes/asm_vol/$ORACLE_DBNAME/asm_disk0*
docker run -d -t --hostname racnode-storage --dns-search=example.com --cap-add SYS_ADMIN --cap-add AUDIT_WRITE --volume /docker_volumes/asm_vol/$ORACLE_DBNAME:/oradata --init --network=rac_pzriv1_nw --ip=192.168.17.80 --tmpfs=/run --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --name racnode-storage oracle/rac-storage-server:19.3.0
docker logs -f racnode-storage
docker volume ls
docker volume rm racstorage
docker volume create --driver local --opt type=nfs --opt o=addr=192.168.17.80,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 --opt device=192.168.17.80:/oradata racstorage
docker create -t -i --hostname racnoded1 --volume /boot:/boot:ro --volume /dev/shm --tmpfs /dev/shm:rw,exec,size=4G --volume /opt/containers/rac_host_file:/etc/hosts --volume /opt/.secrets:/run/secrets:ro --dns=172.16.1.25 --dns-search=example.com --privileged=false --volume racstorage:/oradata --cap-add=SYS_NICE --cap-add=SYS_RESOURCE --cap-add=NET_ADMIN -e DNS_SERVERS="172.16.1.25" -e NODE_VIP=172.16.1.130 -e VIP_HOSTNAME=racnoded1-vip -e PRIV_IP=192.168.17.100 -e PRIV_HOSTNAME=racnoded1-priv -e PUBLIC_IP=172.16.1.100 -e PUBLIC_HOSTNAME=racnoded1 -e SCAN_NAME=racnodedc1-scan -e OP_TYPE=INSTALL -e DOMAIN=example.com -e ASM_DISCOVERY_DIR=/oradata -e ASM_DEVICE_LIST=/oradata/asm_disk01.img,/oradata/asm_disk02.img,/oradata/asm_disk03.img,/oradata/asm_disk04.img,/oradata/asm_disk05.img -e CMAN_HOSTNAME=racnodedc1-cman -e CMAN_IP=172.16.1.164 -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc -e PWD_KEY=pwd.key -e RESET_FAILED_SYSTEMD="true" --restart=always --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --cpu-rt-runtime=95000 --ulimit rtprio=99 --memory 16G --memory-swap 32G --name racnoded1 oracle/database-rac:21.3.0
docker network disconnect bridge racnoded1
docker network connect rac_pub1_nw --ip 172.16.1.100 racnoded1
docker network connect rac_pzriv1_nw --ip 192.168.17.100 racnoded1
docker start racnoded1
docker logs -f racnoded1
07-04-2024 08:00:13 UTC : : ####################################
07-04-2024 08:00:13 UTC : : ORACLE RAC DATABASE IS READY TO USE!
07-04-2024 08:00:13 UTC : : ####################################

[saurabhuja]

your docker volume inspect racstorage shall come like below-

docker volume inspect racstorage
[
{
"CreatedAt": "2024-07-04T07:31:22Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/racstorage/_data",
"Name": "racstorage",
"Options": {
"device": "192.168.17.80:/oradata",
"o": "addr=192.168.17.80,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0",
"type": "nfs"
},
"Scope": "local"
}
]