Neo4j 4.2.4 Operation not permitted

[mdelmoral]

Hi,

I come from here because I thought it was a problem in Testcontainers but we tried to run a single Neo4j container directly on Docker and it didn't work.

The Docker environment in our CI system is:

Connected to docker: 
  Server Version: 18.09.2
  API Version: 1.39
  Operating System: Ubuntu 16.04.2 LTS
  Total Memory: 3008 MB

And the Docker command that we ran was:

docker run \
    --name neo4j42 \
    -p7474:7474 -p7687:7687 \
    -d \
    -e NEO4J_AUTH=neo4j/test \
    -e NEO4J_dbms_memory_heap_max__size=416M \
    -e NEO4J_dbms_recovery_fail__on__missing__files=false \
    -e NEO4J_apoc_export_file_enabled=true \
    -e NEO4J_apoc_import_file_enabled=true \
    -e NEO4J_apoc_import_file_use__neo4j__config=true \
    -e NEO4JLABS_PLUGINS=\[\"apoc\"\] \
    neo4j:4.2.4

The container didn't start and reading the docker logs with docker logs neo4j42 the content is:

stat: cannot statx '/logs': Operation not permitted
Warning: Folder mounted to "/logs" is not writable from inside container. Changing folder owner to neo4j.
stat: cannot statx '/data': Operation not permitted
Warning: Folder mounted to "/data" is not writable from inside container. Changing folder owner to neo4j.
stat: cannot statx '/data/databases': Operation not permitted
Warning: Folder mounted to "/data/databases" is not writable from inside container. Changing folder owner to neo4j.
stat: cannot statx '/data/transactions': Operation not permitted
Warning: Folder mounted to "/data/transactions" is not writable from inside container. Changing folder owner to neo4j.
Changed password for user 'neo4j'.
Fetching versions.json for Plugin 'apoc' from https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json
Installing Plugin 'apoc' from https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.5/apoc-4.2.0.5-all.jar to /var/lib/neo4j/plugins/apoc.jar
stat: cannot statx '/var/lib/neo4j/plugins/apoc.jar': Operation not permitted
Plugin at '/var/lib/neo4j/plugins/apoc.jar' is not readable

Also, we tried with Neo4j 4.2.11 version but we got the same error.

Do you know if I missed some configuration options? Or is a Docker/Neo4j version issue?

Thank you!

[jennyowen]

I tried running this locally:

docker run -it --rm \
    -p7474:7474 -p7687:7687 \
    -e NEO4J_AUTH=neo4j/test \
    -e NEO4J_dbms_memory_heap_max__size=416M \
    -e NEO4J_dbms_recovery_fail__on__missing__files=false \
    -e NEO4J_apoc_export_file_enabled=true \
    -e NEO4J_apoc_import_file_enabled=true \
    -e NEO4J_apoc_import_file_use__neo4j__config=true \
    -e NEO4JLABS_PLUGINS=\[\"apoc\"\] \
    neo4j:4.2.4

and neo4j container started as expected with no errors about running stat on the folders.

It's very weird that it would give you file permission errors even when you're not mounting any folders, that would mean the file permissions are wrong inside of the image, but if that was the case it would happen for me as well.

Are you sure there aren't any already running containers called neo4j42 and also try clearing the neo4j docker image from your cache and trying again.

[mdelmoral]

If I run the same command in my local Docker Desktop (IOS), the container is started perfectly. My Docker Desktop environment is:

Connected to docker:
  Server Version: 20.10.10
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 1986 MB

And we don't have any container called neo4j42 or just running Neo4j. The machine where we're having this issue is our CI system and only Jenkins is running there. We tried running a single Neo4j container because we had the same issue running Testcontainers.
On the other hand, we have a clean image cache because we have tried it only once, downloading the image for the first time and then deleting it. We didn't try again because the error was already 'known'. The command response was:

Unable to find image 'neo4j:4.2.4' locally
4.2.4: Pulling from library/neo4j
eff15d958d66: Pull complete
66aa43e8673f: Pull complete
089381f525cd: Pull complete
6395fb33eddc: Pull complete
30b2a30b76f1: Pull complete
d5c3c5ce74a5: Pull complete
074f654d46e9: Pull complete
ed8932b38b05: Pull complete
Digest: sha256:bf47ffad31d8ba2617d3a332762090b1a75dac8b696d4e64e3c347139e0bce9b
Status: Downloaded newer image for neo4j:4.2.4

[jennyowen]

Someone else reported the same problem, but I've still not been able to reproduce it myself.
Some googling, plus the fact that it's not easily reproducible suggests this is something wrong with docker rather than the Neo4j image specifically.

I found this issue of someone reporting the same/similar errors in another image:
debuerreotype/docker-debian-artifacts#143

The fix there was to update docker and some other libraries.
@mdelmoral You report that the problem was reproduced on an old docker version but not a newer one, so updating might help?

[jennyowen]

OK, after more research I'm pretty sure this is a duplicate of this Docker issue:
debuerreotype/docker-debian-artifacts#97
where really basic system operations don't have permissions to run.

The fix for this is:
debuerreotype/docker-debian-artifacts#97 (comment)

I'm sure enough of this fix that I'll close the issue, but please let me know if updating Docker and/or libseccomp does not fix it.