API sometimes can't find networks created via CLI while API is running

[arctic-alpaca]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running the API service, creating a network via CLI sometimes isn't reflected in the API.

Steps to reproduce the issue:

1. podman system service unix:///home/`whoami`/testing.sock --log-level=debug --time=500

2. podman network create testns && sleep 5 && curl -v -X DELETE --unix-socket /home/`whoami`/testing.sock http://d/v3.0.0/libpod/networks/testns

Describe the results you received:
{"cause":"network not found","message":"unable to find network with name or ID testns: network not found","response":404}
The network can't be found via API. The list networks endpoint also doesn't return the testns network:

alpaca@DESKTOP:~$ curl --unix-socket /home/`whoami`/testing.sock http://d/v3.0.0/libpod/networks/json
[{"name":"podman","id":"2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9","driver":"bridge","network_interface":"cni-podman0","created":"2021-10-01T15:53:27.340335748+02:00","subnets":[{"subnet":"10.88.0.0/16","gateway":"10.88.0.1"}],"ipv6_enabled":false,"internal":false,"dns_enabled":false,"ipam_options":{"driver":"host-local"}}]

Describe the results you expected:
The testns network get's deleted

Additional information you deem important (e.g. issue happens only occasionally):
I'm writing a library wrapping the API, when running tests against the networks endpoints, most fail. Running each test seperate works for every test. When testing the issue with curl, I could reproduce it every time. Restarting the API allows to see the network with the list endpoint and to delete it via API.

Output of podman version:

Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.15.9
Git Commit:   fedd9cc120cae6cc2df19d49ed916f35fc5f5d71
Built:        Fri Oct  1 15:33:42 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.0.31-dev, commit: 8d4355071e51ba4b597fc7ccb410808da73e54d5'
  cpus: 12
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: file
  hostname: DESKTOP
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.16.3-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 6488829952
  memTotal: 13343838208
  ociRuntime:
    name: runc
    package: 'runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0~rc93+ds1
      commit: 1.0.0~rc93+ds1-5+b2
      spec: 1.0.2-dev
      go: go1.15.9
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 7h 13m 52.39s (Approximately 0.29 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/alpaca/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/alpaca/.local/share/containers/storage
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /tmp/podman-run-1000/containers
  volumePath: /home/alpaca/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1633095222
  BuiltTime: Fri Oct  1 15:33:42 2021
  GitCommit: fedd9cc120cae6cc2df19d49ed916f35fc5f5d71
  GoVersion: go1.15.9
  OsArch: linux/amd64
  Version: 4.0.0-dev

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes, using self built latest Github dev version.

Additional environment details (AWS, VirtualBox, physical, etc.):
Windows 10 WSL2, Debian Bullseye

[Luap99]

This is expected in the current implementation. I personally think this is a rare use case.

To fix this we would have to use inotify which causes problem for rootless users or read the config files everytime you do a network create/ls/rm, etc...
To provide the best performance I decided against it but I am open for suggestions.

@mheon @baude @rhatdan WDYT?

[mheon]

How are we deciding when to reload the list of available networks now - every time the API endpoints are hit, or less frequent? It's probably inexpensive enough to do it frequently if that will resolve this without inotify.

[Luap99]

We are never reloading once they are loaded. Only networks that are created and removed via api are updated.

[arctic-alpaca]

I'm using the CLI to test my API wrapper implementation so I don't have to rely on parts of my code to test other parts. If this would be possible with a flag or something similar, this would be awesome.

[Luap99]

Idea, stat the config dir and store the the mtime, if it changed we reload. I think that should work.

[jwhonce]

Implementation Detail: We could add a middleware handler to the necessary API endpoints to reload the list. This could work well with #11828 (comment)

Current middleware handlers that we use log trace data, log panics, support X-Reference-Id headers.

[jwhonce]

@Luap99 9f9ec93 is a POC. Assuming the new networking stack can be extended for that code to call a NetworkListRefresh() function.

[Luap99]

@jwhonce I would prefer to handle this inside the network interface. There is already logic which loads the networks on the first call. This way we only need to check if the network interface is actually used.

[mheon]

I'm still wondering how costly a full refresh is; if it's cheap, there's not much reason not to do it every time.

[Luap99]

Well we need to read and unmarshal every config file in the directory so it is definitely costly. Keep in mind that this process is locked.