CNI cache error pops up when a container is removed

[linggao]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I have seen different errors after the original network is removed from a container, including DNS error, resizing session error and CNI cache error.

Steps to reproduce the issue:

1. podman network create foo-a
   podman network create foo-b

2. podman run --name test --network foo-a -d alpine sleep 10000

3. podman network connect foo-b test
   podman network disconnect foo-a test

4. wait as long as you want

5. podman rm -f test
   ERRO[0010] error loading cached network config: network "foo-b" not found in CNI cache
   WARN[0010] falling back to loading from existing plugins on disk
   a5092f2db3f45dde2e785fae5ef2ba4c70843a5e0c220744fe1f64a4e4234503

Describe the results you received:
Please see output from step 5.

Describe the results you expected:
No errors.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.1.0-dev
API Version:  3.0.0
Go Version:   go1.14.12
Git Commit:   426178a49991106ffe222f12cc42409ae78dd257-dirty
Built:        Tue Mar  2 16:08:11 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.0.27-dev, commit: 7310bf13319ee8ed50799b202509bedc27b36cf8'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: lingvs4.dev.edge-fabric.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-240.15.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 2636328960
  memTotal: 8342470656
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc93+dev
      commit: f245a1d1edbf545549e5a16106cf1aec356a3c7d
      spec: 1.0.2-dev
      go: go1.14.12
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2146758656
  swapTotal: 2146758656
  uptime: 316h 24m 18.01s (Approximately 13.17 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 10
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1614722891
  BuiltTime: Tue Mar  2 16:08:11 2021
  GitCommit: 426178a49991106ffe222f12cc42409ae78dd257-dirty
  GoVersion: go1.14.12
  OsArch: linux/amd64
  Version: 3.1.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman is built from the latest master.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[Luap99]

#9614 should fix this