Regular expression does not work if the name contains the symbol '_'

[6ax]

Hello, I ran into a problem that the regular expression does not work if there is a '_' character in the name.

INFO[0000] dregsy 0.5.0
DEBU[0000] lister max items set to 100
DEBU[0000] lister cache duration set to 1h0m0s
INFO[0000] using credentials from config                 registry="10.15.36.87:9100" username=mycoolusername
INFO[0000] using credentials from config                 registry="192.168.5.231:9100" username=mycoolusername
INFO[0000] skopeo version 1.11.2
INFO[0000] relay ready                                   relay=skopeo
INFO[0000] watching config file, restarting on change    file=/config.yaml
INFO[0000] syncing task                                  source="10.15.36.87:9100" target="192.168.5.231:9100" task=task1
INFO[0000] mapping                                       from="regex:test_test/.*" to=
INFO[0000] refreshing credentials                        registry="10.15.36.87:9100"
DEBU[0000] no auth refresher, skipping
INFO[0000] refreshing credentials                        registry="192.168.5.231:9100"
DEBU[0000] no auth refresher, skipping
DEBU[0000] retrieving repository list
DEBU[0000] registry scheme = http
DEBU[0000] no auth refresher, skipping
DEBU[0000] caching repository list
DEBU[0000] stopping tasks
DEBU[0000] task exited                                   task=task1
INFO[0000] all done
DEBU[0000] exit main

config.yaml

relay: skopeo
watch: true
skopeo:
  binary: skopeo
  certs-dir: /etc/skopeo/certs.d
docker:
  dockerhost: unix:///var/run/docker.sock
  api-version: 1.41
lister:
  maxItems: 100
  cacheDuration: 1h
tasks:
  - name: task1 # required
    verbose: true
    source:
      registry: 10.15.36.87:9100
      skip-tls-verify: true
    target:
      registry: 192.168.5.231:9100
      skip-tls-verify: true
    mappings:
      - from: regex:test_test/.*
        tags: ['test']

[xelalexv]

I don't think an underscore would be a problem, Go's regular expressions lib does support that. You can try test.+test/.* as your regex to double check. My guess would be though that your registry does not properly support the v2/_catalog API.

[6ax]

I think that the problem is that the namespaces are nested in the image names. Like this:

"test_test/test/xelalex/dregsy","testtest/test/xelalex/dregsy

INFO[0000] dregsy 0.5.0
INFO[0000]

Note: Automatic restart after config file change is currently off by default. You can activate
      this by adding 'watch: true' to your config. The default will change to on in the future.

DEBU[0000] lister max items set to 100
DEBU[0000] lister cache duration set to 1h0m0s
INFO[0000] using credentials from config                 registry="10.15.36.87:9100" username=admin
INFO[0000] using credentials from config                 registry="192.168.5.231:9100" username=admin
INFO[0000] skopeo version 1.11.2
INFO[0000] relay ready                                   relay=skopeo
INFO[0000] not watching config file
INFO[0000] syncing task                                  source="10.15.36.87:9100" target="192.168.5.231:9100" task=task1
INFO[0000] mapping                                       from="regex:test_test/.*" to=
INFO[0000] refreshing credentials                        registry="10.15.36.87:9100"
DEBU[0000] no auth refresher, skipping
INFO[0000] refreshing credentials                        registry="192.168.5.231:9100"
DEBU[0000] no auth refresher, skipping
DEBU[0000] retrieving repository list
DEBU[0000] registry scheme = http
DEBU[0000] no auth refresher, skipping
DEBU[0000] caching repository list
DEBU[0000] stopping tasks
DEBU[0000] task exited                                   task=task1
INFO[0000] all done
DEBU[0000] exit main

ubuntu@nexus:~/nexus-sync$ docker run --rm -it -v $(pwd)/config.yaml:/config.yaml -e LOG_LEVEL=trace xelalex/dregsy
INFO[0000] dregsy 0.5.0
INFO[0000]

Note: Automatic restart after config file change is currently off by default. You can activate
      this by adding 'watch: true' to your config. The default will change to on in the future.

DEBU[0000] lister max items set to 100
DEBU[0000] lister cache duration set to 1h0m0s
INFO[0000] using credentials from config                 registry="10.15.36.87:9100" username=admin
INFO[0000] using credentials from config                 registry="192.168.5.231:9100" username=admin
INFO[0000] skopeo version 1.11.2
INFO[0000] relay ready                                   relay=skopeo
INFO[0000] not watching config file
INFO[0000] syncing task                                  source="10.15.36.87:9100" target="192.168.5.231:9100" task=task1
INFO[0000] mapping                                       from="regex:testtest/test/xelalex/.*" to=
INFO[0000] refreshing credentials                        registry="10.15.36.87:9100"
DEBU[0000] no auth refresher, skipping
INFO[0000] refreshing credentials                        registry="192.168.5.231:9100"
DEBU[0000] no auth refresher, skipping
DEBU[0000] retrieving repository list
DEBU[0000] registry scheme = http
DEBU[0000] no auth refresher, skipping
DEBU[0000] caching repository list
DEBU[0000] stopping tasks
DEBU[0000] task exited                                   task=task1
INFO[0000] all done
DEBU[0000] exit main

[xelalexv]

I think that the problem is that the namespaces are nested in the image names. Like this:

That's supported and not a problem. Dregsy's test suite contains test cases that use this.

What type of registry are you using? Does it support the _catalog API I mentioned above, and if so, is it set up and configured correctly? You could try hitting v2/_catalog on your registry manually with curl, and see whether that returns any results.

[6ax]

ubuntu@nexus:~/nexus-sync$ curl  'http://10.15.36.87:9100/v2/_catalog' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 37820  100 37820    0     0   851k      0 --:--:-- --:--:-- --:--:--  858k
{
  "repositories": [
    "test_test/test/xelalex/dregsy",
    "testtest/test/xelalex/dregsy"
  ]
}

[xelalexv]

I think the problem happens when the repo list is retrieved. This is done using CatalogPage from the go-containerregistry lib, which follows OCI distribution spec. Maybe your reg (Nexus?) doesn't like the pagination bit. Could you try appending this query to your curl test: ?last=&n=100. That would sort-of simulate what CatalogPage is doing. It's possible that your reg does not support pagination or does it differently, which could result in an empty list being returned.

[6ax]

Hi, no problem, here is the log:

curl  'http://10.15.36.87:9100/v2/_catalog?last=&n=100' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3599  100  3599    0     0    99k      0 --:--:-- --:--:-- --:--:--  103k
{
  "repositories": [
    "test_test/test/xelalex/dregsy",
    "testtest/test/xelalex/dregsy"
  ]
}

[xelalexv]

OK, not what I expected ;-) I thought this would fail. I pushed a dregsy image to DockerHub, tag issue102, with additional log output. This will show us the list of repos right after it was retrieved from the registry.

[6ax]

Alexander, thanks for the help.
We managed to figure out the problem thanks to the image with the issue102 tag.
In our case, in the production environment, Nexus contained many more images and required the

lister:
  maxItems: 1000 

parameter to be specified.
After setting the correct value, everything worked as it should.
I think this issue should be flagged as wrong.
Thank you again.

[xelalexv]

Great to hear it's solved! I'll keep this issue open for the time being, to remind myself that I need to add a note/hint to the readme about this kind of situation. A log message about hitting the lister image may also be helpful.