Missing keys

[pschuermann]

Hi,
I'm using git2consul version 0.12.13 to export 15 different repos. This worked great for over 1 year. Since one week or so we see issues with missing keys:

I suspect that some content is tripping up git2consul. But I'm having a hard time pinpointing which files might cause the issue.

Here an example

0.) file and config

repos" : [{ "name" : "ltq-eng", "url" : "[email protected]:environments/ltq-eng.git", "mountpoint" : "config", "branches" : ["master"], "include_branch_name" : false, "expand_keys" : true, "ignore_file_extension" : true, "hooks": [{ "type" : "polling", "interval" : "1" }] } ...

1.) Keys get deleted
{"name":"git2consul","hostname":"e36e73527336","pid":1,"level":10,"msg":"Handling record services/hue_service/bridge/research/id of type M","time":"2017-08-17T09:38:08.414Z","v":0} {"name":"git2consul","hostname":"e36e73527336","pid":1,"level":10,"msg":"Attempting to read \"/local/git2consul_cache/ltq-eng/master/services/hue_service/bridge/research/id\"","time":"2017-08-17T09:38:08.414Z","v":0}

2.) Attempting to add keys but it's using the top level instead of config/ltq-eng/services/hue_service/bridge/research/id where it belongs:
{"name":"git2consul","hostname":"e36e73527336","pid":1,"level":10,"msg":"Adding key config/ltq-eng/, value:\nresearch","time":"2017-08-17T09:38:08.707Z","v":0}

Other keys are inserted correctly:
value:\nbranch","time":"2017-08-17T09:38:08.699Z","v":0} {"name":"git2consul","hostname":"e36e73527336","pid":1,"level":10,"msg":"Adding key config/ltq-eng/nomad/website/config-version, value:\nmaster","time":"2017-08-17T09:38:08.699Z","v":0}

Any advice is be welcome. Are there certain characters that shouldn't be used ind the values? e.g. |, #,?

[breser]

Can you try master? I just merged PR #152 that might be relevant to your case. If you have multiple keys with the same prefix it can pull down additional keys and end up deleting the wrong things. But there's not enough info in this Issue to be sure if that's what is happening or some other bug.

Have you tried deleting the git-repo and letting git2consul do a complete fresh load? Does that result in the keys still being deleted?

In the section you say "Keys get deleted" you don't actually have logs showing the keys being deleted. But there should be output since it appears you're running trace logging. So it's hard to be sure what's going on here.

If none of the above works any chance you can obfuscate anything that's sensitive and publish the repo or email it to me? If I can replicate it I can probably fix it. Thanks for taking the time to file an issue.

[pschuermann]

Hi Ben,
I tried the master and it solves my problem. All the keys are present. I'll do some more testing but it looks very good. Thanks a lot !!! Can I send you a beer via post?

Cheers
Pirmin

[breser]

You should really send the beer to @eyalzek. He found and fixed the problem. All I did was get around to writing the test for the problem and merging his change.

[eyalzek]

I accept digital beers :)

[pschuermann]

here you go: 🍻
Merry christmas