etcdserver: purge old snap.db files

[heyitsanthony]

Lots of garbage db files in #7957. Should purge.

[gyuho]

lgtm. thanks!

[xiang90]

lgtm

[fanminshi]

@heyitsanthony can this potentially affect recovering db path? suppose there are 5 .snap files already and 1 xxx.snap.db from leader (maxsnap = 5). while etcdserver is applying snapshot and purgefile go routine deletes xxx.snap.db. then xxx.snap.db can't be found and etcdserver panic.

[heyitsanthony]

@fanminshi if that's the case, why is it safe to purge the v2 .snap files? Can you reproduce this issue where the server needs a backlog of five snapshots?

[fanminshi]

@heyitsanthony it seems to me that it is possible that an etcd can potentially have more than one .snap files and one .snap.db file. That's why we purge then if greater than some threshold right? In the case of #7957, there are already more than N xxx.snap.db files. If the example I gave just have 6 xxx.snap.db files, where the 6th one is from from the leader. while follower is applying the 6th xxx.snap.db file and the snap.db file gets deleted by purge function. will that be a issue then?

[heyitsanthony]

@fanminshi the purge is sorted; etcd will only purge the oldest files. I'm asking if .snap purging is safe, then why wouldn't .snap.db purging also be safe? If .snap purging is not safe, there needs to be a test case around it that breaks using the current purging policy.

[fanminshi]

@heyitsanthony I see. if purge only deletes oldest one, then purge is safe and my concern doesn't matter. LGTM