Prefix directories for scalability

[joshmoore]

It may well be too late, but looking at this for the first time after a discussion with @bgruening, if other communities are likely to join and the list grows further (which would be great! 🎉), I wonder if adding a top-level prefix directory (/1/1000genomes) to reduce the total number of listings in a single directory by a factor of ~36 has been discussed. (Currently there are 38543 entries in the data directory.)

See for example the package links in Debian:

http://ftp.de.debian.org/debian/pool/main/a/acct/acct_6.6.4-5+b1_amd64.deb

[bgruening]

Hi @joshmoore! Yes we should do that and adopt our bots accordingly.