Merge the GMT FTP server into Data Server?

[seisman]

The GMT FTP site (ftp://ftp.soest.hawaii.edu/gmt) is the site for distributing GMT tarballs. Before we migrated to GitHub, it's the only official site for downloading GMT tarballs.

As mentioned in the bus-factors repository, it has a bus factor = 1, as Paul is the only person who can manage files in the FTP server.

Considering that "FTP is almost dead" (https://filecamp.com/blog/top-5-reasons-ftp-dead/, https://www.ghacks.net/2019/08/16/google-chrome-82-wont-support-ftp-anymore/), should we migrate the GMT FTP server to the Data Server? For example, put all tarballs in the releases subdirectory?

Pros:

1. the Data server uses HTTP or HTTPS, not FTP.
2. @PaulWessel @joa-quim @leouieda @seisman all can manage the files in the data server, so the bus factor is 4.
3. The data server already have many mirrors

We can still keep the FTP server for backward compatibility, but we may retire it after 5-10 years.

[seisman]

Ping @PaulWessel @joa-quim @leouieda for comments.

[PaulWessel]

Yes, we can. It adds ~21 Gb so not too much. Keeping the old server still means maintaining it, by me, though. Do the ubuntus of the world need to be told we changed where things are? Can we retire the old server earlier?

[joa-quim]

I'm afraid + 21 Gb will push my quota close to its limits (~100 Gb) for no good reason. Most of the material in the FTP site is repeated stuff or oooold versions. They can be stored in a single site only.

[PaulWessel]

I also see little value in mirroring old versions. Perhaps the ftp cen be the graveyear for old versions and the release directory can hold the current version only.

[leouieda]

@PaulWessel how about archiving the old releases on Zenodo? Then we can leave the FTP site for now but can be retired without loosing those builds.

[PaulWessel]

We could do that. Any complications of going backwards in time when adding to the same "collection"? Usually we add the next version but now we would add old versions?

[seisman]

Please note that old GMT releases may have multiple tarballs, for example, GMT 4.5.6 have five different tarballs

gmt-4.5.6-doc.tar.bz2
gmt-4.5.6-share.tar.bz2
gmt-4.5.6-src.tar.bz2 
gmt-4.5.6-suppl.tar.bz2 
gmt-4.5.6-triangle.tar.bz2

[PaulWessel]

That is true. Might it be acceptable to repackage these in a single archive to avoid this mess? I expect the demand for 4.5.6 is not urgent.

[maxrjones]

The Zenodo option for old releases seems good. I could try one of these old releases to see if Zenodo will accept multiple tarballs for a single release. I prefer that over repackaging old releases, because that seems more accurate to me. The older releases for GMT 4 and 5 do not already have DOI's, correct?

[seisman]

The older releases for GMT 4 and 5 do not already have DOI's, correct?

Yes.

I prefer that over repackaging old releases, because that seems more accurate to me.

It means more work on us (mainly @PaulWessel), and it may also break the old install instructions for these old versions.

[PaulWessel]

Could we just uncompress to *.tar, add the installer script, then zip the 5 tar files into a zip and register that one instead? Upon extraction the user should have all the pieces.

[maxrjones]

Firefox just disabled FTP support: https://www.mozilla.org/en-US/firefox/88.0/releasenotes/?utm_source=firefox-browser&utm_medium=firefox-desktop&utm_campaign=about-dialog. I would prefer to start putting releases on the Data Server.

[maxrjones]

Could we just uncompress to *.tar, add the installer script, then zip the 5 tar files into a zip and register that one instead? Upon extraction the user should have all the pieces.

Seems OK. What is the motivation for repackaging as a single tarball?

[PaulWessel]

Well, it would be a single file to download but it would contain upon expansion 5 tarballs so that the install script still works. if we truly repackaged into a single tar then we would have to waste time rewriting a GMT4 install script and who needs that headache.

[maxrjones]

Well, it would be a single file to download but it would contain upon expansion 5 tarballs so that the install script still works. if we truly repackaged into a single tar then we would have to waste time rewriting a GMT4 install script and who needs that headache.

OK, sounds good. Is the ftp or data server required for some distributions? If not, should we just use Zenodo and GitHub for future releases rather than continuing with 3+ storage locations?

[seisman]

Is the ftp or data server required for some distributions?

The FTP or data server is required for the mirrors.

[maxrjones]

Is the ftp or data server required for some distributions?

The FTP or data server is required for the mirrors.

I understand that the FTP and data server are required for the mirrors. More specifically, do future releases need to be on these mirrors? Is this a requirement for people to be able to download releases in reasonable amounts of time? There is a trade-off between the maintenance efforts + server size (which could dissuade potential mirror institutions) and convenience for people downloading GMT, so I am trying to understand whether the decision has already been made that the improved download speeds is worth the efforts and space required to continue posting new releases on either the FTP or data server in addition to Zenodo/GitHub.