Populate release_date property

[HebaruSan]

Motivation

Users ask fairly frequently for the ability to sort modules by release date. Up to now we can't provide that info because the bot is stateless, and keeping track of the first time that we found a version requires state.

Changes

Now the Indexer will add a release_date property to each module that it processes:

• If the file exists, it will pull the release_date property forward unchanged
• If the file exists but has no release_date property, it will set it to the file creation date in git
• If the file is brand new, it will set the release_date property to now

Note that the schema/spec currently do not have a release_date property. It will have to be added before this can be merged. In the meantime I want to see how many tests fail and why.

Fixes #6.
Fixes KSP-CKAN/CKAN#2916.

[techman83]

I have 2 comments without looking too closely at the code

• This is a break away from our long established single source of generation and letting the indexer make decisions
• The indexer should not be responsible for historical data

Now I'm not opposed to adding this to the meta data, but we really must flesh out the delineation of responsibility.

• We probably should have the discussion on the feature as it feels a little beyond the scope of an implementation detail.
• Historical data should be out of scope for the indexer. We have done this before with the hashes, we crawled all the metadata, added the hashes and mirrored them. In this case we can probably do it in a single pass as it is purely a metadata change, rather than downloading thousands of mods and calculating checksums.

[HebaruSan]

This is a break away from our long established single source of generation and letting the indexer make decisions

I think that's unavoidable. See the comments in KSP-CKAN/CKAN#2916; the Inflator doesn't know whether a file that it has generated already exists, so some other entity has to be in charge of copying forward already established release dates.

The indexer should not be responsible for historical data

What's the reason behind that? The Indexer is built around a Git repo, so the historical data is all right there, and it is responsible for integrating new data into the existing database. Seems like a pretty natural fit.

[techman83]

I think that's unavoidable. See the comments in KSP-CKAN/CKAN#2916; the Inflator doesn't know whether a file that it has generated already exists, so some other entity has to be in charge of copying forward already established release dates.

Yes, I think we're going to have to step away from this slightly (as. But I feel very strongly the source of this truth needs to be the inflator. It gives us scope to implement the release dates from the APIs we consume where available and leaves the indexer with the sole job of deciding of handling/publishing changes. As noted in KSP-CKAN/CKAN#2916

What's the reason behind that? The Indexer is built around a Git repo, so the historical data is all right there, and it is responsible for integrating new data into the existing database. Seems like a pretty natural fit.

To clarify, I'm not opposed to creating a stub like I did for the restore_status. But it doesn't belong in the main indexing logic as it will be a once off task. The method itself looks ok to me, which we could use in a 'populate_historical_release_dates' if we wanted.

It also relies on a deep clone (which we could make optional), which we just don't need in regular operations. Part of my reasoning for using shallow cloning was to reduce our memory usage as during startup, as time goes on and as we add more services that'll become more important.

[HebaruSan]

Ahh right, shallow clones. FWIW, I think I checked this before, but just now I got 137.7 MiB for a deep clone and 100.1 MiB for shallow, a difference of 25% of the size of the deep clone.

[techman83]

Ahh right, shallow clones. FWIW, I think I checked this before, but just now I got 137.7 MiB for a deep clone and 100.1 MiB for shallow, a difference of 25% of the size of the deep clone

That's space on disk, which in reality is neither here nor there. Memory usage during clone is where the bigger difference is.

[HebaruSan]

Ahh, OK. Looks like that's on the order of a 45% difference:

$ /usr/bin/time --format='%M KiB' git clone https://github.com/KSP-CKAN/CKAN-meta.git
48804 KiB

$ /usr/bin/time --format='%M KiB' git clone --depth 1 https://github.com/KSP-CKAN/CKAN-meta.git
26616 KiB

[HebaruSan]

Replaced by KSP-CKAN/CKAN#3059.