Batch retrieval of download counts with /api/download_counts

[HebaruSan]

Motivation

CKAN queries SpaceDock for download counts every day. Currently this happens via the /api/mod/<mod_id> route, which means that for each SpaceDock mod that CKAN knows about, we do:

• One HTTPS request with a lot of extra output that isn't needed
• One SQL request with a lot of data sent that isn't needed

This isn't a big problem today, but it is mildly wasteful of resources that could otherwise be available for users.

Changes

Now a new /api/download_counts route is available that takes multiple mod_id form parameters as input, then runs one SQL query that returns the download counts for all the mods without extraneous info and returns them as one JSON object.

To test locally:

curl -d 'mod_id=1&mod_id=2&mod_id=3' http://localhost:5080/api/download_counts

Once this is live, KSP-CKAN/NetKAN-Infra#228 will allow download counts to be queried with just one HTTPS and SQL request per day.

[DasSkelett]

I tend to prefer putting objects of equal "type" in a list instead of a dict in JSON.
So basically like this:

{
  "download_counts": [
    {
      "id": 0,
      "downloads": 999
    },
    {
      "id": 1,
      "downloads": 666
    }
  ]
}

instead of

{
  "0": 999,
  "1": 666
}

This helps if you need to map the JSON to a language-native class/struct/..., because you can almost always easily specify that something is a list of objects of a certain type, while it might not be easy to model a dict with arbitrary keys.

For CKAN it doesn't really matter because we don't do that and and simply iterate over the results (which is very easy thanks to Python's dynamically typed nature). But it could be useful for other people who might consume the API.

[HebaruSan]

I think it would have been pretty reliable to deserialize that into a Dictionary<int, int>, but I do see some value in being able to represent a single "pair" as one object. Changed, will rewrite the NetKAN-Infra PR next.

[DasSkelett]

Very nice, thank you!

[HebaruSan]

Command to get all counts from alpha:

curl -d $(seq 1 100|head -c -1|sed -e 's/^/mod_id=/'|tr '\n' '&') https://alpha.spacedock.info/api/download_counts|jq