Where to fetch an up-to-date list of recently modified packages?

[lukehorvat]

(I hope this is the right place to post issues regarding the registry's main CouchDB.)

I'm trying to fetch an up-to-date list of the most recently updated npm packages, identical to what is currently shown on the https://npmjs.com homepage.

A couple of different queries I've tried:

• http://isaacs.iriscouch.com/registry/_design/app/_view/updated?descending=true&limit=5
• http://isaacs.iriscouch.com/registry/_design/app/_list/rss/modified?descending=true&limit=5

But as you can see, the very last updated/modified package is atomdoc, all the way back on Aug 26th 2014. Is there any way I can get a more up-to-date list, short of scraping https://npmjs.com?

[julianlam]

I am also interested in this, as my method of retrieving this data stopped working about two weeks ago.

I used to ping https://registry.npmjs.com/-/_list/index/modified?startkey=1439480888643, but the route returns empty data now (with a 200 OK code too, which is just... great).

[julianlam]

Hey, I think I found something...

npm-registry-couchapp/registry/rewrites.js

Line 24 in b93cfff

, { from: "/-/all/since", to:"_list/index/modified", method: "GET" }

☝️ This line seems to suggest that /-/all/since was deprecated and is automatically rewritten to _list/index/modified, so on a whim I tried the old url, and got 301'd to /-/all/static/today.json

So that's progress, I think, and definitely something I can use now...

https://registry.npmjs.org/-/all/static/today.json

[eirikb]

An option is to use https://skimdb.npmjs.com/registry/_changes?since=<seq>, ref http://docs.couchdb.org/en/1.6.1/api/database/changes.html .
This should allow include_docs=true, but doesn't work for me.

[julianlam]

@eirikb Yes, I ended up switching to that method shortly afterwards, as that is the "official" way to stay up to date.

[eirikb]

@julianlam Do you have an example url which includes the documents?

I'm tempted to use ../static/today.json since I'm going to periodically update once a day.

[soldair]

I would recommend using a follower if you can. Here is module https://github.com/npm/concurrent-couch-follower that we use for processes like this.

Just make sure to pass include_docs:true and set the database to skimdb.