DPLA (Digital Public Library of America) Dataset on IPFS

[flyingzumwalt]

The Digital Public Library of America (DPLA) provides open and coherent access to our society’s digitized cultural heritage by aggregating info about all the digital materials held at many of the universities, public libraries, and other public-spirited organizations in the USA. It's a huge trove of metadata with pointers to a massive amount of digital materials that don't get enough attention.

Anyone interested in putting the DPLA dataset on IPFS? @cmh2166 @anarchivist @dchud @mjgiarlo @edsu @bibliotechy

It would also be possible to put the whole DPLA metadata processing pipeline onto IPFS. @chadfennell ?

[danfowler]

@flyingzumwalt did you get anywhere here?

[flyingzumwalt]

@danfowler I haven't gotten any nibbles on this one, but I do know that @mdellabitta has recently done great work converting DPLA's internal ETL workflows to use Apache Spark. This makes me suspect that it would be very easy to pipe a copy of their dataset into IPFS.

This might also be a good point to experiment with using IPFS to track derivative datasets that people produce based on the complete DPLA set. Likewise it might be a good time to explore using IPFS in the aggregation flow from DPLA hubs to the national aggregator.

[cmharlow]

Not specific to DPLA IPFS question, I'm wondering about if we made a ipfs channel on the code4lib slack and had informal regular calls to catch up on various experiments or questions - like the spark channel that emerged after code4lib conf this yet. I know I have particular experiments and questions I'd like to explore with others - this being a part of one. As well as experiments occurring in other GLAM spaces. It might help get shared momentum on an experiment like this or the IPLD and authorities one better running and coordinated so it doesn't fall on one person's schedule alone. (I know slack isn't ideal bc it's a closed system, but it seems the best space for what we want to do above, rn, imho)

[flyingzumwalt]

I definitely want to give these discussions a home that works. Redirecting that discussion to ipfs/community#224