External media libraries

[erquhart]

Third party media integrations are here!! 🎉 🎉

I wrote this in about three days, so go easy 😅

This PR includes the new registerMediaLibrary API and our first integration: Uploadcare. If you haven't heard of them, go check 'em out! Integration docs are included as well.

We originally expected third party media library integrations to be API calls underneath our built in media library, but this PR takes a different route: allowing UI libraries created by services like Uploadcare and Cloudinary to replace our media library entirely. The goals of this PR were:

1. Get third party file services working fast.
2. Allow the integrations themselves to be tiny.
3. Allow projects to opt in by updating config, nothing more.

Read on for details, or try the live demo from this PR's deploy preview.

Prior art

Uploadcare built their own Netlify CMS integration back in June (shoutout to @dmitry-mukhin 🙌 ). They worked around a lot of limitations to provide a script that integrated their widget into Netlify CMS despite our lack of an API to allow it. Beside the Uploadcare specific bits, the integration mostly centered around a custom widget for the editor and an editor component to replace our auto-registered image component in the Markdown editor. Our work on the new API benefited heavily from their work, as it helped to clarify the challenges facing this kind of integration while proving that using external interfaces within the CMS really can provide a solid experience.

Retrofitting FTW

Requiring every integration to create a custom widget, editor component, and undoing our dependency on the built in image editor component requires extra time and maintenance. So we went with retrofitting instead. The current image and file widgets, which most Netlify CMS projects are already using, interact with the built in media library through methods that hook into our Redux actions. They're not actually opinionated on what that media library is, only that it provides a URL value. This applies to the existing image editor component as well.

Retrofitting means that our image and file components, along with any widgets that interact with the media library, can now control whatever media library is registered. No custom widgets necessary.

API crash course

The API is not public yet, but it will be soon. It looks like this:

• registerMediaLibrary accepts an object with properties name and init

// netlify-cms will do this automatically
import uploadcare from 'netlify-cms-media-library-uploadcare'
CMS.registerMediaLibrary(uploadcare)

• init is a function that is only called once, and should return an API object with these methods:
  • show: called when the media library needs to open, args include:
    • id: the uuid of the calling control, if any
    • value: the current value of the control, if any
    • config: the media library config from the calling control to apply to this specific instance, if any
    • imagesOnly: boolean indicating whether this instance should only accept images
  • hide: called when the media library needs to close, may not be necessary for some integrations that block page interactions (possibly the case for all of them)
  • onClearControl: called when the calling control has it's value cleared, args include:
    • id: the uuid of the calling control
  • onRemoveControl: called when the calling control is about to unmount, args include:
    • id: the uuid of the calling control
  • enableStandalone: method that should return a boolean indicating whether the media library should be opened without a calling control, e.g. via the global Media button

Bonus: you can now upload multiple images!

Only with external media libraries, and only with the ones that allow it, but still. Uploadcare's configuration docs show a multiple option for using their multiple selection widget - setting this configuration is all that's needed, the integration handles using the right widget and the image/file editor widgets can now handle multiple values!

Using the Uploadcare integration

Uploadcare's widget isn't a media library - it doesn't show things you've already uploaded - so it returns false from it's enableStandalone method. This means the Media button in the nav is removed when using Uploadcare. #notabug. The integration loads dependencies via injected script tags because they're using a bit of ES3 that breaks our builds, and so could not be fetched via npm.

Other than that, the docs cover how to use. It's just config changes!

That's all folks

We're really excited to get this out into the wild in Netlify CMS 2.1.0! Please check out the PR and the docs, test it out, etc - reviews and feedback are much appreciated ❤️

Closes #432.

TODO

• fix image fields in editor component

[verythorough]

Deploy preview for netlify-cms-www ready!

Built with commit 70655fa

https://deploy-preview-1602--netlify-cms-www.netlify.com

[verythorough]

Deploy preview for cms-demo ready!

Built with commit 70655fa

https://deploy-preview-1602--cms-demo.netlify.com

[tech4him1]

oneOf: [] will prevent media_folder from being set if media_library is. If there is a possibility a media library integration would also want media_folder to be set, we should use anyOf: [] instead. If you do want them to be mutually exclusive, we would leave it as oneOf: [].

It took me forever to understand the difference, so I figured I'd note it here. 😄

[erquhart]

Yeah I actually meant to change that, thanks for the reminder.

[dev7ch]

Cool, sounds and looks promising!

[nd0ut]

@erquhart Nice work! Custom media library API is exactly what we so lacked.

In the provided demo, there is an error when I click on "Choose an image" in the body editor:

mediaLibrary.js:63 Uncaught TypeError: Cannot read property 'toJS' of undefined
    at A (mediaLibrary.js:63)
    at index.js:9
    at bindActionCreators.js:3
    at t (withFileControl.js:131)
    at Object.<anonymous> (react-dom.production.min.js:15)
    at Object.invokeGuardedCallback (react-dom.production.min.js:16)
    at Object.invokeGuardedCallbackAndCatchFirstError (react-dom.production.min.js:16)
    at C (react-dom.production.min.js:20)
    at R (react-dom.production.min.js:22)
    at N (react-dom.production.min.js:22)

export function openMediaLibrary(payload = {}) {
  return (dispatch, getState) => {
    const state = getState();
    const mediaLibrary = state.mediaLibrary.get('externalLibrary');
    if (mediaLibrary) {
      const { controlID: id, value, config, forImage } = payload;
      mediaLibrary.show({ id, value, config: config.toJS(), imagesOnly: forImage }); // this line
    }
    dispatch({ type: MEDIA_LIBRARY_OPEN, payload });
  };
}

Uploadcare's widget isn't a media library - it doesn't show things you've already uploaded - so it returns false from it's enableStandalone method.

Despite the fact that Uploadcare is not a media library, we can store user-uploaded UUIDs in LocalStorage or globally in the repository as a json file. And provide those files with Custom Tab in the Uploadcare Widget, like EffectsTab. What do you think about it?

[owenhoskins]

@nd0ut: I was considering the following direction:

For my use-case the user needs the ability to select and add multiple images into something like a List widget. I am wondering how and if we can treat Uploadcare as a kind of assetStore to fetch files and pass them to the built-in MediaLibrary.
I've been diving into the Uploadcare docs and it seems feasible to use their REST API get-list to populate the built-in Media Library with infinite scrolling.
I am considering exploring this direction by hijacking the loadMedia action in a PR. Perhaps that ground-work could be abstracted into a API for treating services like Uploadcare as an asset store?
Or is there another tack?

Your suggestion may be a better tack indeed. Today I'll start looking into your suggestion to use the Custom Tab in the Uploadcare Widget and share my findings, as I need to provide a Media Library experience and 3rd party storage asap!

[owenhoskins]

Based on @nd0ut's suggestion I've followed the Uploadcare docs and set up a Custom Tab that passes in an array of UUID's via the config.yml for mock data. This approach works extremely well as a simple Media Library.

Currently vanilla Javascript appends the images to the uploadcare modal's DOM, see here.

However, features which we have in the built-in Media Library like pagination and search will be sorely missed, so I am considering if and how I could inject a React Component into the Uploadcare widget.

I am also wondering how to best populate the tab with real data. A JSON file checked in to the repo or a call to the Uploadcare API to fetch paginated lists of files.

@erquhart: I am curious to hear your thoughts and if you'd have any pointers on how to proceed!

[Benaiah]

@erquhart this looks great to me, though I am running into a similar error to that which @Ndout pasted when trying to insert an image in a markdown file (the standalone image control works fine). I would like to see an easier way to optionally show a react control in a modal so that doesn't have to be reimplemented by every media integration, but many media integrations already expect to show their own modal, so this seems like a great API to use and build from and think it's fine to merge without that.

Once the error with the markdown editor is fixed this LGTM.

[erquhart]

@owenhoskins still having a hard time envisioning that kind of interface working for sites with a lot of images like the one you're working on. Your editors would need to manually scroll through 5,000+ images to find the one they're looking for. Wouldn't you need a folder hierarchy, and possibly a nested one?

[owenhoskins]

Thanks for your consideration @erquhart! In this use-case, the file name carries an artist code which acts as a kind of folder system when searching by filename for that code. So far that's worked quiet well and recreating the filename search feature would fulfill this particular use-case.

Uploadcare's Group API is interesting but it is immutable and as such it doesn't seem to be a route to a folder system (or nested one).

If we went the route of saving UUIDs to a json file in source control we could create some kind of folder system based on which "post" the file was uploaded within, for example. Thoughts?

[nd0ut]

Uploadcare's Group API is absolutely not supposed to be used as some kind of folder hierarchy.

I like an idea to use json with UUIDs and metadata. This allows to describe the hierarchy in any needed form, linked with posts or with users, implement access-control and searching.

[dmitry-mukhin]

@owenhoskins
we're adding custom meta tags to files soon, you could try to use that to group files or create some sort of hierarchy

[erquhart]

Fixed the editor component issue - I'm considering this ready to merge, as the other improvements mentioned here can be added on afterward.

Going once, going twice?

Fixed

[ghost]

This isn't an out-of-the-box widget?

[erquhart]

It is, but it isn't released yet. Look for 2.1.0 soon.

_{Sent with GitHawk}

[ghost]

Ok, I thought I would have to stop using CDN links to use this.

[owenhoskins]

Hello all. (especially @erquhart & @nd0ut)

I was considering opening another issue or pull request to share my progress here but I thought I'd start with a follow up comment here.

I've integrated the Search component from the built-in Media Library into a Custom Tab in the Uploadcare widget. Currently I am calling the Uploadcare API to populate the components state, which enables search by filename and is fed into a react-virtualized implementation to lazy-load and infinite-scroll the list of images.

I also recognized that the netlify-cms-widget-image when used with the Uploadcare Preview tab makes for a very useable drag-and-drop gallery manager, nice!

Calling the Uploadcare API in this way requires passing both the public/private key in front-end code so this isn't a viable production option.

The next step would be storing UUIDs / metadata in JSON files but I am not sure how to proceed. Please advise!

[nd0ut]

@owenhoskins Hello! Nice work!

I just deal with the same problem, but on the other hand - I've started from storing JSON file index. This is done with persistMedia action which stores passed File to the store (git repo on production or local storage on dev). You can view my code here. It is working but completely not ready so I didn't create PR.

[nd0ut]

@owenhoskins can you make a PR with your code? I think we could collaborate on this.

[owenhoskins]

@nd0ut I took your branch for a spin -- it's so very satisfying to have the uploads persisted in a JSON file. Great work!

I stumbled a bit getting started because the local storage works as described but when I added a backend repo I encountered JS errors e.g. getState().mediaLibrary.get('files') at file-index.js:15 returning undefined.

After I protected against undefined in a few instances of the media library actions and reducers my the JSON file persisted to the repo.

So it looks like all calls to get('files') return undefined until there is actually a file at assets/uploads/uploadcare-index.json in the repo . Sound plausible?

Other then that I have encountered a few aspects which are probably what you were referring to when you described the branch as 'completely not ready'.

I would love to lend a hand and start integrating the UI that I've been working on. Do you have any suggestions for a communication channel/workflow to collaborate on this? Thanks!

[nd0ut]

@owenhoskins
I understood what the problem was with that state.medialLibrary.files is undefined. You just need to call loadMedia action to load media with provider, after that files will be initialised.

@owenhoskins I wrote to you in the gitter

[franva]

Could someone help?
I am trying to set up a free account with Uploadcare, i can see the website lists a free tier(https://uploadcare.com/pricing/) but when signing up, it only provides a 14 day trial. there is no other options.

[erezrokah]

@franva I suggest you contact Uploadcare support https://uploadcare.com/community/c/support/7

[optlsnd]

Hi @franva, I'm Alex from Uploadcare support. Sorry for the confusion. We're re-working our registration flow and have temporarily removed the option to choose the free plan on signup. We still offer this plan, though, and will be happy to provide you with it. Drop us a line at [email protected], and we'll set it up for you shortly.