Add Git-LFS support to the GitHub backend

[andytough]

Thank you for the Netlify CMS. I am experimenting with it at the moment and I am frequently delighted as I explore the way it works!

- Do you want to request a feature or report a bug?
Feature request

- What is the current behavior?
When using the Netlify CMS media browser for image selection it displays empty thumbnails

- If the current behavior is a bug, please provide the steps to reproduce.
When using the Netlify CMS for image selection the media browser normally displays thumbnails of uploaded images to select from.

When LFS (large file support) is enabled the media browser displays empty thumbnails.

- What is the expected behavior?
I would like Netlify CMS media_folder browser to display thumbnails when LFS is enabled for a repo

-Additional info
If viewing the url of one of the broken thumbnails it is the path to the 'text pointer' for the image file:

version https://git-lfs.github.com/spec/v1
oid sha256:e57917d40d1273633a8ce5d52e14c2534b85cfd289dc2e89a048a6b888eb214c
size 29860

Images that are placed on the page using this method do work correctly and are deployed successfully, it is only the media browser thumbnail that is broken.

Below is a picture showing what happens when the media_folder browser is used on an LFS image. As you can see in the background, an image that has been chosen renders correctly, but the thumbnails in the browser do not display.

The repo this example is taken from is at https://github.com/andytough/dev-andytough-website

[erquhart]

Could you give an example of the url we should be using in this case?

[andytough]

Thank you for your response. I have modified my post, removing some of the links, and editing the content, in the hope that it will make more sense.

[tech4him1]

@andytough Is there an API you know of that we can use to fetch the actual LFS data? I've been looking, but haven't found one yet.

[andytough]

@tech4him1 Sorry, I don't know of any. I am quite new to this!

[erquhart]

Yeah, took a look myself and there doesn't seem to be an API for LFS. Might be worth asking GitHub on Twitter or in a forum.

[andytough]

@erquhart Thank you. I will see what I can find. Shall I change this issue from Bug to Feature Request, as I think from your responses that what I am experiencing is expected behaviour?

[erquhart]

Yeah, adding LFS support would be a feature request.

[andytough]

I have changed the title of this issue, and amended the text so it reads more sensibly for a feature request. I will see if I can find anyone who can advise regarding an API.

[thom4parisot]

I +1 on this one as git-lfs can also be used to manage large assets, like PDFs, videos and such. I suspect these elements could have remote metadata if they are indexed in a documentstore somewhere (like Algolia, ElasticSearch, etc.)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[lookea]

Any news on github side on this ?

[erquhart]

We now support git lfs for Netlify Large Media, not certain what that does for lfs in general. We should at least be close, all of the code is there for working with pointer files.

_{Sent with GitHawk}

[lookea]

thanks @erquhart yes Netlify own storage works, but when trying to use github lfs, all commits when uploading images are not lfs references but the image in self (bytes), this should be able to be handle I believe easily within the cms code, and I will really appreciate if that is done (I do not see a way to workaround this in the configuration) I know that other stuff like having the previews working requires implementation and github http server details.

[lookea]

We now support git lfs for Netlify Large Media, not certain what that does for lfs in general. We should at least be close, all of the code is there for working with pointer files.

Sent with GitHawk

Hi @erquhart I can confirm netlify own lfs solution works, but there is a problem, only pages that are merge into master can render the image preview, when using editorial workflow when creating a draft all images are shown in the preview but after going out and back into it the images are then out of the cache and no longer rendered, they only show if you go from draft to publish, merge into master, I think this is some kind of issue with netlify-cms code because images are indeed available on the lfs

[erezrokah]

Linking #3106 as reference

[lookea]

What is the https://app_name.netlify.app/.netlify/large-media api call done ? that seems to work together with git-gateway auth, it has a /verify resource, and why it is different from the lfs service url that is something like https://XXXXXXXXXX.netlify.com/.netlify/large-media very similar structure. I don't see that webservice anywere on the github repository of netlify, and it doesn't follow the git lfs spec, so is something different

[erezrokah]

Hi @lookea, those calls are equivalent expect using a different domain name for the site. The API stays the same.
The way Git LFS works is by uploading files that match a certain patten to a 3rd party service (Netlify Large Media in this case) instead of saving them directly in the repo.
What's saved in the repo is a pointer file containing information that can be used to extract the file from the 3rd party service.
Netlify CMS detects if you're using Netlify Large Media and automatically uploads matching files to that service using dedicated API methods like /verify.
To support Git-LFS for GitHub, the GitHub API would need to provide similar API methods to facilitate that functionality.

[srchulo]

@erezrokah so even though GitHub supports LFS, it's missing some API call to upload media as LFS through Netlify CMS? Can't Netlify CMS add media files by committing using LFS?

[erezrokah]

Can't Netlify CMS add media files by committing using LFS?

That would require the CMS to have a JavaScript implementation of the git-lfs protocol and also has some challenges on using the OAuth token for such operations.

Of course I'm open to someone from the community looking into this.

[lookea]

I understand how it works so it is dependent on the oauth handling of git-gateway in order to support pushing the lfs files, anyway is there any reason why the code for the large-media endpoint to not be public available ? I think having that I can try to adapt it to git lfs, and work on the required cms js stuff side given to me it is only supporting netlify lfs only

[lookea]

@erezrokah so even though GitHub supports LFS, it's missing some API call to upload media as LFS through Netlify CMS? Can't Netlify CMS add media files by committing using LFS?

Is sad but the only direct way of communication happens with git backend implementation, and even there given how github works you require a middle man to get the oauth token, that is ok given it can easily be implemented, netlify provide it out of the box of course, but LFS is a little more complex, given you have more auth stuff to handle, netlify does this by using the git-gateway a more complete and complex solution that handles username/password authentication as a bridge to get github access. But yes the magic of having github as a database was to eliminate the need of a backend for a lot of small projects.

[erezrokah]

I think this discussion is getting a bit out of focus.

You don't need git-gateway to have LFS.

We recently had a community contributed PR to support LFS for Bitbucket - that was possible since Bitbucket exposes its LFS API.

Since GitHub doesn't expose an LFS API adding such an implementation is not straightforward when used directly.

A possible solution at the moment (not involving Netlify open sourcing Netlify Large Media service code or switching to Bitbucket) will be to write your own backend and hook into media related operations to write media files to any destination other than GitHub.

I agree that is not ideal, but I don't see any easy fix to this issue (unless API access is available).

Regardless, I don't think we want the core github backend to be tied to any LFS service that is not provided by github.

[naavis]

I guess this is not going to happen any time soon, given no one has commented in over a year?

[erezrokah]

Hi @naavis, unless GitHub added LFS support to their API I don't see a way forward.
Happy to accept a contribution for it if someone figures out a solution.

[JohannesFerner]

Please add to the docs of github backend, that LFS is not supported for now, as it's a pain to revert those changes in a live-repository which is actively maintained by content-editors.

[erezrokah]

Hi @JohannesFerner, we'd be happy to receive a docs contribution for it.
There's an Edit this page button one can use to submit a change.

[JohannesFerner]

Hi @JohannesFerner, we'd be happy to receive a docs contribution for it. There's an Edit this page button one can use to submit a change.

thanks @erezrokah for the hint - tried that now- doesnt work btw.

MD to add:

## Git Large File Storage (LFS)

Please note that the Github-Backend configuration is **not yet** supporting [git-lfs](https://git-lfs.github.com/), see [\#1206](https://github.com/netlify/netlify-cms/issues/1206) for details and status.

[erezrokah]

Thanks @JohannesFerner, not sure why it's not working. I created #6175 and will merge soon

[lukebouch]

To be clear, does this work with Netlify large media?

[erezrokah]

To be clear, does this work with Netlify large media?

Yes with the git-gateway backend. This issue is a feature request to add the same functionally for the github backend.

[JohnJMills]

Wondering about the status of this issue. Are there any plans to pick this up?

[jezcope]

Netlify Large Files has been deprecated since 2023 and cannot be added to new sites. It would be good to have an alternative!