Add support for remote Pinning Services

[jacobheun]

Summary

The goal of this is to add a Pinning Services client in go-ipfs that can integration with Pinning Services that conform to the Pinning Services Spec. Users will be able to add Pinning Services to their IPFS configuration and then leverage those services while pinning.

Initial Use Case

The desired use case being fulfilled here is to enable IPFS Desktop / WebUI users to configure their local IPFS node to remotely manage pins with pinning services on their behalf.

Implementation Details

Pinning Services Spec: https://github.com/ipfs/pinning-services-api-spec
Spec Version: v1.0.0 (to be tagged after go-ipfs changes ship in 0.8.0)

Stages

The implementation may be implemented in a phased approach. This will allow for an iterative rollout of support.

1. Remote Pinning Support

This phase will implement the client and add support for manual pinning. This will enable users to specify the remote services for a single cid.

• Add support for Pinning.RemoteServices in the config feat: added support for pinning service configs go-ipfs-config#104
  • Pinning.RemoteServices[].Api.Key must be write only.
• Implement the Pinning Services client for the spec version mentioned above. Add support for remote Pinning Services #7559
• Update the pin API to accept a list of services to pin remotely Add support for remote Pinning Services #7559
  • Should be able to do this in the CLI and over the http API
  • All remote pins are recursive only
  • The pin should also occur locally
  • When Pins are created/modified go-ipfs should connect to the provider multiaddrs specified by the Pinning Service to accelerate transfer times.

2. MFS Pinning Policy

This phase adds support for pinning policies and introduces the "all in MFS" policy. Policies will be configurable per Pinning Service to allow for more control.

• Add support for watching changes to MFS and updating the pin for MFS root on the configured remote services add remote pinning policy for mfs #7798
  • When the MFS root changes, the client should make a modify pin request to the Pinning Service. Retention policies are dependent on the service and are not yet supported in the api
• Add support for Pinning.RemoteServices[].Policy in the config.
• Add the implicit manual(default) and all-in-mfs policies.
  • manual has the same behavior mentioned in phase 1.
  • all-in-mfs will watch for changes in MFS and update the pins for all Services with the policy

Related/References

• Pinning Services Spec
• IPFS/GUI Pinning Service Integration
• Doc: Pinning Services APIs for IPFS WebUI

[hsn10]

That is interesting idea.

It would be good to have user editable, at any time, comment for each pinning entry so you can see what exactly do you have remotely pinned and check if you still need it. I discovered while using 3rd party pinning services that this feature is critical for efficient organising of pins.

On other way, I discovered that pinning expiration (like 2 months) is not often needed because you can't in most times predict how long you will need to store data.

[jacobheun]

It would be good to have user editable, at any time, comment for each pinning entry so you can see what exactly do you have remotely pinned and check if you still need it

The basic pin commands will be supported, so you will be able to perform crud operations on the pins, including list pagination. You can see the spec linked in the description for the capabilities.

[lidel]

• This shipped in https://github.com/ipfs/go-ipfs/releases/v0.8.0 in form of ipfs pin remote --help
• The MFS policy, was added in add remote pinning policy for mfs #7798
• WebUI with GUI for remote pins will be added by chore: update webui to 2.12.2 #8097 which will close this issue.

🥳