Skip to content

Commit

Permalink
WIP: DOC: Document IPFS data upload
Browse files Browse the repository at this point in the history
  • Loading branch information
thewtex committed Nov 8, 2022
1 parent 913ede9 commit eded441
Show file tree
Hide file tree
Showing 8 changed files with 47 additions and 32 deletions.
Binary file added Documentation/Contribute/CloseDropFiles.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Documentation/Contribute/DropFiles.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Documentation/Contribute/LogInHighlighted.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

This file was deleted.

1 change: 0 additions & 1 deletion Documentation/Contribute/PublicFolderHighlighted.png.cid

This file was deleted.

76 changes: 47 additions & 29 deletions Documentation/Contribute/UploadBinaryData.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,31 +16,49 @@ A `solution to this problem`_ which has been adopted by this project is to store
binary files, such as images, in a separate location outside the Git repository,
then download the files at build time with CMake_.

A "content link" file contains an identifying `SHA512 hash`_. The content link
A "content link" file contains an identifying `Content Identifier (CID)`_. The content link
is stored in the Git_ repository at the path where the file would exist, but
with a ".sha512" extension appended to the file name. CMake will find these
with a ".cid" extension appended to the file name. CMake will find these
content link files at *build* time, download them from a list of server
resources, and create symlinks or copies of the original files at the
corresponding location in the *build tree*.

The `Content Identifier (CID)`_ is self-describing hash following the `multiformats `_ standard created by the Interplanetary Filesystem (`IPFS`_`) community.
A file with a CID for its filename is content verifable. Locating files according to their
CID makes content-addressed, as opposed to location-addressed, data exchange possible. This practice is the foundation of the decentralized web, also known as the dWeb or Web3. By adopting Web3, we gain:

- Permissionless data uploads
- Robust, redundant storage
- Local and peer-to-peer storage
- Scalability
- Sustainability

Contributors to the examples upload their data through an easy-to-use, permissionless, free service, `web3.storage`_.

Data used in the exmaples Git repository is periodically tracked in a `dedicated Datalad repository`_ and stored across redundant locations so it can be retrieved from any of the following:

- Local `IPFS`_ nodes
- Peer `IPFS`_ nodes
- `web3.storage`_
- `estuary.tech`_`
- `pinata.cloud`_
- Kitware's HTTP Server

Prerequisites
-------------

The `data.kitware.com`_ server is an ITK community resource where any
`web3.storage`_ server a decentralized IPFS storage provider where any ITK
community member can upload binary data files. There are two methods available
to upload data files:

1. The `Girder web interface`_.
2. The `girder-cli` command line executable that comes with the
girder-client_ Python package.

Before uploading data, please visit `data.kitware.com`_ and
register for an account.
1. The web3.storage web interface.
2. The `w3` command line executable that comes with the
`@web3-storage/w3` Node.js NPM package.

Once files have been uploaded to your account, they will be publicly
available and accessible since data is content addressed. At release time,
the release manager will upload and archive repository data references in the
`ITK collection`_ and other redundant storage locations.
available and accessible since data is content addressed on the IPFS peer-to-peer network.
At release time, the release manager will upload and archive repository data references
in other redundant storage locations.

Upload Via the Web Interface
----------------------------
Expand All @@ -51,29 +69,28 @@ Upload Via the Web Interface
:align: center
:width: 400

After logging in, you will be presented with the welcome page. Click on the
**personal data space** link.
First, log in with your GitHub or email.

.. figure:: PersonalDataSpaceHighlighted.png
:alt: Personal data space
.. figure:: UploadFiles.png
:alt: Upload files button
:align: center
:width: 400

Next, select the **Public** folder of your personal data space.
Next, select the **Upload files** button.

.. figure:: PublicFolderHighlighted.png
:alt: Public folder
.. figure:: DropFiles.png
:alt: File drag and drop
:align: center
:width: 400

Click the **green upload button**.
Click or drag and drop the file.

.. figure:: UploadHereHighlighted.png
:alt: The Upload files dialog
.. figure:: CloseDropFiles.png
:alt: Close the drop files dialog
:align: center
:width: 400

Click the **Browse or drop files** to select the files to upload.
Close the upload files dialog.

.. figure:: UploadHereFilesSelectedHighlighted.png
:alt: The Upload files dialog with files selected
Expand Down Expand Up @@ -198,10 +215,11 @@ your commit::

.. _CMake: https://cmake.org/
.. _Git: https://git-scm.com/
.. _Insight Community mailing list: https://itk.org/mailman/listinfo/community
.. _ITK collection: https://data.kitware.com/#collection/57b5c9e58d777f126827f5a1
.. _SHA512 hash: https://en.wikipedia.org/wiki/SHA-2
.. _data.kitware.com: https://data.kitware.com/
.. _Girder web interface: https://girder.readthedocs.io/en/latest/user-guide.html
.. _girder-client: https://girder.readthedocs.io/en/latest/python-client.html#the-command-line-interface
.. _solution to this problem: https://blog.kitware.com/cmake-externaldata-using-large-files-with-distributed-version-control/
.. _Content Identifier (CID): https://proto.school/anatomy-of-a-cid
.. _multiformats: https://multiformats.io/
.. _IPFS: https://ipfs.io/
.. _web3.storage: https://web3.storage/
.. _dedicated Datalad repository: https://github.com/InsightSoftwareConsortium/ITKSphinxExamplesData
.. _estuary.tech: https://estuary.tech
.. _pinata.cloud: https://pinata.cloud
Binary file added Documentation/Contribute/UploadFiles.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 0 additions & 1 deletion Documentation/Contribute/UploadHereHighlighted.png.cid

This file was deleted.

0 comments on commit eded441

Please sign in to comment.