Trustful IPFS Store #3727
Trustful IPFS Store #3727
Conversation
either /ipfs or /ipns URIs can be used to address a store. To change the default values (127.0.0.1, 5001) for the local API `host` and `port` are used in the query such as /ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn?host=192.168.42.1&port=1111 If a gateway should be used instead of a local daemon `use_gateway` is set to 1. If the another gateway should be used, `gateway` can be changed. In combination: /ipns/cache.example.com?gateway=https://gw.example.com
Build fails as:
$ make
CXX src/libmain/stack.o
src/libmain/stack.cc: In function 'void nix::sigsegvHandler(int, siginfo_t*, void*)':
src/libmain/stack.cc:21:21: error: 'ucontext' was not declared in this scope
sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
^~~~~~~~
src/libmain/stack.cc:21:21: note: suggested alternative: 'ucontext_t'
sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
^~~~~~~~
ucontext_t
It's caused by upstream rename:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=251287734e89a52da3db682a8241eb6bccc050c9
which basically changes
typedef struct ucontext {} ucontext_t;
to
typedef struct ucontext_t {} ucontext_t;
The change uses ucontext_t.
Signed-off-by: Sergei Trofimovich <[email protected]>
This adds a post bool that can be used to POST content instead of GET or PUT it.
need to use a multipart form to correctly insert the data. this mirrors how curl -Ffile=asdf works.
This makes a number of changes: - Use ipfs:// and ipns:// for URI, this is a proper URI and now documented by ipfs - Remove ipfs.hh, move everything into store - Store daemonUri, which is the, replacing individual API settings - Remove support for gateways - Remove nar info cache - ipfs should do all of the caching for us - Only retry getFile once - if it doesn’t work the ipfs daemon is probably down - Use /object/stat to check if file exists instead of cat
To check if IPFS daemon works, we need to do a test that should alway work. In the future, we may want to have a min IPFS daemon that we support, but for now any version will work.
This is okay for /ipns/ because we can always mutate it, but it’s a hard error for ipfs.
This is really slow, but is useful to test things. It does a few API calls to work: - /api/v0/add : to add the new file - /api/v0/object/patch/add-link : to insert the file into unixfs - /api/v0/name/publish : to publish the new ipfs object to ipns The last step is the slow part since we need to wait for it to clear.
unfortunately we don’t have a good way to handle this for now, avoid rethrowing and just accept 500
this doesn’t wait for the final response, but updates ipns internally.
To avoid race conditions while doing /object/patch/add-link & /name/publish, we need to lock some state. This adds a bool for inProgressUpsert so that we can know when other threads are also trying to publish to IPNS.
this is called at the end of a sequence of addToStores calls to tell the store that it can now commit the data.
Hopefully we can instead pass arguments like Hydra does, soon. (cherry picked from commit 4ced63c)
This uses the Sync primitive to store an ipfsPath that is written to as we add stuff to IPNS. IPNS is still optional. When finished, we do a publish on the ipnsPath.
…o ipfs-binary-cache-with-upload
… into ipfs-binary-cache-with-upload-sync
It isn't very safe, so best to do it once right after the condition. If only we had real pattern matching...
…load-sync Update ipfs binary cache with upload support (sync version)
…load Update ipfs binary cache with upload support
WIP: Get IPFS store working with StoreConfig refactor
|
Hello. Has this stalled? I see no new commits on ipfs-develop and these PRs have still not been merged |
|
That's rather a shame, I was excited for the hype on this to continue.. Is there a place where the final news was announced? I missed it, I think |
|
Sorry we haven't done a final blog or anything like that. Currently working though the details of content-addressed derivations is taking up the majority of my bandwidth with @edolstra. But if January has gone by and there is no further review of IPFS Nix PRs, do make a fuss --- I shouldn't have any excuses by then :). |
|
@bqv the other thing is that this PR in particular could be turned into a plugin, as the JSON code and whatnot is easy to move from where it is today. While the IPFS store could be plugin in all cases, the other branches require more meaningful extensions to the rest of the Nix in the new ways to content-address things. So in some sense this PR is only worth merging if we know the others will be, and I suppose I hadn't yet turned it into a a plugin as that felt like prematurely throwing in the towel, but maybe that is a bad call on my part. |
|
That might be cool. I wasn't singling out this PR though, don't worry, I just don't know where the bulk of the work is and whats been merged |
|
Relevant PRs are:
and as you yourself said, it is the https://github.com/obsidiansystems/nix/tree/ipfs-develop branch that puts everything together. |
|
I marked this as stale due to inactivity. → More info |
@Ericson2314 here I am, making a fuss :) What is going on in the IPFS+Nix world? |
|
I marked this as stale due to inactivity. → More info |
|
Probably best to keep this open until RFC. |
This is a rewritten https://github.com/NixIPFS/nix/tree/ipfs_binary_cache_legacy (thanks for pioneering!) that
Needless to say, this is a WIP PR so the the store format is subject to change, so if you try this, and then
git pull, then find you can't make use of the data you previous put in IPFS, don't be sad.