Ipns/patches #1739

whyrusleeping · 2015-09-21T23:43:22Z

This PR contains three distinct commits that make ipns more reliable as a temporary fix until finalize the ipld records implementation.

Details on each commit provided in commit messages.

whyrusleeping · 2015-09-22T00:06:23Z

weird failures i'm fairly certain arent related: https://travis-ci.org/ipfs/go-ipfs/jobs/81498938#L247

jbenet · 2015-09-23T02:32:44Z

core/core.go

@@ -104,6 +105,7 @@ type IpfsNode struct {
 	Diagnostics  *diag.Diagnostics   // the diagnostics service
 	Ping         *ping.PingService
 	Reprovider   *rp.Reprovider // the value reprovider system
+	IpnsRepub    *ipnsrp.Republisher


We'll eventually want to organize this stuff cleaner. maybe bundle all the IPNS parts into one struct.

That makes sense. an IpfsNode does a lot

jbenet · 2015-09-23T02:38:46Z

needs some more tests. can manipulate the timeouts to test the various edge cases.

whyrusleeping · 2015-09-24T01:20:18Z

@jbenet what tests are you thinking of? About all I could come up with is to shorten the interval and check that the publish actually happened.

jbenet · 2015-09-24T03:30:21Z

check the publish happens
check republish happens
check records disappear after interval if (re)publisher goes down.
check records re-appear if publisher returns.

(what was the problem with iptb on travis?)

whyrusleeping · 2015-09-24T22:24:05Z

the issue with iptb was that it didnt use 'port zero' since it would have had to start up a daemon, get the address it chose from the output, and write that to the other nodes bootstrap configs. I still use iptb all the time locally

whyrusleeping · 2015-09-25T07:01:10Z

hey look! our favorite type of race/panic!

https://travis-ci.org/ipfs/go-ipfs/jobs/82099648

whyrusleeping · 2015-09-25T16:15:27Z

@jbenet I've added a test for the republisher

jbenet · 2015-09-25T17:25:01Z

@whyrusleeping durability concerns o/

whyrusleeping · 2015-09-25T18:41:50Z

@jbenet ping. There isnt anything to worry about here as far as durability goes. It the node goes down, we restart it and things continue where they left off.

jbenet · 2015-09-25T19:07:27Z

core/core.go

@@ -226,6 +228,12 @@ func (n *IpfsNode) startOnlineServicesWithHost(ctx context.Context, host p2phost
 	// setup name system
 	n.Namesys = namesys.NewNameSystem(n.Routing)

+	// setup ipns republishing
+	n.IpnsRepub = ipnsrp.NewRepublisher(n.Routing, n.Repo.Datastore(), n.Peerstore)
+	n.IpnsRepub.AddName(n.Identity)


ok so this is why we don't yet need durability o/ -- we provide durability elsewhere, and always load up the state ourselves.

jbenet · 2015-09-25T19:17:21Z

Feeling good about this. i think we should have some sharness and/or iptb tests around this. it's a large complicated system with lots of expectations.

whyrusleeping · 2015-09-25T23:31:38Z

@jbenet So, i fixed iptb so that it should work with port zero stuff. and then i realized that in order to be able to tweak the timeouts and intervals of the rebroadcaster through the cli, it would take a lot of reworking things... do you think thats worth the time investment?

jbenet · 2015-09-25T23:43:41Z

Could be a config option. Probably should be so we can play with it.

May also want to make it default to every 4hrs for now actually, because network is still small and churn. Prob confirm that it isn't less than every 5min

whyrusleeping · 2015-09-26T08:17:56Z

@jbenet what would we call this config option? I have the change to four hours in, and it shouldnt be difficult at all to grab a value from the config if it exists, just gotta decide on where it lives.

whyrusleeping · 2015-09-29T01:38:21Z

Things are looking as good as i think theyre gonna. The sharness test seems to fail occasionally. I'm guessing something takes a long time for no reason at all and the republisher fires. ping @jbenet

jbenet · 2015-09-29T07:40:32Z

@whyrusleeping feedback above o/

whyrusleeping · 2015-09-29T15:54:43Z

I think it hangs because of a potential bug in the dht query code. On a localhost connection like this, it should quickly exhaust all nodes in the network in a query, and stop the query when it gets there...

whyrusleeping · 2015-09-29T16:38:41Z

@jbenet alright, lookin good!

This commit adds a sequence number to the IpnsEntry protobuf that is used to determine which among a set of entries for the same key is the 'most correct'. GetValues has been added to the routing interface to retrieve a set of records from the dht, for the caller to select from. GetValue (singular) will call GetValues, select the 'best' record, and then update that record to peers we received outdated records from. This will help keep the dht consistent. License: MIT Signed-off-by: Jeromy <[email protected]>

Queries previously would sometimes only query three (alpha value) peers before halting the operation. This PR changes the number of peers grabbed from the routing table to start a query to K. Dht nodes would also not respond with enough peers, as per the kademlia paper, this has been changed to from 4 to 'K'. The query mechanism itself also was flawed in that it would pull all the peers it had yet to query out of the queue and 'start' the query for them. The concurrency rate limiting was done inside the 'queryPeer' method after the goroutine was spawned. This did not allow for peers receiver from query replies to be properly queried in order of distance. License: MIT Signed-off-by: Jeromy <[email protected]>

This commit adds a very basic process that will periodically go through a list of given ids and republish the values for their ipns entries. License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping · 2015-09-29T16:53:29Z

aaaand rebased.

whyrusleeping · 2015-09-30T02:05:03Z

@jbenet RFM

jbenet · 2015-09-30T09:28:26Z

jbenet · 2015-09-30T10:03:47Z

@whyrusleeping

License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping · 2015-09-30T18:14:41Z

kingpin is the flags library that iptb uses.

whyrusleeping · 2015-09-30T18:25:05Z

@jbenet addressed comments, and split up the vendoring commit.

jbenet · 2015-09-30T20:40:22Z

total ordering 3c7c165#commitcomment-13527896
"correct them" context cancel fix 3c7c165#commitcomment-13527951
bound resolution context with 3c7c165#commitcomment-13528001

License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping · 2015-10-01T05:53:26Z

did those things, @jbenet the things were done did.

jbenet · 2015-10-01T19:55:28Z

@whyrusleeping great! thanks! 👏 👏

I'm not 100% this will work perfectly, but let's merge it and see how it does?

Ipns/patches

Ipns/patches This commit was moved from ipfs/kubo@5b2d2eb

jbenet added the status/in-progress In progress label Sep 21, 2015

whyrusleeping force-pushed the ipns/patches branch from a08c8ed to 3b29e26 Compare September 21, 2015 23:57

whyrusleeping force-pushed the ipns/patches branch from 3b29e26 to 8f5c610 Compare September 22, 2015 00:50

whyrusleeping assigned jbenet Sep 22, 2015

whyrusleeping force-pushed the ipns/patches branch from 8f5c610 to 513bc6e Compare September 22, 2015 17:15

jbenet reviewed Sep 23, 2015
View reviewed changes

whyrusleeping force-pushed the ipns/patches branch from 513bc6e to 2d46667 Compare September 24, 2015 00:51

whyrusleeping force-pushed the ipns/patches branch from 2d46667 to 29304ab Compare September 25, 2015 06:55

jbenet reviewed Sep 25, 2015
View reviewed changes

whyrusleeping force-pushed the ipns/patches branch from 29304ab to a389f6d Compare September 25, 2015 19:17

whyrusleeping force-pushed the ipns/patches branch from b98e5f2 to f2d6e70 Compare September 29, 2015 01:17

whyrusleeping force-pushed the ipns/patches branch from f2d6e70 to 0de9680 Compare September 29, 2015 16:22

whyrusleeping added 3 commits September 29, 2015 09:47

Implement ipns republisher

815a031

This commit adds a very basic process that will periodically go through a list of given ids and republish the values for their ipns entries. License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping force-pushed the ipns/patches branch from 0de9680 to 41db338 Compare September 29, 2015 16:53

whyrusleeping added 2 commits September 30, 2015 11:03

vendor in iptb and kingpin

e202c65

License: MIT Signed-off-by: Jeromy <[email protected]>

make publish more configurable and add test for repub

e5512b4

License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping force-pushed the ipns/patches branch from 41db338 to 3c7c165 Compare September 30, 2015 18:14

ghost mentioned this pull request Sep 30, 2015

Sprint Sep 28 ipfs/team-mgmt#35

Closed

88 tasks

Addressing comments from CR

3b9ad30

License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping force-pushed the ipns/patches branch from 3c7c165 to 3b9ad30 Compare October 1, 2015 05:50

jbenet added a commit that referenced this pull request Oct 1, 2015

Merge pull request #1739 from ipfs/ipns/patches

5b2d2eb

Ipns/patches

jbenet merged commit 5b2d2eb into master Oct 1, 2015

jbenet removed the status/in-progress In progress label Oct 1, 2015

jbenet deleted the ipns/patches branch October 1, 2015 19:55

hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023

Merge pull request ipfs/kubo#1739 from ipfs/ipns/patches

10bbdad

Ipns/patches This commit was moved from ipfs/kubo@5b2d2eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ipns/patches #1739

Ipns/patches #1739

whyrusleeping commented Sep 21, 2015

whyrusleeping commented Sep 22, 2015

jbenet Sep 23, 2015

whyrusleeping Sep 23, 2015

jbenet commented Sep 23, 2015

whyrusleeping commented Sep 24, 2015

jbenet commented Sep 24, 2015

whyrusleeping commented Sep 24, 2015

whyrusleeping commented Sep 25, 2015

whyrusleeping commented Sep 25, 2015

jbenet commented Sep 25, 2015

whyrusleeping commented Sep 25, 2015

jbenet Sep 25, 2015

jbenet commented Sep 25, 2015

whyrusleeping commented Sep 25, 2015

jbenet commented Sep 25, 2015

whyrusleeping commented Sep 26, 2015

whyrusleeping commented Sep 29, 2015

jbenet commented Sep 29, 2015

whyrusleeping commented Sep 29, 2015

whyrusleeping commented Sep 29, 2015

whyrusleeping commented Sep 29, 2015

whyrusleeping commented Sep 30, 2015

jbenet commented Sep 30, 2015

jbenet commented Sep 30, 2015

whyrusleeping commented Sep 30, 2015

whyrusleeping commented Sep 30, 2015

jbenet commented Sep 30, 2015

whyrusleeping commented Oct 1, 2015

jbenet commented Oct 1, 2015

Ipns/patches #1739

Ipns/patches #1739

Conversation

whyrusleeping commented Sep 21, 2015

whyrusleeping commented Sep 22, 2015

jbenet Sep 23, 2015

Choose a reason for hiding this comment

whyrusleeping Sep 23, 2015

Choose a reason for hiding this comment

jbenet commented Sep 23, 2015

whyrusleeping commented Sep 24, 2015

jbenet commented Sep 24, 2015

whyrusleeping commented Sep 24, 2015

whyrusleeping commented Sep 25, 2015

whyrusleeping commented Sep 25, 2015

jbenet commented Sep 25, 2015

whyrusleeping commented Sep 25, 2015

jbenet Sep 25, 2015

Choose a reason for hiding this comment

jbenet commented Sep 25, 2015

whyrusleeping commented Sep 25, 2015

jbenet commented Sep 25, 2015

whyrusleeping commented Sep 26, 2015

whyrusleeping commented Sep 29, 2015

jbenet commented Sep 29, 2015

whyrusleeping commented Sep 29, 2015

whyrusleeping commented Sep 29, 2015

whyrusleeping commented Sep 29, 2015

whyrusleeping commented Sep 30, 2015

jbenet commented Sep 30, 2015

jbenet commented Sep 30, 2015

whyrusleeping commented Sep 30, 2015

whyrusleeping commented Sep 30, 2015

jbenet commented Sep 30, 2015

whyrusleeping commented Oct 1, 2015

jbenet commented Oct 1, 2015