Update to ipfs v0.5.0 #170

ianopolous · 2020-07-29T01:23:50Z

This fixes all the breaking changes in ipfs v0.5.0.

As well as upgrading dependencies.

@lidel Does this qualify for the two bounties?

(use POST instead of GET)

(pin update switched to multihash rather than multiaddr return)

src/main/java/io/ipfs/api/IPFS.java

.github/workflows/ant.yml

install-run-ipfs.sh

lidel

Thank you @ianopolous, yes, iiuc this ticks all the boxes from #157:

Support for POST-only HTTP API from go-ipfs 0.5+
Support for CIDv1 in Base32 and Base36
Fix tests and CI

Before we merge and close the bounty, some small questions/asks inline.

ps. just to have another set of eyes familiar with Java,
@NadeeraM are you able to check if this branch works for you too?

src/main/java/io/ipfs/api/IPFS.java

.github/workflows/ant.yml

install-run-ipfs.sh

Co-authored-by: Marcin Rataj <[email protected]>

ianopolous · 2020-07-29T22:13:02Z

That looks like maven is still using the old dependencies. I don't actually use maven myself, but I'll have a look.

ianopolous · 2020-07-29T22:48:34Z

@NadeeraM I've just fixed the maven file I think. Are you able to try again?

NadeeraM · 2020-07-29T22:56:27Z

@NadeeraM I've just fixed the maven file I think. Are you able to try again?

@ianopolous yes I will check

ianopolous · 2020-07-30T08:38:07Z

@NadeeraM jitpack.io had got itself into a strange state, but I've bumped a few versions in the dependencies for maven and it is working now.

NadeeraM · 2020-07-30T22:03:14Z

@ianopolous yes can compile the code successfully. But Still tests are failing. This is the tests that fail

    public void pubsubSynchronous() throws Exception {
        String topic = "topic" + System.nanoTime();
        List<Map<String, Object>> res = Collections.synchronizedList(new ArrayList<>());
        new Thread(() -> {
            try {
                ipfs.pubsub.sub(topic, res::add, t -> t.printStackTrace());
            } catch (IOException e) {
                throw new RuntimeException(e);}
        }).start();```


Stack trace Error:

```pub => sub took 1
child1: zb2rhXvZfBVfxehiE7VfgL5LbJpUXjkY64bms6hVqpE86zEff
root1: zdpuAte25GgAQNVY9pgWHTj1a33wmWVFmyhLzELamTwAq7eBS
java.net.SocketTimeoutException: Read timed out
        at java.base/java.net.SocketInputStream.socketRead0(Native Method)
        at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
        at java.base/sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552)
        at java.base/sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
        at java.base/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
        at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505)
        at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3498)
        at io.ipfs.api.IPFS.getObjectStream(IPFS.java:743)
        at io.ipfs.api.IPFS.retrieveAndParseStream(IPFS.java:682)
        at io.ipfs.api.IPFS.access$400(IPFS.java:15)
        at io.ipfs.api.IPFS$Pubsub.sub(IPFS.java:298)
        at io.ipfs.api.APITest.lambda$pubsubSynchronous$6(APITest.java:416)
        at java.base/java.lang.Thread.run(Thread.java:834)

Tests run: 39, Failures: 0, Errors: 2, Skipped: 2, Time elapsed: 243.865 sec <<< FAILURE! - in io.ipfs.api.APITest
pinUpdate(io.ipfs.api.APITest)  Time elapsed: 0 sec  <<< ERROR!
java.lang.IllegalStateException: Unknown Multibase type: /
        at io.ipfs.api.APITest.pinUpdate(APITest.java:253)

rawLeafNodePinUpdate(io.ipfs.api.APITest)  Time elapsed: 0.021 sec  <<< ERROR!
java.lang.IllegalStateException: Unknown Multibase type: /
        at io.ipfs.api.APITest.rawLeafNodePinUpdate(APITest.java:287)


Results :

Tests in error: 
  APITest.pinUpdate:253 » IllegalState Unknown Multibase type: /
  APITest.rawLeafNodePinUpdate:287 » IllegalState Unknown Multibase type: /
  AddTest.add:37 IllegalState Error returned from IPFS: {"Name":"html/chap/ch01....

Tests run: 50, Failures: 0, Errors: 3, Skipped: 2

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:09 min
[INFO] Finished at: 2020-07-31T09:48:55+12:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project java-ipfs-http-client: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/nadeera/Desktop/Work/ipfs/java-ipfs-http-client/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

ianopolous · 2020-07-30T22:19:36Z

The tests all run fine in CI, and if I run them locally using maven against ipfs v0.5.0 then they all pass:

Tests run: 50, Failures: 0, Errors: 0, Skipped: 2

The errors you're getting, @NadeeraM look like your running against an older version of ipfs, 0.4.23 or earlier (when pin update returned multiaddr rather than multihashes). Could you confirm the ipfs version you are testing against?

NadeeraM · 2020-07-30T23:29:58Z

The tests all run fine in CI, and if I run them locally using maven against ipfs v0.5.0 then they all pass:

Tests run: 50, Failures: 0, Errors: 0, Skipped: 2

The errors you're getting, @NadeeraM look like your running against an older version of ipfs, 0.4.23 or earlier (when pin update returned multiaddr rather than multihashes). Could you confirm the ipfs version you are testing against?

@ianopolous Yes i am running local ipfs using docker-compose file in this project. Version is "Version": "0.4.16".
Hope better to update that file version as well. :)

java-ipfs-http-client/docker-compose.yml

Line 4 in fc82516

image: 'ipfs/go-ipfs:v0.4.16'

Thank you

ianopolous · 2020-07-31T06:51:18Z

@lidel Any other comments on this? I think it's good to go now.

lidel

@ianopolous I noticed tests did not run against bundled junit and cid.jar (?) and after adding both in f092411 CI no longer passed (example).

I've re-run the job and it is now green – any idea why previous run failed? Anything we can do to make tests more deterministic, or at least tell us what exactly failed?

Instead use 95% delivery of messages

ianopolous · 2020-07-31T12:44:52Z

@ianopolous I noticed tests did not run against bundled junit and cid.jar (?) and after adding both in f092411 CI no longer passed (example).

It includes the cid jar via the ipfs.jar manifest.

I've re-run the job and it is now green – any idea why previous run was flaky?

Some of the tests are not strict unit tests because they impose a time limit and are thus more of a performance test of ipfs, like the pubsubSynchronous test. I've rewritten that to ignore the time now and instead require 95% delivery of messages (ipfs pubsub isn't reliable even on the same instance it seems).

lidel · 2020-07-31T15:29:23Z

Thank you @ianopolous, expecting 95% delivery of messages should work better than time-based one 👍

👉 I've confirmed this PR works for me locally and satisfies requirements of both bounties.
I believe this closes the double bounty listed in #157 – please merge this PR and make a new release of this library 🙏

We will finalize bounty details in #157

ianopolous added 12 commits July 29, 2020 00:42

Update to fix breaking change in ipfs v0.5.0

3f12b3a

(use POST instead of GET)

Update to fix breaking change in ipfs v0.5.0

710c4ed

(pin update switched to multihash rather than multiaddr return)

Fix test

e69db95

Update java and ipfs in CI

cfe967e

Fix object call which no longer tolerates other fields

65ddb42

Attempt to use github CI

f83a495

Fix github CI?

7c2c60c

make script executable

62a7348

Update test

be119b8

print test errors in CI

cd8134e

Fix test in CI environment

2cf8266

Remove travis

b40bcb1

odisseus reviewed Jul 29, 2020

View reviewed changes

src/main/java/io/ipfs/api/IPFS.java Outdated Show resolved Hide resolved

odisseus reviewed Jul 29, 2020

View reviewed changes

.github/workflows/ant.yml Show resolved Hide resolved

odisseus reviewed Jul 29, 2020

View reviewed changes

install-run-ipfs.sh Outdated Show resolved Hide resolved

ianopolous added 4 commits July 29, 2020 08:11

Expand tests

5a6a978

Upgrade to ipfs v0.6.0 in CI

fe6db8c

Add block.rm command

a992fe6

Ignore ipfs timeouts in publish

86650be

This was referenced Jul 29, 2020

Use the fresh versions of dependencies #165

Closed

feat: support go-ipfs 0.5 and Java 10+ #164

Closed

lidel requested changes Jul 29, 2020

View reviewed changes

src/main/java/io/ipfs/api/IPFS.java Outdated Show resolved Hide resolved

src/main/java/io/ipfs/api/IPFS.java Show resolved Hide resolved

.github/workflows/ant.yml Show resolved Hide resolved

install-run-ipfs.sh Outdated Show resolved Hide resolved

ianopolous and others added 2 commits July 29, 2020 16:07

use test config for ipfs

43b8890

Co-authored-by: Marcin Rataj <[email protected]>

Make timeouts configurable

51683d5

This comment has been minimized.

Sign in to view

fix maven dependencies

4371bbd

This comment has been minimized.

Sign in to view

bump multiaddr version to fix maven

fc82516

update docker compose ipfs version

74062b0

fix: missing deps during tests

f092411

lidel requested changes Jul 31, 2020

View reviewed changes

Remove max time on pubsub test.

676ed5b

Instead use 95% delivery of messages

ianopolous added 2 commits July 31, 2020 14:05

Remove spurious classpath entries for tests

fce310d

Make sure names of failing tests are printed to github console

e909b6e

lidel mentioned this pull request Jul 31, 2020

[BOUNTY] Support for go-ipfs v0.5+ #157

Closed

2 tasks

ianopolous merged commit 57269c7 into master Jul 31, 2020

lidel deleted the fix/ipfs-v0.5.0 branch August 18, 2020 15:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to ipfs v0.5.0 #170

Update to ipfs v0.5.0 #170

ianopolous commented Jul 29, 2020 •

edited by lidel

Loading

lidel left a comment

This comment has been minimized.

ianopolous commented Jul 29, 2020

ianopolous commented Jul 29, 2020

NadeeraM commented Jul 29, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

ianopolous commented Jul 30, 2020

NadeeraM commented Jul 30, 2020 •

edited

Loading

ianopolous commented Jul 30, 2020

NadeeraM commented Jul 30, 2020 •

edited

Loading

ianopolous commented Jul 31, 2020

lidel left a comment •

edited

Loading

ianopolous commented Jul 31, 2020

lidel commented Jul 31, 2020 •

edited

Loading

Update to ipfs v0.5.0 #170

Update to ipfs v0.5.0 #170

Conversation

ianopolous commented Jul 29, 2020 • edited by lidel Loading

lidel left a comment

Choose a reason for hiding this comment

This comment has been minimized.

ianopolous commented Jul 29, 2020

ianopolous commented Jul 29, 2020

NadeeraM commented Jul 29, 2020 • edited Loading

This comment has been minimized.

This comment has been minimized.

ianopolous commented Jul 30, 2020

NadeeraM commented Jul 30, 2020 • edited Loading

ianopolous commented Jul 30, 2020

NadeeraM commented Jul 30, 2020 • edited Loading

ianopolous commented Jul 31, 2020

lidel left a comment • edited Loading

Choose a reason for hiding this comment

ianopolous commented Jul 31, 2020

lidel commented Jul 31, 2020 • edited Loading

ianopolous commented Jul 29, 2020 •

edited by lidel

Loading

NadeeraM commented Jul 29, 2020 •

edited

Loading

NadeeraM commented Jul 30, 2020 •

edited

Loading

NadeeraM commented Jul 30, 2020 •

edited

Loading

lidel left a comment •

edited

Loading

lidel commented Jul 31, 2020 •

edited

Loading