Support for TruffleRuby

[ioquatix]

Following on from oracle/truffleruby#1436 ...

We've migrated timers to us Process.clock_gettime. There might be other issues too.

You can run a large suite of tests by checking out the async source code and running:

rake external.

It will run specs for the following code bases:

async/Rakefile

Lines 25 to 29 in de63b27

	clone_and_test("async-io")
	clone_and_test("async-websocket")
	clone_and_test("async-dns")
	clone_and_test("async-http")
	clone_and_test("falcon")

cc @jjyr @eregon

[ioquatix]

Also, I'd like to add support for travis testing across the entire eco-system, but we are currently blocking on travis-ci/travis-ci#9803

[deepj]

I've tried to run a simple rack app on Falcon, but it failed on

D, [2018-11-06T08:24:09.330000 #22009] DEBUG -- Async::IO::Socket: Binding to #<Addrinfo: 127.0.0.1:9292 TCP>
D, [2018-11-06T08:24:09.355000 #22009] DEBUG -- Async::IO::Socket: Binding to #<Addrinfo: [::1]:9292 TCP>
D, [2018-11-06T08:24:09.370000 #22009] DEBUG -- <Async::Reactor:0x6ee stopped=false>: Exiting run-loop because finished.
~/.rubies/truffleruby-1.0.0-rc9/lib/truffle/truffle/cext.rb:221:in `execute_without_conversion': Cannot convert LLVMTypedForeignObject+0x0 to LLVMNativePointer (IllegalStateException) (RuntimeError)
	from java.lang.Throwable.<init>(Throwable.java:287)
	from java.lang.Exception.<init>(Exception.java:84)
	from java.lang.RuntimeException.<init>(RuntimeException.java:80)
	from java.lang.IllegalStateException.<init>(IllegalStateException.java:75)
	from com.oracle.truffle.llvm.runtime.nodes.api.LLVMToNativeNode$LLVMObjectToNativeNode.transitionToNative(LLVMToNativeNode.java:117)
	from com.oracle.truffle.llvm.runtime.nodes.api.LLVMToNativeNodeGen$LLVMObjectToNativeNodeGen.executeAndSpecialize(LLVMToNativeNodeGen.java:301)
	from com.oracle.truffle.llvm.runtime.nodes.api.LLVMToNativeNodeGen$LLVMObjectToNativeNodeGen.executeWithTarget(LLVMToNativeNodeGen.java:201)
	from com.oracle.truffle.llvm.runtime.nodes.api.LLVMToNativeNode.doOther(LLVMToNativeNode.java:78)
	from com.oracle.truffle.llvm.runtime.nodes.api.LLVMToNativeNodeGen.executeAndSpecialize(LLVMToNativeNodeGen.java:129)
	from com.oracle.truffle.llvm.runtime.nodes.api.LLVMToNativeNodeGen.executeWithTarget(LLVMToNativeNodeGen.java:50)
Caused by:
Message not supported: TO_NATIVE (UnsupportedMessageException)
	from java.lang.Throwable.<init>(Throwable.java:265)
	from java.lang.Exception.<init>(Exception.java:66)
	from com.oracle.truffle.api.interop.InteropException.<init>(InteropException.java:52)
	from com.oracle.truffle.api.interop.UnsupportedMessageException.<init>(UnsupportedMessageException.java:57)
	from com.oracle.truffle.api.interop.UnsupportedMessageException.raise(UnsupportedMessageException.java:84)
	from org.truffleruby.interop.RubyMessageResolution$ForeignToNativeNode.access(RubyMessageResolution.java:190)
	from org.truffleruby.interop.RubyMessageResolutionForeign$ForeignToNativeSubNode.accessWithTarget(RubyMessageResolutionForeign.java:503)
	from org.truffleruby.interop.RubyMessageResolutionForeignFactory$ForeignToNativeSubNodeGen.executeAndSpecialize(RubyMessageResolutionForeignFactory.java:602)
	from org.truffleruby.interop.RubyMessageResolutionForeignFactory$ForeignToNativeSubNodeGen.executeWithTarget(RubyMessageResolutionForeignFactory.java:588)
	from org.truffleruby.interop.RubyMessageResolutionForeign$ForeignToNativeSubNode$TO_NATIVERootNode.execute(RubyMessageResolutionForeign.java:520)
Translated to internal error
	from ~/.rubies/truffleruby-1.0.0-rc9/lib/truffle/truffle/cext.rb:221:in `execute_with_mutex'
	from ~/.rubies/truffleruby-1.0.0-rc9/lib/truffle/truffle/cext_ruby.rb:36:in `close'
	from ~/.gem/truffleruby/2.4.4/gems/async-1.10.3/lib/async/reactor.rb:208:in `close'
	from ~/.gem/truffleruby/2.4.4/gems/async-1.10.3/lib/async/reactor.rb:56:in `run'
	from ~/.gem/truffleruby/2.4.4/gems/falcon-0.19.2/lib/falcon/command/serve.rb:97:in `run'
	from ~/.gem/truffleruby/2.4.4/gems/falcon-0.19.2/lib/falcon/command/serve.rb:124:in `invoke'
	from ~/.gem/truffleruby/2.4.4/gems/falcon-0.19.2/lib/falcon/command.rb:70:in `invoke'
	from ~/.gem/truffleruby/2.4.4/gems/falcon-0.19.2/bin/falcon:26:in `<top (required)>'
	from ~/.gem/truffleruby/2.4.4/bin/falcon:23:in `load'
	from ~/.gem/truffleruby/2.4.4/bin/falcon:23:in `load'
	from ~/.gem/truffleruby/2.4.4/bin/falcon:23:in `<main>'

To reproduce:

run lambda {|env| [200, {}, ["Hello World"]]}

Then run it:

falcon --verbose serve -c the-script-above.rb

@eregon it ended up with this issue oracle/truffleruby#1441

[ioquatix]

What can I do to help resolve this issue? @eregon

[jjyr]

All specs run well on my machine after upgrade timers to 4.2.0, I'll make a PR after the Travis issue resolved.

[ioquatix]

I installed truffle ruby 1.0.0rc9 and tried to run specs for async on my laptop. Some pass, some fail, ran into the same Message not supported: TO_NATIVE (UnsupportedMessageException) error above.

[eregon]

@deepj Could you report as a separate issue on the TruffleRuby bug tracker, to make it easier to track?

That TO_NATIVE error means a C extension is storing a Ruby object in native memory.
I guess that could be nio4r, which gets installed when installing falcon.
We'll need to dig deeper to see the actual place and see what can be done about it.
We're also currently investigating a general solution for this TO_NATIVE issues.

[eregon]

nio4r seems to have a pure-Ruby backend https://github.com/socketry/nio4r#supported-backends
That should work out of the box.
It looks like it can be used by setting NIO4R_PURE=true: https://github.com/socketry/nio4r/blob/e84f25ede87dcba8c52c9b86aa9bfbfbff0d9a59/lib/nio.rb#L17

@ioquatix What's the trade-off of using the pure-Ruby backend instead of the C extension? Should TruffleRuby try to support the C extension or there is no much advantage and just using the pure-Ruby backend is fine?

[deepj]

@ioquatix With NIO4R_PURE=true I can move futher. But falcon won't be working since it uses a forking model. So it stopped due this.

[ioquatix]

Can we make an extension to nio4r?

[ioquatix]

It already supports JRuby natively

[ioquatix]

You can start falcon using threaded model too.

[deepj]

@ioquatix How? I couldn't find anything about like that option in Falcon's README.md

[ioquatix]

—help and —threaded

[deepj]

I could move, it seems no issue from async gems perspective so far but I got this
socketry/falcon#33

[ioquatix]

Okay, it's working, I just release falcon v0.19.3

[ioquatix]

So, even though basic site is working, there are tons of issues.

• ObjectSpace.trace_object_allocations appears to cause infinite recursion oracle/truffleruby#1456
• write_nonblock doesn't support exception: false oracle/truffleruby#1457
• Re-raise exception is missing backtrace oracle/truffleruby#1459
• Socket#setsockopt broken with nil or false value. oracle/truffleruby#1460
• Socket#connect_nonblock working? oracle/truffleruby#1461

Most of these issues are relating to async-io.

I can't make progress until these issues are resolved.

[eregon]

@ioquatix Thank you for trying and filing the issues!
They seem relatively small problems, and so should be easy to fix :)

[ioquatix]

Please fix them and I will try again @eregon :)

[ioquatix]

@eregon I was just following up on this and noticed travis is broken again.

https://travis-ci.org/socketry/async/jobs/473835428

Is this something we can fix?

[eregon]

It's caused by https://travis-ci.community/t/new-gpg-key-for-rvm-releases/1521.
It's an issue between RVM and TravisCI, I cannot do much about it unfortunately.

[ioquatix]

rvm/rvm#4561 is tracking the issue on RVM.

[ioquatix]

Currently TruffleRuby is mostly failing because of #37

[ioquatix]

All tests are now passing with TruffleRuby.

[picatz]

That's awesome @ioquatix 👏