tracking: Building an application that uses `Blink.jl` #5

NHDaly · 2018-04-23T21:45:57Z

I'm pulling out this issue to track our attempt to build and distribute an App that uses Blink.jl.

Things we still need to solve:

Activity Monitor doesn't know the name of the App, so the program can be opened a second time (which hangs) (tracking: Building an application that uses Blink.jl #5 (comment))
JULIA_HOME not defined (juliac fails to compile Blink.jl: JULIA_HOME not defined JuliaLang/PackageCompiler.jl#47) (Fixed: Fix JULIA_HOME manually for all builds. JuliaLang/PackageCompiler.jl#55)
Blink.resources doesn't seem to change (Generalisable package? #1 (comment)) (Fixed: tracking: Building an application that uses Blink.jl #5 (comment))
Find all absolute paths that must be changed to relative paths. (Fixed: tracking: Building an application that uses Blink.jl #5 (comment))
change_dir_if_bundle() doesn't work when overriding JULIA_HOME via above (Fixed: change_dir_if_bundle() can't find binary path if JULIA_HOME is overriden #4)

The text was updated successfully, but these errors were encountered:

NHDaly · 2018-04-23T21:55:11Z

@ranjanan: Regarding your comment about Blink.resources, I'm also seeing an error that's probably the same thing. It looks like the problem I'm running into now is this call to @init:
https://github.com/JunoLab/Blink.jl/blob/v0.6.2/src/content/content.jl#L54-L56

Is this what you're seeing, too?:

fatal: error thrown and no exception handler available.
Base.InitError(mod=:Blink, error=Base.MethodError(f=typeof(Base.Filesystem.joinpath)(), args=(nothing, "..", "..", "res", "blink.js"), world=0x0000000000005
f00))
rec_backtrace at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_throw at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_method_error_bare at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_method_error at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_lookup_generic_ at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_apply_generic at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_f__apply at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
resolve at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
macro expansion at /Users/daly/.julia/v0.6/Blink/src/content/content.jl:55 [inlined]
#23 at /Users/daly/.julia/v0.6/Lazy/src/macros.jl:338
unknown function (ip: 0x11d276a6f)
__init__ at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jlcall___init___6545 at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jl_module_run_initializer at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
_julia_init at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
julia_init at /Users/daly/src/build-jl-app-bundle.bak/builddir/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
main at /Users/daly/src/build-jl-app-bundle.bak/./builddir/HelloBlink.app/Contents/MacOS/blink (unknown line)

NHDaly · 2018-04-24T02:40:53Z

Okay!! I think i've fixed most of the problems! I've just uploaded examples/Blink.jl, which builds into a working app!

Here is the whole command I used to build the app (including signing it so I could share it here):

$ julia ~/.julia/v0.6/ApplicationBuilder/build_app.jl \
   -R ~/.julia/v0.6/Blink/deps/Julia.app \
   -R ~/.julia/v0.6/Blink/src/AtomShell/main.js \
   -R ~/.julia/v0.6/Blink/src/content/main.html \
   -R ~/.julia/v0.6/Blink/res \
   -L ~/.julia/v0.6/HttpParser/deps/usr/lib/libhttp_parser.dylib \
   -L ~/.julia/v0.6/MbedTLS/deps/usr/lib/libmbedcrypto.2.7.1.dylib \
   --bundle_identifier "com.nhdalyMadeThis.HelloBlink" --app_version=0.1 --certificate "Developer ID Application: nhdalyMadeThis, LLC" \
   examples/blink.jl "HelloBlink"  &&  ./builddir/HelloBlink.app/Contents/MacOS/blink

For JULIA_HOME, I'm currently using Fix JULIA_HOME manually for all builds. JuliaLang/PackageCompiler.jl#55.
I think I fixed Blink.resources via the following things:
- -R ~/.julia/v0.6/Blink/res copied the entire res directory to .app/Contents/Resources/res.
- In order to fix the above InitError I was seeing, I had to delete the __init__ callback created by @init lines inside Blink.jl (this for loop in content.jl). I did that by empty!ing the __inits__ array here:
  https://github.com/NHDaly/ApplicationBuilder.jl/blob/2f5bf16/examples/blink.jl#L34-L36
  But that also meant that I deleted this other callback setting port in config.jl.
  
  So I had to put that line back after clearing the __inits__:
  https://github.com/NHDaly/ApplicationBuilder.jl/blob/2f5bf16/examples/blink.jl#L33
  
  Although I just realized that this means it's now picking a port at compile-time, instead of every time it opens at runtime... Maybe there could've been a smarter way to do that, clearing out just the offending @init line? (Just figure out which of the two is the bad one and pop it out.)

And I think I found all the absolute paths that needed to be fixed! Here are the compile-time modifications I used (examples/blink.jl#L24-L42):

if get(ENV, "COMPILING_APPLE_BUNDLE", "false") == "true"
    println("Overriding Blink dependency paths.")
    eval(Blink.AtomShell, :(_electron = "Julia.app/Contents/MacOS/Julia"))
    eval(Blink.AtomShell, :(mainjs = "main.js"))
    eval(Blink, :(buzz = "main.html"))
    eval(Blink, :(resources = Dict("spinner.css" => "res/spinner.css",
                             "blink.js" => "res/blink.js",
                             "blink.css" => "res/blink.css",
                             "reset.css" => "res/reset.css")))
    eval(Blink, :(const port = get(ENV, "BLINK_PORT", rand(2_000:10_000))))
    # Clear out Blink.__inits__, since it will attempt to evaluate hardcoded paths.
    # (We've defined all the variables manually, above: `resources` and `port`.)
    eval(Blink, :(empty!(__inits__)))

    eval(HttpParser, :(lib = "libhttp_parser.dylib"))
    eval(MbedTLS, :(const libmbedcrypto = "libmbedcrypto.2.7.1.dylib"))

    println("Done changing dependencies.")
end

NHDaly · 2018-04-24T02:46:13Z

So that's good news! The one remaining problem I'm having is that the OS seems to forget the application is open, so if you double-click it again while it's already open, a second version of the app will open and then hang forever. It even refused to Force Quit (I had to restart to get it to quit). :(

I'm assuming it's something to do with the fact that the application opens the embedded Julia.app after starting, and it's then running two Mac Applications, and this somehow bothers the OS.

Interestingly, it shows up in Activity Monitor as two different apps: blink and Julia, neither of which are its actual name, HelloBlink.app. The other test apps all show up correctly with their app bundle name. blink is the name of the executable. So that's something to look into...

NHDaly · 2018-04-24T03:10:35Z

So actually, reading a bit more, I wonder if the right way to do this for apps using Blink (ie, apps using Electron) is to utilize the already existing electron app building tools? For example:
https://github.com/electron-userland/electron-builder
https://medium.com/@flaqueEau/releasing-an-electron-app-on-the-mac-app-store-c32dfcd9c2bd

That is, perhaps instead of embedding a prebuilt Electron app inside our application, maybe we should be building a new Electron app which contains our compiled code?

Figuring that out seems like a fair amount of work, though....

ranjanan · 2018-04-24T06:25:15Z

@NHDaly thank you for all your work and progress. Yes, that is indeed what I was referring to when I said I ran into another problem with Blink: @init was initialising the paths at run time, so I kind of just removed @init so it basically initialised the resources dict at compile time. One other option I am considering playing with is changing the paths at run time instead of compile time like we are currently doing. Anyhoo, I managed to build my Blink app by just removing the @init.

I have made some progress on Windows, and I have a Blink app that works whenever you double click on the blink.exe. I have some code as well for bundling it all together. Would you mind if I sent a PR? The PR would also involve reorganising build_app.jl as a Julia package instead of a command line utility. Other than that, the windows version will also have the option to create an installer, which I think most people will like.

As for the electron builder stuff, that's a complete tangent. I think people only care about having their Julia code be packaged and shipped in as easy a manner as possible, and I think what we have now for Blink is good enough.

More soon, along with my PR!

NHDaly · 2018-04-24T06:37:34Z

:D That all sounds excellent! Thanks for your help! I'm excited to see a windows build!!

Anyhoo, I managed to build my Blink app by just removing the @init.

Great! I'm glad we got that figured out.

Would you mind if I sent a PR?

!! Yes please! :D

The PR would also involve reorganising build_app.jl as a Julia package instead of a command line utility.

Yeah i think that's a good idea. I've been thinking it would make sense to follow the format of https://github.com/JuliaLang/PackageCompiler.jl, where all the functionality is in src/static_julia.jl, but they still provide a command line utility which simply does arg-parsing and then calls the library: juliac.jl. But we can talk more about specific details if you send a PR over! :)

NHDaly · 2018-04-24T06:37:48Z

As for the electron builder stuff, that's a complete tangent. I think people only care about having their Julia code be packaged and shipped in as easy a manner as possible, and I think what we have now for Blink is good enough.

sounds good!

NHDaly · 2018-08-04T19:20:56Z

Hey, @ranjanan.

Are you still able to statically compile a Blink application using this package? I've found that when I try now, I can't get it to work. See this branch for the failing test:
https://github.com/NHDaly/ApplicationBuilder.jl/tree/blink_bundled
Specifically, here:

ApplicationBuilder.jl/test/BuildApp.jl

Line 65 in c1e5a54

    
           @test testRunAndKillProgramSucceeds(`$builddir/HelloBlink.app/Contents/MacOS/blink`)

I'm testing that it is in fact working without julia installed -- and sadly, it isn't. I am currently planning on using a Blink example in my talk, but I'm sad that I can't show it fully self-contained. :( Are you able to get this to work?

The problem I'm currently stuck on is that MacroTools uses this animals_file in its __init__() method, so we cannot override it. :/

lucatrv · 2018-08-05T17:54:46Z

@NHDaly you could try to go back in history with git to find out which commit introduced the issue.

ranjanan · 2018-08-05T20:55:11Z

Hi @NHDaly, do you happen to have the error trace in a gist somewhere?

NHDaly · 2018-08-05T21:52:53Z

@lucatrv thanks that's a good idea. Sadly, the problem isn't with this package, it's that Blink and all of its dependent packages have changed. We might still be able to get it to work, but my suspicion is that it's joined GTK and TK and the other packages whose __init__ methods prevent them from being able to make a standalone app right now (because the __init__ methods reference hard-coded paths and as far as I can tell, you can't override the __init__ method in a module because it's defined and ran before you have a chance to override it. :/

So I think this class of problems can only be solved by modifying the packages themselves, and/or BinDeps, BinaryProvider, etc, which actually set those hard-coded paths.

NHDaly · 2018-08-05T22:11:00Z

@ranjanan No, i don't yet.. I'll put something together to share! :)

ranjanan · 2018-08-05T22:14:11Z

@NHDaly, I think we should be able to send PRs to make generic changes to every package we need to make them statically compilable (and therefore shippable). Packages will definitely have an interest in being shippable.

NHDaly · 2018-08-05T22:16:12Z

@ranjanan Okay, it's not a gist, but will this do? I've pushed up a commit with a new @test_broken, which shows what i'm talking about:

74d8667
Here's the Travis run: https://travis-ci.org/NHDaly/ApplicationBuilder.jl/jobs/412419989

But tl;dr, it builds Blink using the current example build file here, and then renames .julia and tries to run the resultant binary. That tests whether or not it can run without julia installed (or whether it's still depending on hard-coded paths inside the packages). This is the output when I run that manually:

17:48:32 $ mv ~/.julia ~/.julia.bak
 17:48:33 $ /var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink
WARNING: redefining constant JULIA_HOME
fatal: error thrown and no exception handler available.
Base.InitError(mod=:MacroTools, error=Base.SystemError(prefix="opening file /Users/daly/.julia/v0.6/MacroTools/src/../animals.txt", errnum=2, extrainfo=nothing))
rec_backtrace at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
jl_throw at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
systemerror at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
open at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
open at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
__init__ at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jlcall___init___4507 at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink.dylib (unknown line)
jl_module_run_initializer at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
_julia_init at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
julia_init at /private/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/libjulia.dylib (unknown line)
main at /var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp9mz15V/HelloBlink.app/Contents/MacOS/blink (unknown line)
 17:48:43 $ mv ~/.julia.bak ~/.julia

You can see it's trying to open a missing file from inside MacroTools's __init__ from inside jl_module_run_initializer.

So then, I tried adding animals.txt to the things I'm redefining (in this commit) -- (and also pulled MacroTools to master to make sure to get the change you made there: MacroTools.jl/pull/77).

But the problem now is that it still can't find animals.txt because it hasn't executed change_dir_if_bundle yet! The __init__ method is run when the module is loaded, which is way before julia_main is run.

Base.InitError(mod=:MacroTools, error=Base.SystemError(prefix="opening file animals.txt", errnum=2, extrainfo=nothing))

So I think the takeaway here is that -- in order to be shippable -- a Package cannot do any initialization that references the filesystem in their __init__ function. If something like that is necessary, they would have to provide a custom init that the user could call after cding to the right place or something.

NHDaly · 2018-08-05T22:17:24Z

@NHDaly, I think we should be able to send PRs to make generic changes to every package we need to make them statically compilable (and therefore shippable). Packages will definitely have an interest in being shippable.

Yeah, I agree. I think you're right. Figuring out the best way to do that, though, is maybe something we can talk to people about this week! :)

ranjanan · 2018-08-05T22:19:48Z

because it hasn't executed change_dir_if_bundle yet! The init method is run when the module is loaded, which is way before julia_main is run.

Yes, the right fix is to put animals.txt in a separate function that runs way later. That's the true fix. I'll discuss this week with you about the list of changes. In fact, maybe we should sit and discuss about our talks as well. :-)

NHDaly · 2018-08-05T22:19:53Z

Or, maybe we could somehow disable running the module initializers when compiling, and then just manually call them all after doing the cd.

In fact, we could even do all that in C... which could be easier.

NHDaly · 2018-08-05T22:20:11Z

In fact, maybe we should sit and discuss about our talks as well. :-)

Yeah that would be excellent!! :)

I'll be in london tomorrow (monday) morning!!!!! :D

NHDaly · 2018-08-05T23:05:53Z

Oh! Okay cool! With ranjan's help offline, i've fixed it! :)
c89a401

It required making this change to MacroTools, which i'll send a PR for later:
NHDaly/MacroTools.jl@2b4034b

yay

gabrielfreire · 2019-05-02T12:41:19Z

Hi Guys,

Nice discussion. @NHDaly thank you for putting so much effort on this.

I had a lot of problems to run my executable in another computer with the HTTP package because of MbedTLS on Windows 10 and Julia 1.1.0.

i solved by injecting this code inside my julia_main

MbedTLS = HTTP.Servers.MbedTLS
if get(ENV, "COMPILING_APPLE_BUNDLE", "false") == "true"
        # i had to clear out this const depsjl_path because 
        # it points to deps.jl which sets the lib paths in runtime
        Core.eval(MbedTLS, :(const depsjl_path = ""))

        # set lib path relative to PROGRAM_FILE path
        # p.s. single \ or double \\ won't work. 
        # joinpath, abspath, dirname, etc will resolve to wrong paths and will break runtime
        # thankfully julia works well with \\\\ and its possible to have 
        # relative paths to the executable installation
        Core.eval(MbedTLS, :(const libmbedcrypto = "..\\\\lib\\\\$(basename(libmbedcrypto))"))
        Core.eval(MbedTLS, :(const libmbedtls = "..\\\\lib\\\\$(basename(libmbedtls))"))
        Core.eval(MbedTLS, :(const libmbedx509 = "..\\\\lib\\\\$(basename(libmbedx509))"))
else
        # Check if PROGRAM_FILE is my program and not PackageCompiler
        if occursin("MyOwnProgramName", PROGRAM_FILE)
            this_path = abspath(dirname(PROGRAM_FILE))
            cd(this_path) # cd there otherwise it won't find the libraries
        end
end

and my libraries array looks like this

build_app_bundle(app_path, 
                appname="MyOwnProgramName", 
                snoopfile="call_functions.jl", 
                verbose=true, 
                create_installer=true,
                libraries=[HTTP.Servers.MbedTLS.libmbedcrypto, 
                               HTTP.Servers.MbedTLS.libmbedtls, 
                               HTTP.Servers.MbedTLS.libmbedx509])

BTW, i think COMPILING_APPLE_BUNDLE should be COMPILING_BUNDLE hehe

Cheers.

NHDaly closed this as completed Apr 24, 2018

NHDaly reopened this Apr 24, 2018

NHDaly added tracking Issues that are used to document ongoing work on an issue/topic help wanted Extra attention is needed labels Apr 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tracking: Building an application that uses `Blink.jl` #5

tracking: Building an application that uses `Blink.jl` #5

NHDaly commented Apr 23, 2018 •

edited

Loading

NHDaly commented Apr 23, 2018

NHDaly commented Apr 24, 2018

NHDaly commented Apr 24, 2018

NHDaly commented Apr 24, 2018

ranjanan commented Apr 24, 2018

NHDaly commented Apr 24, 2018 •

edited

Loading

NHDaly commented Apr 24, 2018

NHDaly commented Aug 4, 2018

lucatrv commented Aug 5, 2018

ranjanan commented Aug 5, 2018

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018

ranjanan commented Aug 5, 2018

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018

ranjanan commented Aug 5, 2018

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018

gabrielfreire commented May 2, 2019

tracking: Building an application that uses Blink.jl #5

tracking: Building an application that uses Blink.jl #5

Comments

NHDaly commented Apr 23, 2018 • edited Loading

NHDaly commented Apr 23, 2018

NHDaly commented Apr 24, 2018

NHDaly commented Apr 24, 2018

NHDaly commented Apr 24, 2018

ranjanan commented Apr 24, 2018

NHDaly commented Apr 24, 2018 • edited Loading

NHDaly commented Apr 24, 2018

NHDaly commented Aug 4, 2018

lucatrv commented Aug 5, 2018

ranjanan commented Aug 5, 2018

NHDaly commented Aug 5, 2018 • edited Loading

NHDaly commented Aug 5, 2018

ranjanan commented Aug 5, 2018

NHDaly commented Aug 5, 2018 • edited Loading

NHDaly commented Aug 5, 2018

ranjanan commented Aug 5, 2018

NHDaly commented Aug 5, 2018 • edited Loading

NHDaly commented Aug 5, 2018 • edited Loading

NHDaly commented Aug 5, 2018

gabrielfreire commented May 2, 2019

tracking: Building an application that uses `Blink.jl` #5

tracking: Building an application that uses `Blink.jl` #5

NHDaly commented Apr 23, 2018 •

edited

Loading

NHDaly commented Apr 24, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018 •

edited

Loading

NHDaly commented Aug 5, 2018 •

edited

Loading