Add bloopgun CLI library and binary #1032

jvican · 2019-09-12T07:12:51Z

Bloopgun is a Scala library to talk to bloop via Nailgun. It can be compiled to a binary via GraalVM and be used as a (shaded) library. The first use case is essential for users that need a fast CLI experience, the second use case is essential for build clients that want to connect to Bloop quickly and robustly.

This pull request adds a lot of infrastructure necessary to make bloopgun possible as well as bloop4j, a bsp4j wrapper to talk to bloop via BSP which will be necessary for the sbt offloading.

On the bloopgun side, there is a new innovation compared to the previous Python bloop nailgun client: bloopgun will start the server if there is no server running. This is a game changer for users of the library because it means neither users nor library authors will need to manager a server running in the background.

Given that some of this was done previously by the launcher, that logic has been removed and the launcher instead relies on bloopgun. The launcher has become less of a launcher --now it only takes care of establishing a bap connection--, so I'm considering changing its name to something else.

jvican · 2019-09-12T07:20:37Z

I'm not asking for review yet given that I'm experimenting/finishing some stuff on the bloop4j front. I'm probably going to leave this out of the minor release this week.

jvican · 2019-09-12T13:03:40Z

Missing items:

Reuse resolution field from a previous configuration file in sbt-bloop. Blocked by the lack of a shaded sbt-bloop. A little bit unrelated but will add it to this pull request.
Publish the shaded sbt-bloop instead of the normal sbt-bloop.
Publish the shaded bloopgun and figure out infrastructure to publish native images and generate them in other operating systems.
Publish the shaded launcher instead of the normal one, especially important now that we depend on coursier. Unfortunately, we cannot compile the launcher to native because we use JNA for sockets and GraalVM doesn't have support for JNA code yet.
Add built-in support for JVM flags to speed up compilation: -XX:UseParallelGC and -XX:MaxInlineLevel=20.

jvican · 2019-09-24T14:59:48Z

Everything is more or less in place except for the GraalVM-native infrastructure in the CI. I'm going to hold off doing that for now given that it's quite difficult. I'll focus on modifying sbt-bloop to keep the resolution modules in the configuration file and this is good to go.

The shaded plugin helps us solve several problems that existed when loading circe dependencies in the sbt bloop plugin. These errors don't happen again.

When `bloop server` is currently run, the current pynailgun script will block for the duration of the server execution. This commit refactors the codebase to imitate the same behavior, which is important for Systemd and Brew services running `bloop server` in the background. If we were returning immediately, these services would try to run the start routines several times per second which would create an unnecessary overhead on the server.

Maven Central had problems releasing snailgun-core for 0.3.0 so we released a new version and now we upgrade to it.

According to our performance guide, running bloop with FullParallelGC and MaxInlineLevel as well as other options is beneficial for performance. This commit allows bloop to try to use those options and if they fail backtracks to the initial server launch.

alexarchambault · 2019-09-26T07:58:26Z

@jvican It seems sbt-scalafix and sbt-scalafmt are around in some projects here. They depend on coursier-small, itself depending on old versions of coursier, which don't play well concurrently when shaded, hence the errors you're seeing. sbt-scalafmt and sbt-scalafix should switch to coursier-interface.

olafurpg · 2019-09-26T08:58:57Z

I opened scalacenter/sbt-scalafix#43 to migrate sbt-scalafix from coursier-small. I'll do the same for sbt-scalafmt.

jvican · 2019-09-26T09:01:51Z

@alexarchambault Thanks for the pointers. We don't use sbt-scalafix but we do use sbt-scalafmt. Unfortunately, disabling sbt-scalafmt doesn't remove the error. Disabling the CoursierPlugin on the offending project did work in a previous attempt, though, for a part of the build. Any ideas what could be wrong here? I tried to see where another coursier version gets pulled in but I couldn't find it.

I should say that /root/.coursier is a symlink and it has been working just fine previously. After the upgrade to 2.0.0-RC3-3, things have started to go wrong, so I really wonder what's happening here 🤔

olafurpg · 2019-09-26T10:25:42Z

PR migrating scalafmt to coursier/interface scalameta/scalafmt#1511

JNA checks fail on Windows CI. Let's just assume all clients are non-interactive, it should be OK since Bloop doesn't do anything fancy based on this information.

When downgrading, `toShadeJars` didn't work as expected. Therefore, we reimplement our own `toShadeJars`, which gives us a lot of power to filter out whatever jars we don't want to shade.

jvican · 2019-09-26T13:20:54Z

Alright, I've run out of hacks. I downgraded coursier and the issue persists, this looks like a bug inside sbt-shading. I'm going to reimplement the functionality I care about in my build and remove the dependency of sbt-shading on coursier by just populating toShadeJars with files from dependencyClasspath and then filtering which jars should be shaded.

alexarchambault · 2019-09-26T13:36:47Z

@jvican Did you check sbt-declarative-packaging (via this)?

alexarchambault · 2019-09-26T13:38:36Z

but yeah, doing that manually might actually be more straightforward

Depending on coursier's sbt-shading is causing us lots of headaches in the CI where we get cryptic errors we haven't found a solution for. This patch implements the important bits of sbt-shading inside the build for our very particular use cases.

olafurpg · 2019-09-26T16:06:33Z

sbt-scalafmt upgrade to a no-coursier-small dependency scalameta/sbt-scalafmt#50

In preparation for the offloading compilation feature in sbt, we need to make sure we never lose the information in the resolutions field as it is important for the good behavior of Metals. Given a build that was exported with source information from the Metals interface, we want sbt-bloop to continue to preserve that information if it runs `bloopGenerate` on its own before offloading compilation to sbt.

jvican · 2019-09-26T22:22:43Z

The error here seems to be caused by a coursier version higher than 1.1.0-M3-2. My intuition is that coursier doesn't like the fact that the coursier cache is a symlink in those versions, while in the current version seems to work OK.

I ended up anyway implementing sbt-shading inside bloop, it makes it easier to modify what jars should be shaded and which should be not. This is just a temporary workaround, in the future I'd like to have an independent sbt-plugin built on top of coursier's sbt-shading. For now, we're taking this shortcut.

CI seems happy after the changes I did. This is good to go, so I'm merging as soon as the CI is green.

Hopefully this will address some race conditions I've been seeing in bsp diagnostics reports where the output of some diagnostics is intermingled.

olafurpg · 2019-09-27T07:55:48Z

🥳👏👏👏

jvican added cli feature integrations nailgun labels Sep 12, 2019

jvican mentioned this pull request Sep 12, 2019

Compile to GraalVM native image tindzk/seed#57

Open

jvican force-pushed the topic/bloopgun branch 2 times, most recently from 0612abf to 0861dc4 Compare September 19, 2019 08:37

jvican force-pushed the topic/bloopgun branch from 22e6b2e to b2c3ad6 Compare September 24, 2019 18:18

jvican added 19 commits September 25, 2019 12:30

Add initial commit for bloopgun

1d3fab1

Play with first bloopgun prototype

3acb58a

Prepare shading/graalvm-native and dependency resolution

d8cdbe9

Move logic to locate server around

854b464

Replace logic to add plugin to itself with shaded plugin

1aa0b2a

The shaded plugin helps us solve several problems that existed when loading circe dependencies in the sbt bloop plugin. These errors don't happen again.

Add logic to start server in the background

83b2d61

Remove unnecessary resolution in bloopgun

33d9eb8

Rename old compiler oracle name

b08eb24

Read JVM options and check bloopgun works

e242318

Make launcher depend on bloopgun

15a300b

Add baby steps to design bloop4j

cb78725

Add method to get project id from URI

14a1246

Make more advancements with low-level build client API

7f40850

Replace --verbose with --nailgun-verbose in Bloopgun

dca533a

Update snailgun to 0.3.1

456920f

Maven Central had problems releasing snailgun-core for 0.3.0 so we released a new version and now we upgrade to it.

Fix build error from rebasing

8b64cb5

Publish shaded bloopgun/launcher modules

9fa6233

Use channels to proxy streams in launcher

6efafc7

jvican force-pushed the topic/bloopgun branch 2 times, most recently from 54c2245 to ae7198f Compare September 26, 2019 06:42

jvican force-pushed the topic/bloopgun branch from bb91152 to 6471af2 Compare September 26, 2019 09:18

jvican added 2 commits September 26, 2019 13:40

Disable interactivity in bloopgun

eba273d

JNA checks fail on Windows CI. Let's just assume all clients are non-interactive, it should be OK since Bloop doesn't do anything fancy based on this information.

Downgrade to coursier 1.1.0-M14-4 and reimplement toShadeJars

95a1386

When downgrading, `toShadeJars` didn't work as expected. Therefore, we reimplement our own `toShadeJars`, which gives us a lot of power to filter out whatever jars we don't want to shade.

jvican force-pushed the topic/bloopgun branch from 44ba82d to 95a1386 Compare September 26, 2019 12:41

Implement toShadeJars in metaproject

cab5def

jvican added 5 commits September 26, 2019 18:14

Minimize coursier-related changes in build

04e0b87

Use 1.0.0-M13-2 exclusively

c5ad201

Fix scripted tests that use shaded sbt-bloop

1ae8e7d

Shell out if detected binary is executable

5e8e687

Synchronize on buffer to register notifications from test

10d6baa

Hopefully this will address some race conditions I've been seeing in bsp diagnostics reports where the output of some diagnostics is intermingled.

jvican merged commit 71ebe9a into master Sep 27, 2019

alexarchambault mentioned this pull request Sep 27, 2019

[debug] Switch to sbt-coursier 1.1.0-M14-3 #1039

Closed

jvican mentioned this pull request Sep 28, 2019

Use -XX:+UseParallelGC in Java 9 and 10 runtimes #624

Closed

alexarchambault mentioned this pull request Oct 14, 2019

Switch to sbt-coursier 2.0.0-RC4 #1066

Closed

tgodzik deleted the topic/bloopgun branch September 7, 2021 16:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bloopgun CLI library and binary #1032

Add bloopgun CLI library and binary #1032

jvican commented Sep 12, 2019 •

edited

Loading

jvican commented Sep 12, 2019

jvican commented Sep 12, 2019 •

edited

Loading

jvican commented Sep 24, 2019

alexarchambault commented Sep 26, 2019 •

edited

Loading

olafurpg commented Sep 26, 2019

jvican commented Sep 26, 2019

olafurpg commented Sep 26, 2019 •

edited

Loading

jvican commented Sep 26, 2019

alexarchambault commented Sep 26, 2019

alexarchambault commented Sep 26, 2019

olafurpg commented Sep 26, 2019

jvican commented Sep 26, 2019 •

edited

Loading

olafurpg commented Sep 27, 2019

Add bloopgun CLI library and binary #1032

Add bloopgun CLI library and binary #1032

Conversation

jvican commented Sep 12, 2019 • edited Loading

jvican commented Sep 12, 2019

jvican commented Sep 12, 2019 • edited Loading

jvican commented Sep 24, 2019

alexarchambault commented Sep 26, 2019 • edited Loading

olafurpg commented Sep 26, 2019

jvican commented Sep 26, 2019

olafurpg commented Sep 26, 2019 • edited Loading

jvican commented Sep 26, 2019

alexarchambault commented Sep 26, 2019

alexarchambault commented Sep 26, 2019

olafurpg commented Sep 26, 2019

jvican commented Sep 26, 2019 • edited Loading

olafurpg commented Sep 27, 2019

jvican commented Sep 12, 2019 •

edited

Loading

jvican commented Sep 12, 2019 •

edited

Loading

alexarchambault commented Sep 26, 2019 •

edited

Loading

olafurpg commented Sep 26, 2019 •

edited

Loading

jvican commented Sep 26, 2019 •

edited

Loading