Limiting file name length to 240 chars.

[flurdy]

Setting file name length limit on generated classes.

Encrypted disks or containers via Docker often has file name length limits that Shapeless often exceeds.
Docker for instance has a limit of 242 charachters.

This is easily worked around by moving the build to non encrypted partition e.g. /tmp if that is available.
For full disk encrypted machines and always with Docker that is not an option.

The other option is adding "-Xmax-classfile-name 240" to your project's scalacOption.
But that only solves it for your own generated classes not your dependencies.

If your build uses for instance sbt-assembly, which my Docker build does, then it will extract
all classes out of your dependencies jars and some may violate the file name length.
So some individual transitive dependency may also need to be buildt with Xmax-classfile-name, in my case Shapeless :(

This PR sets the "-Xmax-classfile-name" to 240 characters in the scalacOption.

If this is approved, I will create PRs for the same change on the 1.2.4 branches.

Please note I do not know if there are other side effects of setting the physical file name length limit.

[milessabin]

Thanks for the report. I asked around about the implications of using this option, and @mpilquist pointed me in the direction of a discussion on scala-internals. The most relevant excerpt is from this response by Lukas Rytz,

The danger of using this option is that it affects binary compatibility.
...
You need to use the same value for all of your dependencies. Given that you (most likely) depend
on existing libraries, such as the Scala standard library, you have to use the same value that was
used for compiling those.

So I think this is not something we can do in general. You can, of course, compile your entire application with -Xmax-classfile-name set to a smaller value, but then you might have to deal with the next point that Lukas raises: "The drawback of using a smaller value is that some names are gone, which can make debugging harder".

My recommendation would be to try and find some mechanism for working around the 240 character filename limit in Docker (if using sbt-assembly in that environment is non-negotiable). Could you create a local filesystem (within your encrypted volume, maybe something FUSE based?) and perform the assembly step there?

TL;DR this is basically wonfix but I'm sympathetic, and I'll leave this ticket open for comments for a while. If you do come up with a workaround I'd very much appreciate it if you would come back and describe it here.

[flurdy]

Thanks for looking into it. I thought there would be some complications with this option.

With the example linked if the original A.scala was compiled with Xmax-classfile-name, then Test.scala works both when it is compiled with Xmax-classfile-name and when not. In our case A.scala could be the Shapeless library. Further path tests could probably break it though and as mentioned not consistent.

The debug issue of lost names mentioned would still be a problem. Xmax-classfile-name does have a default value of 255 that probably causes some name loss in existing projects already.

I do feel this is shaky ground that I don't want to break so I agree with the wontfix.

Sbt-assembly should really be done on a CI server in a company environment, so this problem would normally not be an issue. But in this specific instance it was for a small Github project where building and running of a Spray app is done with Docker Compose and required container side Sbt assembly.
However I will roll an unmanaged custom version of 1.2.4 with the file name length fix in it for now.

I can foresee more people coming across this problem soon as container/Docker based use is spreading into more than just runtime and probably even Docker based CI nodes are in use already.

Mounting another filesystem inside Docker may be a better solution but it seems a bit overkill but I will keep looking into it and comment here if I find out any more. Changing the default value in Scala to accommodate Docker and not the other way round is probably not the solution. Convincing everyone to switch to Amiga SFS with its 32,000 character limit is probably not the solution either.

[milessabin]

Thanks for the update. I'll close this for now, but please report back if there are any further developments.

[reactormonk]

Since it's a local problem, why couldn't you add

scalacOptions ++= Seq("-Xmax-classfile-name","240")

or an equivalent to ~/.sbt/0.13/local.sbt?

[flurdy]

Thanks reactormonk, good thinking I will add that to my standard sbt docker images so I don't need to pollute all projects with that setting.

However it wont solve this issue as it is for precompiled binaries dependencies (in this instance Shapeless) that may include long file names that cause issues when the jar is exploded as e.g. part of sbt-assembly.

( Also set as a wontfix for sbt-assembly project sbt/sbt-assembly#69 )

[nickpoorman]

+1 Running into this issue myself. When packaging assembly in a Docker container, extracting shapeless is causing the build to fail.

[milessabin]

If you read the earlier part of this thread you'll see that this isn't something we can do.

If someone could contribute a write up of the steps required to put together a local build suitable for incorporating in a Docker build for the shapeless wiki that would be very helpful and very much appreciated.

[flurdy]

I just added https://github.com/milessabin/shapeless/wiki/Shapeless-with-SBT-Assembly-inside-Docker to the Wiki. It explains the problem and the work around that I use.

If it is too long or belong as just a paragraph elsewhere then let me know.

[milessabin]

@flurdy That's completely awesome! Thank you very much :-)