Internally use objcopy --add-section on macOS and Linux

[RaisinTen]

objcopy --add-section works on Linux - https://man7.org/linux/man-pages/man1/objcopy.1.html.

After https://reviews.llvm.org/D66283 landed, llvm-objcopy on macOS too started supporting this feature: https://llvm.org/docs/CommandGuide/llvm-objcopy.html#cmdoption-llvm-objcopy-add-section

This demo is using the Node.js binary that has been built from nodejs/node#45038 on macOS:

$ ls
index.js node
$ cat index.js
console.log('Hello, world!');
$ /usr/local/opt/llvm/bin/llvm-objcopy --add-section __POSTJECT,__NODE_JS_CODE=index.js node sea
$ ./sea
Hello, world!

Advantages:

• We can avoid maintaining any complex code that adds a new section into a binary and reuse existing tools.
• Much faster than the current state of postject.

  $ /usr/bin/time /usr/local/opt/llvm/bin/llvm-objcopy --add-section __POSTJECT,__NODE_JS_CODE=index.js node sea
          0.37 real         0.22 user         0.11 sys
  $ /usr/bin/time postject sea NODE_JS_CODE index.js
         37.59 real        40.05 user         1.12 sys

• Possibly works on AIX also? I would need access to such a system to make sure.

Disadvantage:

• objcopy is present by default on Linux but users would have to download it manually on macOS along with other tools using brew install llvm.

@nodejs/single-executable wdyt?

@bnoordhuis I'd also like to hear your thoughts on this since you were sharing some insights on the resource injection part in nodejs/node#45066.

[jviotti]

Makes total sense to me!

[tony-go]

+1 👏

[RaisinTen]

Another problem I can think of is that, it won't let you embed your source code into a an executable format that's not used on that platform, like you won't be able to embed data into a Mach-O file on Windows. Pkg is currently able to produce ELF, Mach-O as well as PE executables regardless of the host platform. @jesec, is that a hard requirement of pkg?

[jviotti]

That is a good point indeed, and worth checking what Node.js folks think too.

[ovflowd]

objcopy is present by default on Linux but users would have to download it manually on macOS along with other tools using brew install llvm.

That feels like would become a dependency of node as if sea is part of node-core, all dependencies/pre-reqs need to be installed with node or bundled with node, no?

And feels here like llvm would be an external runtime dependency, right?

[RaisinTen]

I didn't mention it during the meeting but it seems like the cross-platform binary creation might actually not even work if your project has native bindings, so maybe this still makes sense?

To @ovflowd's point, maybe we can find a way to ship just llvm-objcopy as a standalone executable?

[RaisinTen]

PR - #64

[dsanders11]

I'm a fairly strong -1 on this idea for several reasons.

This is going to be a somewhat lengthy comment, so before getting further into it, I want to point out that objcopy and llvm-objcopy are not the same. This issue and PR #64 use them interchangeably which is ripe for confusion. While llvm-objcopy may try to support the same options (and more) as objcopy so that it's a drop-in replacement in most cases, the implementation can and is quite different, like the support for Mach-O files. In particular the implementation of --add-section is different for ELF (I'm pretty sure) than in objcopy and the documentation for the option is different as a result, with llvm-objcopy providing more details on the implementation (regarding note sections). In what follows I'm explicit about which one I'm discussing and use the name accordingly.

That said, first and foremost, I don't think it works.

When exploring ways to accomplish the injection (before the Postject project existed) I evaluated objcopy and discarded it as not a solution to the problem. With ELF executables if you use objcopy --add-section, you're adding a section, not a segment - we went through this once already with the initial implementation for ELF. In ELF, sections are for the linker, but serve no purpose after that point. You can strip the section header table (SHT) from the binary and it will continue to work just fine. When using objcopy --add-section for ELF you'd also need to use --set-section-flags to set alloc and readonly. If you do that you'll see the warning warning: allocated section '.foo' not in segment - because objcopy isn't adding a segment, it's adding a section. Without the section being in a segment, it won't be paged into memory and can't be accessed at run-time like we need for Postject. Here's a very brief forum post explaining that issue and a more in-depth mailing list post. In ELF binaries sections with the same flags are grouped together into segments with those same properties (such as read-only) at linking time, and then the SHT keeps track of where the sections ended up (virtual address) but that information isn't needed at run-time for normal execution, which is why the SHT can be stripped and execution is normal. I don't really know what the intended use-case of objcopy --add-section is, but it doesn't fit our needs since we need the section to be in a loadable segment which can be accessed at run-time. cc @robertgzr as we've previously discussed the differences between sections and segments in ELF and how sections are not a run-time concept.

I have not explored llvm-objcopy --add-section to see if the above is still true for their implementation on ELF. It may be, it may not be. There appear to be other problems with llvm-objcopy --add-section though, as pointed out on #64. For Mach-O it is adding the segment after LC_CODE_SIGNATURE, which causes problems. It doesn't appear to support --set-section-flags for Mach-O at all (we need that to make the segment read-only). Finally, when testing it with Electron, it appears to be significantly altering the executable - just running llvm-objcopy --add-section changes the size of the executable from 315 MB to 267 MB and understandably it no longer runs.

Beyond not being suitable for the purpose (IMO), I'd like to address the advantages/disadvantages discussed here.

We can avoid maintaining any complex code that adds a new section into a binary and reuse existing tools.

Is this referring to the code using LIEF in Postject, or upstream LIEF? I'd argue that relying on llvm-objcopy adds more maintenance complexity, not less. It's still under development, and we can and will find bugs (see previous point about it drastically changing the Electron executable), which we'll have to fix upstream, and LLVM is a more complex codebase than LLVM (IMO, I've worked in both). We also have no control over when those fixes actually land if we're having users bring their own copy of llvm-objcopy, through brew for example. Finally, if it's bring your own copy of llvm-objcopy, then we significantly increase our maintenance headache since we don't control it and we'd be stuck with the problem of versions for that tool, we'd have to determine which ones are supported, test new ones when they are released, etc. With LIEF we're freezing to a known good version and that will never change under our feet.

It's also just weird for postject as a CLI tool to then rely on the user to separately install llvm-objcopy.

I don't think bundling llvm-objcopy is feasible either. The whole reason we changed to WASM was to avoid native dependencies, so bundling it would just bring that back. Even if we don't bundle it and rely on the user to bring their own copy, we're still back into needing native dependencies, which is not where we want to be.

Now, it is potentially interesting to consider building llvm-objcopy to WASM and consuming it programmatically that way, like we currently do with LIEF, which would eliminate the native dependency issue and the users bringing their own copy issue since we'd pin it and consume a known good version. That said, there's still the "it doesn't actually work" issue detailed above. Also the downside that while LIEF is a solution for PE executables as well, llvm-objcopy is not, so we'd end up increasing the size of Postject even more. There could be an LLVM solution for PE executables, but research would need to be done.

Much faster than the current state of postject.

There are improvements that can be made here at the WASM level, that I haven't opened a PR for yet. That said, I don't think the speed of the injection is a major concern, and shouldn't be a major motivator for a significant implementation change like this.

objcopy is present by default on Linux but users would have to download it manually on macOS along with other tools using brew install llvm.

See opening points, but for consistency we do not want to mix objcopy and llvm-objcopy, that would be confusing and error prone. We'd want to use llvm-objcopy across the board, which means you wouldn't be using the system objcopy on Linux machines.

Another problem I can think of is that, it won't let you embed your source code into a an executable format that's not used on that platform, like you won't be able to embed data into a Mach-O file on Windows.

I think this is a major drawback. Being platform-agnostic for the injection is a major benefit and makes Postject easy to get started with immediately for testing. I'd want strong reasons to abandon this benefit, especially after we did the work to use WASM and make Postject completely platform-agnostic (heck, you could provide a website that does the injection in-browser and not even require the user to install anything).

I didn't mention it during the meeting but it seems like the cross-platform binary creation might actually not even work if your project has native bindings, so maybe this still makes sense?

That sounds like a bug - cross-compiling is the standard solution there and is supported by tools like node-gyp.

I'm in no way opposed to replacing LIEF or the underlying implementation - that's one of the main benefits to building to WASM and being entirely self-contained. But I don't think this is the right direction and I don't really see what is gained.

[RaisinTen]

We can avoid maintaining any complex code that adds a new section into a binary and reuse existing tools.

Is this referring to the code using LIEF in Postject, or upstream LIEF?

I was referring to maintaining a pure JS implementation of a resource injector for all binary formats.

cross-compiling is the standard solution there and is supported by tools like node-gyp

nodejs/node-gyp#829 tells me that it is not supported. 🤔

---

Anyways, excellent points, thanks! I'll close this and instead work on the pure JS resource injector.

[dsanders11]

nodejs/node-gyp#829 tells me that it is not supported. 🤔

Good catch, I was thinking cross-compiling, not cross-platform. That indeed might not be possible.