Nilable property issue in YAML::Serializable on macOS

[zhangkaizhao]

File ser.cr:

require "yaml"

class Employee
  include YAML::Serializable

  property title : String?
  property name : String
end

employee = Employee.from_yaml("title: Manager\nname: John")
puts employee.title # => "Manager"
puts employee.name  # => "John"

employee.name = "Jenny"
puts employee.name # => "Jenny"

Run it under Linux:

docker run --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:latest-alpine \
    crystal run ser.cr

got expected:

Manager
John
Jenny

But run it under macOS:

CC="zig cc" crystal run ser.cr

got exception:

Failed to raise an exception: END_OF_STACK
[0x0] ???

Tried to raise:: Expected Nil, not "Manager" at line 1, column 8 (YAML::ParseException)

Crystal version on Linux:

docker run --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:latest-alpine \
    crystal --version

got

Crystal 1.3.2 [932f193ae] (2022-01-18)

LLVM: 10.0.0
Default target: x86_64-unknown-linux-musl

Crystal version on macOS:

crystal --version

got

Crystal 1.3.2 [932f193ae] (2022-01-18)

LLVM: 10.0.0
Default target: x86_64-apple-macosx

Note: I am using zig cc as C compiler on macOS. Please see here for the details.

[kostya]

for me its ok:

Crystal 1.3.2 (2022-01-18)

LLVM: 13.0.0
Default target: aarch64-apple-darwin20.6.0

[asterite]

What if you don't use zig cc?

[zhangkaizhao]

What if you don't use zig cc?

crystal run ser.cr

Undefined symbols for architecture x86_64:
  "_iconv", referenced from:
      _*Crystal::Iconv#convert<Pointer(Pointer(UInt8)), Pointer(UInt64), Pointer(Pointer(UInt8)), Pointer(UInt64)>:UInt64 in C-rystal5858I-conv.o
  "_iconv_close", referenced from:
      _*Crystal::Iconv#close:Nil in C-rystal5858I-conv.o
  "_iconv_open", referenced from:
      _*Crystal::Iconv#initialize<String, String, (Symbol | Nil)>:Nil in C-rystal5858I-conv.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `cc "${@}" -o /Users/kaizhao/.cache/crystal/crystal-run-ser.tmp  -rdynamic -L/usr/local/programs/crystal/embedded/lib -L/opt/pkg/lib -lyaml  -lpcre -lgc -lpthread -levent -liconv -ldl`

as I mentioned here.
I think the issue may be fixed in #11876 .

[zhangkaizhao]

OK. I followed the suggestion in #10653 (comment) :

crystal build --link-flags -L/usr/lib ser.cr

and run the built binary executable:

./ser

then got expected

Manager
John
Jenny

otool -L ser

ser:
	/usr/lib/libpcre.0.dylib (compatibility version 1.0.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

now Crystal compiler uses system provided libiconv.

[zhangkaizhao]

From documentation https://crystal-lang.org/install/on_mac_os/

The Crystal compiler will by default use pkg-config to find the locations of libraries to link with.

This should be the root cause of the libiconv issue.
I have pkgin/pkgsrc with pkgconf and libiconv packages installed. The zig cc compiler used them correctly but the default compiler on macOS not? I don't know why.
Although zig cc resolved the libiconv issue but resulted in unexpected issue, I just don't know which one I can use.
The suggestion --link-flags -L/usr/lib is not friendly too.

[straight-shoota]

This looks like two different problems. One is a weird deserialization error (that apparently only happens when linking with zig cc?) and the other is about linking libiconv with the system linker. Is that correct? If so, please open a separate issue for the other one. Thanks.

[zhangkaizhao]

@straight-shoota Yes. Now there are two different problems in my case. But I will not open a seperate issue for the second one because I don't know the status of linking libiconv now. And there are some other issues still opened about it. e.g. #11882 , #8413 , etc.

[HertzDevil]

zig cc forwards to lld. Can you replicate the failures using something like crystal build --link-flags=-fuse-ld=lld?

[zhangkaizhao]

zig cc forwards to lld. Can you replicate the failures using something like crystal build --link-flags=-fuse-ld=lld?

There is no lld on macOS by default.

crystal build --link-flags=-fuse-ld=lld ser.cr

got

clang: error: invalid linker name in argument '-fuse-ld=lld'
Error: execution of command failed with code: 1: `cc "${@}" -o /Users/kaizhao/lab/crystal/yaml/ser -fuse-ld=lld -rdynamic -L/usr/local/programs/crystal/embedded/lib -L/opt/pkg/lib -lyaml  -lpcre -lgc -lpthread -levent -liconv -ldl`

pkgin/pkgsrc has the lld package depending on the llvm package.
And I guess the llvm package on Homebrew should have the lld linker.
But I don't want to install either of them. It seems like unnecessary because there are Zig binary distribution and Crystal binary distribution already.

[zhangkaizhao]

Oh. There is another issue raised when compiling with option -d, --debug Add full symbolic debug info on macOS:

crystal build --debug ser.cr

got

Assertion failed: (Idx < NumElements && "Invalid element idx!"), function getElementOffset, file /var/cache/omnibus/src/llvm/llvm-10.0.0.src/include/llvm/IR/DataLayout.h, line 608.
/Users/kaizhao/bin/crystal: line 102: 78296 Abort trap: 6           "$INSTALL_DIR/embedded/bin/crystal" "$@"

crystal build --debug --link-flags -L/usr/lib ser.cr

got the same

Assertion failed: (Idx < NumElements && "Invalid element idx!"), function getElementOffset, file /var/cache/omnibus/src/llvm/llvm-10.0.0.src/include/llvm/IR/DataLayout.h, line 608.
/Users/kaizhao/bin/crystal: line 102: 78332 Abort trap: 6           "$INSTALL_DIR/embedded/bin/crystal" "$@"

CC="zig cc" crystal build --debug ser.cr

got the same

Assertion failed: (Idx < NumElements && "Invalid element idx!"), function getElementOffset, file /var/cache/omnibus/src/llvm/llvm-10.0.0.src/include/llvm/IR/DataLayout.h, line 608.
/Users/kaizhao/bin/crystal: line 102: 78387 Abort trap: 6           "$INSTALL_DIR/embedded/bin/crystal" "$@"

Note: /Users/kaizhao/bin/crystal is just an soft link to the bin/crystal in the Crystal binary distribution.

/Users/kaizhao/bin/crystal@ -> /usr/local/programs/crystal/bin/crystal

There is no such issue when compiling the hello word example and most of the example code on the homepage of the official website of Crystal, e.g. Macros, Concurrency Model, etc.

[zhangkaizhao]

Oh. There is another issue raised when compiling with option -d, --debug Add full symbolic debug info on macOS:

crystal build --debug ser.cr

got

Assertion failed: (Idx < NumElements && "Invalid element idx!"), function getElementOffset, file /var/cache/omnibus/src/llvm/llvm-10.0.0.src/include/llvm/IR/DataLayout.h, line 608.
/Users/kaizhao/bin/crystal: line 102: 78296 Abort trap: 6           "$INSTALL_DIR/embedded/bin/crystal" "$@"

crystal build --debug --link-flags -L/usr/lib ser.cr

got the same

Assertion failed: (Idx < NumElements && "Invalid element idx!"), function getElementOffset, file /var/cache/omnibus/src/llvm/llvm-10.0.0.src/include/llvm/IR/DataLayout.h, line 608.
/Users/kaizhao/bin/crystal: line 102: 78332 Abort trap: 6           "$INSTALL_DIR/embedded/bin/crystal" "$@"

CC="zig cc" crystal build --debug ser.cr

got the same

Assertion failed: (Idx < NumElements && "Invalid element idx!"), function getElementOffset, file /var/cache/omnibus/src/llvm/llvm-10.0.0.src/include/llvm/IR/DataLayout.h, line 608.
/Users/kaizhao/bin/crystal: line 102: 78387 Abort trap: 6           "$INSTALL_DIR/embedded/bin/crystal" "$@"

Note: /Users/kaizhao/bin/crystal is just an soft link to the bin/crystal in the Crystal binary distribution.

/Users/kaizhao/bin/crystal@ -> /usr/local/programs/crystal/bin/crystal

There is no such issue when compiling the hello word example and most of the example code on the homepage of the official website of Crystal, e.g. Macros, Concurrency Model, etc.

Confirm the above issue is fixed in the 1.8.0 release. <3 @HertzDevil

% crystal --version
Crystal 1.8.0 [14bfa992e] (2023-04-14)

LLVM: 15.0.7
Default target: x86_64-apple-macosx

% crystal build --debug ser.cr
% otool -L ser
ser:
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
% ./ser
Manager
John
Jenny

% zig version
0.10.1
% CC="zig cc" crystal build --debug ser.cr
% otool -L ser
ser:
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
% ./ser
Failed to raise an exception: END_OF_STACK
[0x0] ???

Tried to raise:: Expected Nil, not "Manager" at line 1, column 8 (YAML::ParseException)