Update rbsys to support Ruby 3.3

[djrodgerspryor]

The tests pass and this seems to work perfectly as far as I can tell.

[bmulholland]

Fixes #20

[karlentwistle]

@bmulholland Maybe I've done something wrong but I'm not sure this fixes #20

• I created a new Rails app here https://github.com/karlentwistle/tiktoken_heroku_example using Ruby 3.3.0
• Applied this commit karlentwistle/tiktoken_heroku_example@5888659
• Attempted to push to Heroku and got the following error

Full deploy output including error

Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 603 bytes | 603.00 KiB/s, done.
Total 4 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Updated 82 paths from f5b0d5a
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Building on the Heroku-22 stack
remote: -----> Using buildpack: heroku/ruby
remote: -----> Ruby app detected
remote: -----> Installing bundler 2.3.25
remote: -----> Removing BUNDLED WITH version in the Gemfile.lock
remote: -----> Compiling Ruby/Rails
remote: -----> Using Ruby version: ruby-3.3.0
remote: -----> Installing dependencies using bundler 2.3.25
remote:        Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4
remote:        Fetching https://github.com/StileEducation/tiktoken_ruby.git
remote:        Fetching https://github.com/StileEducation/tiktoken_ruby.git
remote:        Fetching gem metadata from https://rubygems.org/.........
remote:        Fetching rb_sys 0.9.86
remote:        Installing rb_sys 0.9.86
remote:        Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
remote:
remote:        current directory:
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/bin/ruby extconf.rb
remote:        checking for gcc... yes
remote:        checking for g++... yes
remote:        checking for gcc-ar... yes
remote:        checking for cargo... no
remote:
remote:        current directory:
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby
remote:        make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m
remote:        sitelibdir\=./.gem.20240108-198-77bm6m clean
remote:
remote:        current directory:
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby
remote:        make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m
remote:        sitelibdir\=./.gem.20240108-198-77bm6m
remote:        info: downloading installer
remote:        info: profile set to 'minimal'
remote:        info: default host triple is x86_64-unknown-linux-gnu
remote:        info: skipping toolchain installation
remote:
remote:
remote:        Rust is installed now. Great!
remote:
remote:        To get started you need Cargo's bin directory
remote:        (/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
remote:        in your PATH
remote:        environment variable. This has not been done automatically.
remote:
remote:        To configure your current shell, run:
remote:        source
remote:        "/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
remote:        info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
remote:        info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)
remote:        info: downloading component 'cargo'
remote:        info: downloading component 'rust-std'
remote:        info: downloading component 'rustc'
remote:        info: installing component 'cargo'
remote:        info: installing component 'rust-std'
remote:        info: installing component 'rustc'
remote:
remote:        stable-x86_64-unknown-linux-gnu installed - rustc 1.75.0 (82e1608df
remote:        2023-12-21)
remote:
remote:        info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
remote:        info: checking for self-update
remote:        info: using existing install for 'stable-x86_64-unknown-linux-gnu'
remote:        info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
remote:
remote:        stable-x86_64-unknown-linux-gnu unchanged - rustc 1.75.0 (82e1608df
remote:        2023-12-21)
remote:
remote:        info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in
remote:        use (environment override by RUSTUP_TOOLCHAIN)
remote:        generating target/release/libtiktoken_ruby.so (release)
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo
remote:        rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release
remote:        -- -C linker=gcc -L native=/tmp/build_91a6107a/vendor/ruby-3.3.0/lib -C
remote:        link-arg=-lm -l pthread
remote:            Updating crates.io index
remote:            Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
remote:            Updating git submodule `https://github.com/zurawiki/tiktoken`
remote:         Downloading crates ...
remote:          Downloaded base64 v0.21.0
remote:          Downloaded once_cell v1.17.1
remote:          Downloaded bstr v1.4.0
remote:          Downloaded smallvec v1.10.0
remote:          Downloaded bit-vec v0.6.3
remote:          Downloaded cexpr v0.6.0
remote:          Downloaded lock_api v0.4.9
remote:          Downloaded syn v2.0.31
remote:          Downloaded shlex v1.1.0
remote:          Downloaded serde v1.0.157
remote:          Downloaded seq-macro v0.3.5
remote:          Downloaded proc-macro2 v1.0.66
remote:          Downloaded unicode-ident v1.0.8
remote:          Downloaded shell-words v1.1.0
remote:          Downloaded scopeguard v1.1.0
remote:          Downloaded regex-syntax v0.6.28
remote:          Downloaded rb-sys v0.9.81
remote:          Downloaded regex v1.7.1
remote:          Downloaded nom v7.1.3
remote:          Downloaded libc v0.2.140
remote:          Downloaded bit-set v0.5.3
remote:          Downloaded bindgen v0.66.1
remote:          Downloaded rb-sys-env v0.1.2
remote:          Downloaded rb-sys-build v0.9.81
remote:          Downloaded quote v1.0.33
remote:          Downloaded peeking_take_while v0.1.2
remote:          Downloaded parking_lot_core v0.9.7
remote:          Downloaded parking_lot v0.12.1
remote:          Downloaded minimal-lexical v0.2.1
remote:          Downloaded memchr v2.5.0
remote:          Downloaded magnus-macros v0.6.0
remote:          Downloaded magnus v0.6.1
remote:          Downloaded libloading v0.7.4
remote:          Downloaded lazycell v1.3.0
remote:          Downloaded lazy_static v1.4.0
remote:          Downloaded glob v0.3.1
remote:          Downloaded clang-sys v1.6.0
remote:          Downloaded bitflags v2.4.0
remote:          Downloaded autocfg v1.1.0
remote:          Downloaded anyhow v1.0.70
remote:          Downloaded aho-corasick v0.7.20
remote:          Downloaded fancy-regex v0.11.0
remote:          Downloaded rustc-hash v1.1.0
remote:          Downloaded regex-automata v0.1.10
remote:          Downloaded cfg-if v1.0.0
remote:           Compiling memchr v2.5.0
remote:           Compiling proc-macro2 v1.0.66
remote:           Compiling glob v0.3.1
remote:           Compiling unicode-ident v1.0.8
remote:           Compiling clang-sys v1.6.0
remote:           Compiling libc v0.2.140
remote:           Compiling quote v1.0.33
remote:           Compiling cfg-if v1.0.0
remote:           Compiling minimal-lexical v0.2.1
remote:           Compiling syn v2.0.31
remote:           Compiling nom v7.1.3
remote:           Compiling libloading v0.7.4
remote:           Compiling aho-corasick v0.7.20
remote:           Compiling regex-syntax v0.6.28
remote:           Compiling bindgen v0.66.1
remote:           Compiling regex v1.7.1
remote:           Compiling cexpr v0.6.0
remote:           Compiling bitflags v2.4.0
remote:           Compiling lazy_static v1.4.0
remote:           Compiling rustc-hash v1.1.0
remote:           Compiling shlex v1.1.0
remote:           Compiling peeking_take_while v0.1.2
remote:           Compiling lazycell v1.3.0
remote:           Compiling shell-words v1.1.0
remote:           Compiling autocfg v1.1.0
remote:           Compiling lock_api v0.4.9
remote:           Compiling rb-sys-build v0.9.81
remote:           Compiling parking_lot_core v0.9.7
remote:           Compiling rb-sys v0.9.81
remote:           Compiling smallvec v1.10.0
remote:        error: failed to run custom build command for `rb-sys v0.9.81`
remote:
remote:        Caused by:
remote:        process didn't exit successfully:
remote:        `/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/target/release/build/rb-sys-0b34e8f38d6837b7/build-script-main`
remote:        (exit status: 101)
remote:          --- stdout
remote:          cargo:rerun-if-env-changed=RUBY
remote:          cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
remote:          cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
remote:          cargo:rerun-if-env-changed=RBCONFIG_platform
remote:          cargo:rerun-if-env-changed=RUBY_ROOT
remote:          cargo:rerun-if-env-changed=RUBY_VERSION
remote:          cargo:rerun-if-env-changed=RUBY
remote:          cargo:rerun-if-changed=build/features.rs
remote:          cargo:rerun-if-changed=build/version.rs
remote:          cargo:rerun-if-changed=build/stable_api_config.rs
remote:          cargo:rerun-if-changed=build/main.rs
remote:          cargo:rerun-if-env-changed=RUBY_STATIC
remote:          cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
remote:          cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
remote:          cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
remote:          cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
remote:          cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
remote:          cargo:rerun-if-env-changed=TARGET
remote:          cargo:rerun-if-env-changed=TARGET
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
remote:
remote:          --- stderr
remote:        INFO: using bindgen with clang args:
remote:        ["-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0",
remote:        "-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0/x86_64-linux",
remote:        "-fms-extensions", "-O3", "-fno-fast-math", "-g", "-Wall", "-Wextra",
remote:        "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond",
remote:        "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith",
remote:        "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0",
remote:        "-Wmissing-noreturn", "-Wno-cast-function-type",
remote:        "-Wno-constant-logical-operand", "-Wno-long-long",
remote:        "-Wno-missing-field-initializers", "-Wno-overlength-strings",
remote:        "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign",
remote:        "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value",
remote:        "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn",
remote:        "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"]
remote:        thread 'main' panicked at
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31:
remote:        Unable to find libclang: "couldn't find any valid shared libraries matching:
remote:        ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the
remote:        `LIBCLANG_PATH` environment variable to a path where one of these files can be
remote:        found (invalid: [])"
remote:          note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
remote:        warning: build failed, waiting for other jobs to finish...
remote:        make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101
remote:
remote:        make failed, exit code 2
remote:
remote:        Gem files will remain installed in
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c
remote:        for inspection.
remote:        Results logged to
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/extensions/x86_64-linux/3.3.0/tiktoken_ruby-d7bdeb57336c/gem_make.out
remote:
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:125:in
remote:        `run'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:51:in
remote:        `block in make'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in
remote:        `each'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in
remote:        `make'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/ext_conf_builder.rb:42:in
remote:        `build'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:193:in
remote:        `build_extension'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:227:in
remote:        `block in build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in
remote:        `each'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in
remote:        `build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/installer.rb:852:in
remote:        `build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:76:in
remote:        `build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path/installer.rb:28:in
remote:        `post_install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path.rb:245:in
remote:        `generate_bin'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/git.rb:202:in
remote:        `install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:54:in
remote:        `install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:16:in
remote:        `install_from_spec'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:132:in
remote:        `do_install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:123:in
remote:        `block in worker_pool'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:62:in
remote:        `apply_func'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:57:in
remote:        `block in process_queue'
remote:          <internal:kernel>:187:in `loop'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:54:in
remote:        `process_queue'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:90:in
remote:        `block (2 levels) in create_threads'
remote:
remote:        An error occurred while installing tiktoken_ruby (0.0.6), and Bundler cannot
remote:        continue.
remote:
remote:        In Gemfile:
remote:          tiktoken_ruby
remote:        Bundler Output: Fetching https://github.com/StileEducation/tiktoken_ruby.git
remote:        Fetching https://github.com/StileEducation/tiktoken_ruby.git
remote:        Fetching gem metadata from https://rubygems.org/.........
remote:        Fetching rb_sys 0.9.86
remote:        Installing rb_sys 0.9.86
remote:        Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
remote:
remote:        current directory:
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/bin/ruby extconf.rb
remote:        checking for gcc... yes
remote:        checking for g++... yes
remote:        checking for gcc-ar... yes
remote:        checking for cargo... no
remote:
remote:        current directory:
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby
remote:        make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m
remote:        sitelibdir\=./.gem.20240108-198-77bm6m clean
remote:
remote:        current directory:
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby
remote:        make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m
remote:        sitelibdir\=./.gem.20240108-198-77bm6m
remote:        info: downloading installer
remote:        info: profile set to 'minimal'
remote:        info: default host triple is x86_64-unknown-linux-gnu
remote:        info: skipping toolchain installation
remote:
remote:
remote:        Rust is installed now. Great!
remote:
remote:        To get started you need Cargo's bin directory
remote:        (/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
remote:        in your PATH
remote:        environment variable. This has not been done automatically.
remote:
remote:        To configure your current shell, run:
remote:        source
remote:        "/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
remote:        info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
remote:        info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)
remote:        info: downloading component 'cargo'
remote:        info: downloading component 'rust-std'
remote:        info: downloading component 'rustc'
remote:        info: installing component 'cargo'
remote:        info: installing component 'rust-std'
remote:        info: installing component 'rustc'
remote:
remote:        stable-x86_64-unknown-linux-gnu installed - rustc 1.75.0 (82e1608df
remote:        2023-12-21)
remote:
remote:        info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
remote:        info: checking for self-update
remote:        info: using existing install for 'stable-x86_64-unknown-linux-gnu'
remote:        info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
remote:
remote:        stable-x86_64-unknown-linux-gnu unchanged - rustc 1.75.0 (82e1608df
remote:        2023-12-21)
remote:
remote:        info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in
remote:        use (environment override by RUSTUP_TOOLCHAIN)
remote:        generating target/release/libtiktoken_ruby.so (release)
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo
remote:        rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release
remote:        -- -C linker=gcc -L native=/tmp/build_91a6107a/vendor/ruby-3.3.0/lib -C
remote:        link-arg=-lm -l pthread
remote:            Updating crates.io index
remote:            Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
remote:            Updating git submodule `https://github.com/zurawiki/tiktoken`
remote:         Downloading crates ...
remote:          Downloaded base64 v0.21.0
remote:          Downloaded once_cell v1.17.1
remote:          Downloaded bstr v1.4.0
remote:          Downloaded smallvec v1.10.0
remote:          Downloaded bit-vec v0.6.3
remote:          Downloaded cexpr v0.6.0
remote:          Downloaded lock_api v0.4.9
remote:          Downloaded syn v2.0.31
remote:          Downloaded shlex v1.1.0
remote:          Downloaded serde v1.0.157
remote:          Downloaded seq-macro v0.3.5
remote:          Downloaded proc-macro2 v1.0.66
remote:          Downloaded unicode-ident v1.0.8
remote:          Downloaded shell-words v1.1.0
remote:          Downloaded scopeguard v1.1.0
remote:          Downloaded regex-syntax v0.6.28
remote:          Downloaded rb-sys v0.9.81
remote:          Downloaded regex v1.7.1
remote:          Downloaded nom v7.1.3
remote:          Downloaded libc v0.2.140
remote:          Downloaded bit-set v0.5.3
remote:          Downloaded bindgen v0.66.1
remote:          Downloaded rb-sys-env v0.1.2
remote:          Downloaded rb-sys-build v0.9.81
remote:          Downloaded quote v1.0.33
remote:          Downloaded peeking_take_while v0.1.2
remote:          Downloaded parking_lot_core v0.9.7
remote:          Downloaded parking_lot v0.12.1
remote:          Downloaded minimal-lexical v0.2.1
remote:          Downloaded memchr v2.5.0
remote:          Downloaded magnus-macros v0.6.0
remote:          Downloaded magnus v0.6.1
remote:          Downloaded libloading v0.7.4
remote:          Downloaded lazycell v1.3.0
remote:          Downloaded lazy_static v1.4.0
remote:          Downloaded glob v0.3.1
remote:          Downloaded clang-sys v1.6.0
remote:          Downloaded bitflags v2.4.0
remote:          Downloaded autocfg v1.1.0
remote:          Downloaded anyhow v1.0.70
remote:          Downloaded aho-corasick v0.7.20
remote:          Downloaded fancy-regex v0.11.0
remote:          Downloaded rustc-hash v1.1.0
remote:          Downloaded regex-automata v0.1.10
remote:          Downloaded cfg-if v1.0.0
remote:           Compiling memchr v2.5.0
remote:           Compiling proc-macro2 v1.0.66
remote:           Compiling glob v0.3.1
remote:           Compiling unicode-ident v1.0.8
remote:           Compiling clang-sys v1.6.0
remote:           Compiling libc v0.2.140
remote:           Compiling quote v1.0.33
remote:           Compiling cfg-if v1.0.0
remote:           Compiling minimal-lexical v0.2.1
remote:           Compiling syn v2.0.31
remote:           Compiling nom v7.1.3
remote:           Compiling libloading v0.7.4
remote:           Compiling aho-corasick v0.7.20
remote:           Compiling regex-syntax v0.6.28
remote:           Compiling bindgen v0.66.1
remote:           Compiling regex v1.7.1
remote:           Compiling cexpr v0.6.0
remote:           Compiling bitflags v2.4.0
remote:           Compiling lazy_static v1.4.0
remote:           Compiling rustc-hash v1.1.0
remote:           Compiling shlex v1.1.0
remote:           Compiling peeking_take_while v0.1.2
remote:           Compiling lazycell v1.3.0
remote:           Compiling shell-words v1.1.0
remote:           Compiling autocfg v1.1.0
remote:           Compiling lock_api v0.4.9
remote:           Compiling rb-sys-build v0.9.81
remote:           Compiling parking_lot_core v0.9.7
remote:           Compiling rb-sys v0.9.81
remote:           Compiling smallvec v1.10.0
remote:        error: failed to run custom build command for `rb-sys v0.9.81`
remote:
remote:        Caused by:
remote:        process didn't exit successfully:
remote:        `/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/target/release/build/rb-sys-0b34e8f38d6837b7/build-script-main`
remote:        (exit status: 101)
remote:          --- stdout
remote:          cargo:rerun-if-env-changed=RUBY
remote:          cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
remote:          cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
remote:          cargo:rerun-if-env-changed=RBCONFIG_platform
remote:          cargo:rerun-if-env-changed=RUBY_ROOT
remote:          cargo:rerun-if-env-changed=RUBY_VERSION
remote:          cargo:rerun-if-env-changed=RUBY
remote:          cargo:rerun-if-changed=build/features.rs
remote:          cargo:rerun-if-changed=build/version.rs
remote:          cargo:rerun-if-changed=build/stable_api_config.rs
remote:          cargo:rerun-if-changed=build/main.rs
remote:          cargo:rerun-if-env-changed=RUBY_STATIC
remote:          cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
remote:          cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
remote:          cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
remote:          cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
remote:          cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
remote:          cargo:rerun-if-env-changed=TARGET
remote:          cargo:rerun-if-env-changed=TARGET
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
remote:          cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
remote:
remote:          --- stderr
remote:        INFO: using bindgen with clang args:
remote:        ["-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0",
remote:        "-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0/x86_64-linux",
remote:        "-fms-extensions", "-O3", "-fno-fast-math", "-g", "-Wall", "-Wextra",
remote:        "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond",
remote:        "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith",
remote:        "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0",
remote:        "-Wmissing-noreturn", "-Wno-cast-function-type",
remote:        "-Wno-constant-logical-operand", "-Wno-long-long",
remote:        "-Wno-missing-field-initializers", "-Wno-overlength-strings",
remote:        "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign",
remote:        "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value",
remote:        "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn",
remote:        "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"]
remote:        thread 'main' panicked at
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31:
remote:        Unable to find libclang: "couldn't find any valid shared libraries matching:
remote:        ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the
remote:        `LIBCLANG_PATH` environment variable to a path where one of these files can be
remote:        found (invalid: [])"
remote:          note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
remote:        warning: build failed, waiting for other jobs to finish...
remote:        make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101
remote:
remote:        make failed, exit code 2
remote:
remote:        Gem files will remain installed in
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c
remote:        for inspection.
remote:        Results logged to
remote:        /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/extensions/x86_64-linux/3.3.0/tiktoken_ruby-d7bdeb57336c/gem_make.out
remote:
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:125:in
remote:        `run'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:51:in
remote:        `block in make'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in
remote:        `each'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in
remote:        `make'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/ext_conf_builder.rb:42:in
remote:        `build'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:193:in
remote:        `build_extension'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:227:in
remote:        `block in build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in
remote:        `each'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in
remote:        `build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/installer.rb:852:in
remote:        `build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:76:in
remote:        `build_extensions'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path/installer.rb:28:in
remote:        `post_install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path.rb:245:in
remote:        `generate_bin'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/git.rb:202:in
remote:        `install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:54:in
remote:        `install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:16:in
remote:        `install_from_spec'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:132:in
remote:        `do_install'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:123:in
remote:        `block in worker_pool'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:62:in
remote:        `apply_func'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:57:in
remote:        `block in process_queue'
remote:          <internal:kernel>:187:in `loop'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:54:in
remote:        `process_queue'
remote:        /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:90:in
remote:        `block (2 levels) in create_threads'
remote:
remote:        An error occurred while installing tiktoken_ruby (0.0.6), and Bundler cannot
remote:        continue.
remote:
remote:        In Gemfile:
remote:          tiktoken_ruby
remote:
remote:  !
remote:  !     Failed to install gems via Bundler.
remote:  !
remote:  !     Push rejected, failed to compile Ruby app.
remote:
remote:  !     Push failed
remote:  !
remote:  ! ## Warning - The same version of this code has already been built: 5888659e95baa9ba3bb99063346e68ec861aa781
remote:  !
remote:  ! We have detected that you have triggered a build from source code with version 5888659e95baa9ba3bb99063346e68ec861aa781
remote:  ! at least twice. One common cause of this behavior is attempting to deploy code from a different branch.
remote:  !
remote:  ! If you are developing on a branch and deploying via git you must run:
remote:  !
remote:  !     git push heroku <branchname>:main
remote:  !
remote:  ! This article goes into details on the behavior:
remote:  !   https://devcenter.heroku.com/articles/duplicate-build-version
remote:
remote: Verifying deploy...
remote:
remote: !	Push rejected to tiktoken-ruby.
remote:
To https://git.heroku.com/tiktoken-ruby.git
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/tiktoken-ruby.git'

[djrodgerspryor]

@karlentwistle you will still need to install libclang

[andreaslillebo]

Does anyone know a good way to install libclang on Heroku? It appears to no longer be included in the build environment on the heroku-22 stack.

[karlentwistle]

@karlentwistle you will still need to install libclang

Ah, I see, thanks very much for letting me know 🙇

Does anyone know a good way to install libclang on Heroku? It appears to no longer be included in the build environment on the heroku-22 stack.

I got it working with karlentwistle/tiktoken_heroku_example@583a5c7 and https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-apt

---

Can also now confirm this PR does indeed fix #20 😄

[xfalcox]

Howdy @IAPark, this is blocking projects from moving to Ruby 3.3. Any chance this can be merged?

[ScotterC]

Curious. Why is libclang required now when it wasn't on Ruby 3.2? Did Ruby 3.2 handle it or did Tiktoken not require the dependency?

[bmulholland]

@ScotterC Maybe because there's a prebuilt version for ruby 3.2, but not 3.3? Ref: #20 (comment)

[ScotterC]

Yup here's where the builds are created https://github.com/IAPark/tiktoken_ruby/blob/main/.github/workflows/build.yml#L9. If I understood more clearly how that build ends up creating the precompiled version on rubygems or locally then I'd create a PR for it. I'm loathe to add another buildpack to Heroku since I'm feeling the platform lock-in lately.

[xfalcox]

Did anyone here fork the gem for Ruby 3.3 ?

[ScotterC]

@xfalcox only forks updated recently are similar to this PR. Updating rb_sys. Here's thoughtbot's https://github.com/thoughtbot/tiktoken_ruby

[IAPark]

Closed in favor of #22 (kinda by accident)

[IAPark]

Also, the truth is I created this gem in a weekend because I was annoyed at work and we've sense moved in a direction where it's irrelevant. It kinda just gives me a bit of stress now when I'm reminded of it. I think I should probably put a notice up asking if anyone wants to take it over

[IAPark]

In theory this should be out now though cross compiling seems broken and I might not be able to get back to it until next weekend at the earliest

[ScotterC]

@IAPark thanks for the work! I'll see if I can help find a maintainer