You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ruby | Support installing the gem via git and some other small build tweaks (#21061)
## Overview
This PR introduces a couple of small tweaks to the Ruby `google-protobuf` gem build. We've confirmed that the packaged `.gem` that gets created as part of `bazel build //ruby:release` is identical to that in the latest `google-protobuf` Ruby gem release. We also had an internal review of these changes, and if it's helpful, that can be found here: Shopify#12
## Make the gem installable via git
Ruby `Gemfile`s typically allow you to define a `git` source for gems to easily be able to test custom branches. However, given the following in a `Gemfile`:
```
# Gemfile
gem "google-protobuf", git: "https://github.com/Shopify/protobuf.gif", branch: "master", submodules: true
```
we get the following build output when we try to `bundle install`:
```
Fetching https://github.com/Shopify/protobuf.git
Fetching gem metadata from https://pkgs.shopify.io/basic/gems/ruby/...
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Resolving dependencies...
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
/opt/rubies/3.4.1/bin/ruby extconf.rb
creating Makefile
current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i clean
current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i
compiling protobuf.c
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:1018:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
1018 | *overrun = ptr - e->end;
| ~ ~~~~^~~~~~~~
./ruby-upb.h:1151:64: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1151 | upb_EpsCopyInputStream_CheckDataSizeAvailable(e, ptr, size);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
./ruby-upb.h:1222:65: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1222 | if (!upb_EpsCopyInputStream_CheckDataSizeAvailable(e, *ptr, size)) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
./ruby-upb.h:1228:66: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1228 | const char* ret = upb_EpsCopyInputStream_Copy(e, *ptr, data, size);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
./ruby-upb.h:1314:27: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
1314 | e->limit = e->limit_ptr - e->end;
| ~ ~~~~~~~~~~~~~^~~~~~~~
./ruby-upb.h:1390:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
1390 | return a.size - b.size;
| ~~~~~~ ~~~~~~~^~~~~~~~
./ruby-upb.h:4358:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
4357 | array = UPB_PRIVATE(_upb_Array_New)(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~
4358 | arena, 4, UPB_PRIVATE(_upb_MiniTableField_ElemSizeLg2)(f));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./ruby-upb.h:275:24: note: expanded from macro 'UPB_PRIVATE'
275 | #define UPB_PRIVATE(x) x##_dont_copy_me__upb_internal_use_only
| ^
<scratch space>:19:1: note: expanded from here
19 | _upb_MiniTableField_ElemSizeLg2_dont_copy_me__upb_internal_use_only
| ^
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:14850:10: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32]
14850 | *tag = val;
| ~ ^~~
./ruby-upb.h:14886:11: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'int' [-Wshorten-64-to-32]
14886 | *size = size64;
| ~ ^~~~~~
./ruby-upb.h:15299:10: fatal error: 'utf8_range.h' file not found
15299 | #include "utf8_range.h"
| ^~~~~~~~~~~~~~
9 warnings and 1 error generated.
make: *** [protobuf.o] Error 1
make failed, exit code 2
Gem files will remain installed in /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby for inspection.
Results logged to /Users/dave/.gem/ruby/3.4.1/bundler/gems/extensions/arm64-darwin-23/3.4.0/protobuf-8b63023562e0-ruby/gem_make.out
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in 'Gem::Ext::Builder.run'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in 'block in Gem::Ext::Builder.make'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Array#each'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Gem::Ext::Builder.make'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in 'Gem::Ext::Builder#build_extension'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in 'block in Gem::Ext::Builder#build_extensions'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Array#each'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Gem::Ext::Builder#build_extensions'
/opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/rubygems_gem_installer.rb:111:in 'Bundler::RubyGemsGemInstaller#build_extensions'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path/installer.rb:28:in 'Bundler::Source::Path::Installer#post_install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path.rb:234:in 'Bundler::Source::Path#generate_bin'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/git.rb:212:in 'Bundler::Source::Git#install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:55:in 'Bundler::GemInstaller#install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:17:in 'Bundler::GemInstaller#install_from_spec'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:133:in 'Bundler::ParallelInstaller#do_install'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:124:in 'block in Bundler::ParallelInstaller#worker_pool'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue'
<internal:kernel>:168:in 'Kernel#loop'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue'
/Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads'
An error occurred while installing google-protobuf (4.31.0), and Bundler cannot continue.
In Gemfile:
grpc was resolved to 1.71.0, which depends on
googleapis-common-protos-types was resolved to 1.19.0, which depends on
google-protobuf
```
This is because building the `protobuf_c` Ruby C extension requires the `copy_third_party` rake task to copy the `third_party/utf8_range` library into `ruby/ext/google/protobuf_c`. Fortunately, the `extensions` array in a `Gemfile` can accept either a relative path to an `extconf.rb`, or a `Rakefile`. In the case of the latter, it runs the default `rake`, which in the case of `google-protobuf` is what we need to successfully build the native bindings. The `bazel` build process takes care of these steps which is why `bazel build //ruby:release` works as expected.
However, this `Rakefile` contains tasks that reference files outside of `ruby/`, which won't exist when installing the packaged `.gem` (the artifact that gets published to rubygems.org). The changes here will conditionally set the relevant `extensions` entry to either `ext/google/protobuf_c/extconf.rb` or `Rakefile` depending on the broader build context. This also required a small tweak in the `Rakefile` to modify the `Gem::PackageTask`.
## Missing `google/protobuf/plugin_pb.rb` in `rake gem` artifact
Another issue we noticed is the compiled `src/google/protobuf/compiler/plugin.proto` is absent in the packaged `.gem` that gets outputted when following the [Ruby development build instructions](https://github.com/protocolbuffers/protobuf/tree/main/ruby#installation-from-source-building-gem). This is because this particular proto file was absent from the `well_known_protos` definition in the `Rakefile`. This adds that missing proto to the list of well known protos and makes a small tweak to ensure it ends up in the gem's `lib/google/protobuf` directory instead of `lib/google/protobuf/compiler`. I'm not entirely sure why that directory tree gets a bit flattened, but we wanted to ensure the output remained consistent. Again, this is another issue that the `bazel` build takes care of.
## Tweak `ruby/BUILD.bazel` to also work on macOS
Finally, this PR introduces a small change to `ruby/BUILD.bazel` so `bazel build //ruby:release` can run on macOS as well as Linux. `cp --parents` is not supported in BSD's it seems.
Closes#21061
COPYBARA_INTEGRATE_REVIEW=#21061 from Shopify:ruby-installable-from-git e6b1f0c
PiperOrigin-RevId: 744700849
0 commit comments