Ruby | Support installing the gem via git and some other small build tweaks

[davebenvenuti]

Overview

This PR introduces a couple of small tweaks to the Ruby google-protobuf gem build. We've confirmed that the packaged .gem that gets created as part of bazel build //ruby:release is identical to that in the latest google-protobuf Ruby gem release. We also had an internal review of these changes, and if it's helpful, that can be found here: Shopify#12

Make the gem installable via git

Ruby Gemfiles typically allow you to define a git source for gems to easily be able to test custom branches. However, given the following in a Gemfile:

# Gemfile
gem "google-protobuf", git: "https://github.com/Shopify/protobuf.gif", branch: "master", submodules: true

we get the following build output when we try to bundle install:

Fetching https://github.com/Shopify/protobuf.git
Fetching gem metadata from https://pkgs.shopify.io/basic/gems/ruby/...
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Resolving dependencies...
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
/opt/rubies/3.4.1/bin/ruby extconf.rb
creating Makefile

current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i clean

current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i
compiling protobuf.c
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:1018:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
 1018 |   *overrun = ptr - e->end;
      |            ~ ~~~~^~~~~~~~
./ruby-upb.h:1151:64: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1151 |          upb_EpsCopyInputStream_CheckDataSizeAvailable(e, ptr, size);
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~         ^~~~
./ruby-upb.h:1222:65: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1222 |     if (!upb_EpsCopyInputStream_CheckDataSizeAvailable(e, *ptr, size)) {
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~          ^~~~
./ruby-upb.h:1228:66: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1228 |     const char* ret = upb_EpsCopyInputStream_Copy(e, *ptr, data, size);
      |                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~                ^~~~
./ruby-upb.h:1314:27: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
 1314 |   e->limit = e->limit_ptr - e->end;
      |            ~ ~~~~~~~~~~~~~^~~~~~~~
./ruby-upb.h:1390:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1390 |     return a.size - b.size;
      |     ~~~~~~ ~~~~~~~^~~~~~~~
./ruby-upb.h:4358:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 4357 |     array = UPB_PRIVATE(_upb_Array_New)(
      |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4358 |         arena, 4, UPB_PRIVATE(_upb_MiniTableField_ElemSizeLg2)(f));
      |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./ruby-upb.h:275:24: note: expanded from macro 'UPB_PRIVATE'
  275 | #define UPB_PRIVATE(x) x##_dont_copy_me__upb_internal_use_only
      |                        ^
<scratch space>:19:1: note: expanded from here
   19 | _upb_MiniTableField_ElemSizeLg2_dont_copy_me__upb_internal_use_only
      | ^
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:14850:10: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32]
 14850 |   *tag = val;
       |        ~ ^~~
./ruby-upb.h:14886:11: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'int' [-Wshorten-64-to-32]
 14886 |   *size = size64;
       |         ~ ^~~~~~
./ruby-upb.h:15299:10: fatal error: 'utf8_range.h' file not found
 15299 | #include "utf8_range.h"
       |          ^~~~~~~~~~~~~~
9 warnings and 1 error generated.
make: *** [protobuf.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby for inspection.
Results logged to /Users/dave/.gem/ruby/3.4.1/bundler/gems/extensions/arm64-darwin-23/3.4.0/protobuf-8b63023562e0-ruby/gem_make.out

  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in 'Gem::Ext::Builder.run'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in 'block in Gem::Ext::Builder.make'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Array#each'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Gem::Ext::Builder.make'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in 'Gem::Ext::Builder#build_extension'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in 'block in Gem::Ext::Builder#build_extensions'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Array#each'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Gem::Ext::Builder#build_extensions'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/rubygems_gem_installer.rb:111:in 'Bundler::RubyGemsGemInstaller#build_extensions'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path/installer.rb:28:in 'Bundler::Source::Path::Installer#post_install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path.rb:234:in 'Bundler::Source::Path#generate_bin'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/git.rb:212:in 'Bundler::Source::Git#install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:55:in 'Bundler::GemInstaller#install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:17:in 'Bundler::GemInstaller#install_from_spec'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:133:in 'Bundler::ParallelInstaller#do_install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:124:in 'block in Bundler::ParallelInstaller#worker_pool'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue'
  <internal:kernel>:168:in 'Kernel#loop'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads'

An error occurred while installing google-protobuf (4.31.0), and Bundler cannot continue.

In Gemfile:
  grpc was resolved to 1.71.0, which depends on
    googleapis-common-protos-types was resolved to 1.19.0, which depends on
      google-protobuf

This is because building the protobuf_c Ruby C extension requires the copy_third_party rake task to copy the third_party/utf8_range library into ruby/ext/google/protobuf_c. Fortunately, the extensions array in a Gemfile can accept either a relative path to an extconf.rb, or a Rakefile. In the case of the latter, it runs the default rake, which in the case of google-protobuf is what we need to successfully build the native bindings. The bazel build process takes care of these steps which is why bazel build //ruby:release works as expected.

However, this Rakefile contains tasks that reference files outside of ruby/, which won't exist when installing the packaged .gem (the artifact that gets published to rubygems.org). The changes here will conditionally set the relevant extensions entry to either ext/google/protobuf_c/extconf.rb or Rakefile depending on the broader build context. This also required a small tweak in the Rakefile to modify the Gem::PackageTask.

Missing google/protobuf/plugin_pb.rb in rake gem artifact

Another issue we noticed is the compiled src/google/protobuf/compiler/plugin.proto is absent in the packaged .gem that gets outputted when following the Ruby development build instructions. This is because this particular proto file was absent from the well_known_protos definition in the Rakefile. This adds that missing proto to the list of well known protos and makes a small tweak to ensure it ends up in the gem's lib/google/protobuf directory instead of lib/google/protobuf/compiler. I'm not entirely sure why that directory tree gets a bit flattened, but we wanted to ensure the output remained consistent. Again, this is another issue that the bazel build takes care of.

Tweak ruby/BUILD.bazel to also work on macOS

Finally, this PR introduces a small change to ruby/BUILD.bazel so bazel build //ruby:release can run on macOS as well as Linux. cp --parents is not supported in BSD's it seems.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[JasonLunn]

The co-author of f96caa1 needs to sign the CLA as well before we can review this PR.

[davebenvenuti]

I believe @rwstauner already has but the commit message had the wrong email address. I've amended the commit, let's see if this resolves it....