@fiedl Out-of-the-box webpacker doesn't support engines yet. There are quite a few moving pieces that needs to be considered for this setup to work properly and usually people will have different use cases. I started doing some work on this but it's going to take time.

Feel free to leave any ideas/suggestions 👍

@gauravtiwari Yes, after diving into this, I see your point. One would have to decide for each component where it belongs---to the engine, the main app, or both---and tie the pieces together in the right manner.

Some thoughts on this:

• The bin/webpack-dev-server is needed in the main app. One could add it to the Procfile of the main app to start the dev server with Foreman. Is there already a way to have the webpack dev server started by Pow?
• But one would need the ability to add a package from the engine.
• The app/javascript files have to be loaded from both, the engine and the main app. The main app files need to have the final say. I guess if there are several engines, the files from all engines need to be loaded.
• There are some files like .babelrc, .postcssrc.yml that would seem right in the main app root folder; but I'm not sure what these files are doing, yet.
• The config/webpack files appear to be tricky. In my use case, I would like to minimise the work to be done in the main app. But for other use cases, I imagine, the main app would need more control over the configuration. I really like how initializers work. I have most of them in the engine, but those of the main app have the last say in case of a conflict. Maybe we could utilise this mechanism: Would it make sense to tell webpacker in an initializer where to look for the webpack config? If no initializer is present, just use config/webpack from the rails root folder.
• For package.json and yarn.lock, one would need a mechanism similar to what is used with the *.gemspec, Gemfile and Gemfile.lock. If the engine decides that a certain package version is required, the package will be updated together with the engine when running bundle update my_engine from the main app. But the last say regarding the package versions has the main app as they are locked in the lock file. After all, the yarn.lock looks like a Gemfile.lock.
• This leads to the question how the upgrade process would look like from the view of the main app. I guess, just running bundle install from the main app would not cover any javascript package updates from the engine.

@gauravtiwari I started adding webpacker support to ManageIQ, where the UI is just an engine (so, app root != UI root), and we may need to support reading assets from multiple engines, and we need to output them to the rails root folder, not engine root.

To achieve that, I needed to override most of the webpacker methods to use the engine root instead of hardcoding Rails.root, I also needed to add a rake task that outputs Rails.root and call that from webpack config (didn't find another way of providing that path to webpack). (This also means that we need to include the initializer that overrides those paths from our Rakefile, which is a bit unfortunate :).)

.. And I still haven't figured out what would be needed to be able to call webpacker:compile, so far it it works if you manually call Webpacker.bootstrap but only when called from the root app, not when called from the engine.

I still haven't finished work on the "read assets from multiple engines" bit, but so far, I have a rake task that outputs a json of all the engines and their root paths (and filters those engines by existence of /app/javascript, until there's a proper way to register this), and intending to read that in the webpack config.

So.. I think at least these changes would be needed:

• don't assume all paths are relative to the current directory
• don't hardcode Rails.root anywhere
• come up with a sane way of passing those engine paths all the way to JS

If you're interested in the changes that we needed (and feel free to criticize and/or make suggestions), ManageIQ/manageiq-ui-classic#1132 .

Furthermore (and these are only my personal opinion):

• we should default to expecting the assets compiled, UI devels can enable webpack-dev-server, and for the rest, it should Just Work as long as they don't make UI changes.
• we should never deal with yarn, yarn and webpack are separate tools, and webpacker's job starts when you have all the assets, not before
• all those safety checks (compile depends on verify_install, verify_install depends on check_node and check_yarn, ..) are out of scope and needlessly complicate these tasks. All we need is to see the real error when it fails.

EDIT: Oh, and if there was a nice way of disabling the assets:precompile hook, it would be lovely, when wrapping that task, you may want the other taks to run on precompile :).

EDIT2: Aand making all the compile-time yarn deps as --dev, so that non-dev is reserved for actual UI dependencies, but I guess that's a separate issue :)

Right now i'm doing this, but obviously the solution is quite incomplete. Here is what I'm doing, what I see as the potential solution

by default wepbacker will always look for the manifest in the /public/packs folder of the Root application, that's because the default webpacker.yml is configured as such. However when coming from an engine environment it should respect the setting of the engine's webpacker.yml as well. which means webpacker needs to be aware of multiple namespaces.

For webpacker to be successful in an engine environment we need things to be isolated like every other part.

Which means, when we install the engine into an app, it needs to have an 'install step' that basically installs the pre-built modules into the rails app /public/packs/engine/name/manifest.json

We will probably need to modify the stylesheet_pack_tag and javascript_pack_tag helper to be aware of namespaces. So basically if you want to use the engine's assets you should be able to do something like javascript_pack_tag 'engine/name/vendor' etc... or javascript_pack_tag 'vendor', module: 'Engine::Name' which resolves to /public/packs/engine/name/vendor.js

I will have more clarity on this as I develop the solution. right now for me developing the engine with JS code I am just using a symlink from the dummy app's public/packs folder to the engine's public/packs folder, which works for development purposes.

A question for @dhh is also that should the Root app using the engine have to worry about compiling an engine's assets? If not it should make thing simpler since we would expect the JS code from the engine to already be compiled and usable which would mean we just need to make the javascript_pack_tag and stylesheet_pack_tag aware of those assets based on namespaces. or the module: 'Engine::Name' option that gets passed in.

I've made a POC using webpack and rails engines that I believe that can be reused to solve this request: see ohadlevy/foreman@2afa796 for a reference.

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman project.
they key was to create a simple ruby script that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths

Should the Root app using the engine have to worry about compiling an engine's assets?

On the surface, it's a fair argument that an asset engine (which is essentially what a webpack app gem is) could be expected to have all the assets pre-compiled, because... once it's released the assets aren't going to change... but looking at how assets in engines currently work, that is not the case, probably to allow including uncompiled assets within other uncompiled assets in the Root app (e.g. application.scss could use mixins from an engine's scss files) if that's the intent of the engine. I'm not totally sure whether that would ever be the case with a webpack engine I guess... is it feasible that there might be a use case for an engine that provides pieces to be used within a larger webpack app? Perhaps the engine would expose some Elm module or React component, and the Root app's UI would utilize that component from its own Elm/React app?

In Elm at least, that actually doesn't make sense as a use case, because all modules have to be explicitly included in the elm-package.json, and will then be added to the dependency graph independently of any Rails Engine. But I'm not sure the same can be said of every potential webpacked app - for instance, what if your Engine's app is just straight ES6 that exposes some modules? Could the Root app import those modules from the engine to be used as potential internal components of it's reactive UI?

If that kind of integration is possible (and useful, which seems likely to me if it's possible), then in order to facilitate it, a Root app obviously needs to be able to compile the engine's assets (after they are included and used however they are going to be used).

My main argument against allowing this type of integration, through a Rails Engine specifically, would be that it's mixing what are becoming ever more the separate responsibilities of the server-side app and the client-side app. That is, if your reactive UI requires some component, it should import it using a javascript dependency manager like npm, whereas the Gemfile should manage dependencies for your server-side Rails app (which may include some compiled reactive UI app to be used on the front-end, like any other asset to be sent to the client)

My question is: is that a good enough argument to make it impossible using webpacker (by assuming there will be no precompiled integration between the UI components of an engine and its root app, and hence not providing any considerations to facilitate such integration)?

I was originally patiently waiting for this kind of support so I could upgrade a gem I manage to be used with webpacker, but @mltsy's argument resonated pretty well with me.

Javascript provided by an engine to be used through webpacker could just be put in (an) npm package(s). This would also allow the javascript to become part of the npm ecosystem, becoming a dependency or depending on other packages.

The engine could still provide CSS and JS when needed through sprockets. One downside is the potential overlap of an engine providing a page that depends on javascript that is also provided in its npm package. It could lead to a few use-cases of an end user having to download the same javascript code from multiple locations

@mltsy @tomprats I partially agree with this view.

However, let's say the engine is a pluggable CMS that defines its own JS dependencies (similarly to gems in .gemspec). I would hope there would be a way for the main Rails app to pickup these dependencies as well.

As of now, in my case, most of the JS dependencies defined in a Rails engine are facilitated via https://rails-assets.org. It would be fantastic to be able to define them via webpacker.

Yeah - this is a fairly unique situation where we have essentially two dependency managers in a single application (bundler and npm), and specifically the situation you're describing, where we have a Rails app with a bundler dependency on a Rails Engine, and an npm dependency on the "webpack app" (for lack of a better term, assuming it's not published as an npm package) that is defined in the Rails Engine. And that's the sticking point - the Rails engine "depends on" the webpack app that is implemented in its javascript/packs directory, but that dependency isn't really part of bundler or npm. So it's hard to say how/if it should be exposed in the contexts of npm and bundler. If we expose that somehow, we would essentially be adding javascript dependencies via the ruby dependency manager, which confuses things a bit when it comes to the javascript dependency manager... (there are some things included via an external channel, which creates somewhat of a mess when it comes to dependency management).

The best solution I can think of to handle that would be to somehow create/expose a kind of meta-package (or legitimate package?) out of the webpack app(s) contained within the Rails Engine, and automatically add that package to the packages.json of the Rails app before compiling with webpack (or maybe when the Engine is installed?). But this is all very theoretical, so maybe I should just shut my mouth and let someone who has attempted something like this share their wisdom ;)

@mltsy @tomasc @tomprats @ohadlevy @soundasleep @fiedl:

The beta version of React on Rails is built on top of Webpacker and here are the docs for using it.

I just released v9 beta.1, built on top of:

• #601
• shakacode/react_on_rails#908

How about specifying more than one in source_path of webpacker.yml?

default: &default
  source_path:
    - app/javascript
    - engine/javascript
  source_entry_path: packs
  public_output_path: packs
  cache_path: tmp/cache/webpacker

Specify the entry point of the application and the entry point of the engine, and build it with the application.
If it is dynamic, it will be glad even if it can be done dynamically with ERB etc.

Thanks @chimame, looks like a good idea, I will give it a try and report back.

@chimame I have successfully done something similar to your suggestion. Since all the engines in my app are in vendor/gems, my source_path definition in webpacker.yml is:

source_path: '**/app/javascript'

And this pulls in the modules from all the engines as well as the main app.

This is working very well for me, except I had to hack a single line in the rails webpacker npm module, as the manifest.json keys were not correct. I would happily create a PR for this, but I think there may be other use cases I'm not thinking of. Also, although my hack doesn't break the tests, I don't think this line of code is covered by tests and it's not clear to me why it was written that way.

I have a small example app with this implemented at https://github.com/lazylester/webpack_example

Hmm... You're right! That's cool! At least partly... I mean that example app isn't quite the proof of concept that is necessary, because it doesn't actually require a module defined in the engine, but I bet it could!

I was slightly misunderstanding the role of npm and webpack. npm defines and downloads the sources (dependencies) from which webpack can choose what to include in any given pack. An engine is just another source for webpack to pull from, so it supplements the sources supplied by npm. Then webpack can choose from either npm's dependencies or the additional sources provided by the engine(s)...

Now... that is still a little bit disconcerting, just because those sources are outside of npm's dependency graph, meaning... the other thing we still need to solve is how to install the dependencies of the engine's pack(s). I see your "inventory" engine has no package.json (no JS dependencies) - that's not too likely in the real world, I imagine. Would you want to try adding and using one dependency and see if that works? I can't imagine running npm install on the main app would also install the engine's dependencies. But maybe there's a way to get around that... ? (I'm no expert in npm, so someone else here might have a better idea than I do about how to solve that issue)

@mitsy you're right, I don't currently require any modules in the engine. I have done it elsewhere but forgot to include it in the example app. I'll update the example and post here. However my engines do not have any of their own npm dependencies, just local modules, within the engine, and a global (ractivejs) module that is in the main app's node modules.

So, yes, my approach has limitations. and doesn't cover many use cases. But it gets me to production!

@justin808 - now that I understand what I'm looking for more I looked through the docs for using react_on_rails to see how you're handling this issue too. I see you're using a rails task to run a command provided by the engine (configuration.build_production_command) to compile the engine's pack, but (and this may be my ignorance about webpack) I see two issues in using this as a general solution:

1. It looks like that just runs webpack... when do the dependencies get installed? Does webpack install them? Does the user need to install them before running the task? (I see a yarn build:production version of the command in the test suite, but it looks like that script is not actually implemented or used)

2. If that command runs webpack with a totally separate webpack.yml, that wouldn't allow the parent application to include any of the engine's modules, etc. right? It's just compiling a totally independent pack? I see there's a comment just above the command saying if you don't want the engine to build the pack for you, that command can be set to nil, but I don't see any documentation on how to compile your pack in that case. Have you designed a workflow for compiling a webpack in the parent app that uses components/modules from your engine?

Aha! Maybe we could use npm's local dependency feature?
https://docs.npmjs.com/files/package.json#local-paths

Yarn also respects this syntax... so we could tell the user to add this local dependency (file:vendor/gems/engine/? I'm not sure where the "package" root actually is - I would assume whatever directory the engine's package.json ends up in) to package.json in the parent app if they want to use any of the locally provided modules in their parent app's pack, and if not, provide a task for compiling the engine's pack independently! (could even make the installation part of an engine:install task)

I have to try this... when I get a chance... (if anybody else does, I'd love to hear the result)

@mltsy I think it's a good idea.

If both the idea to add to my webpacker.yml and the plan to add to your package.json come true, you can do the following.

• Write javascript_pack_tag 'engine_ javascript' in View (if it exists in webpacker.yml)
• import engine from 'engine_ javascript' can be defined in JavaScript (if it exists in package.json)

@mitsy I updated my example app https://github.com/lazylester/webpack_example.

The engine now has a local module of its own and npm modules. You'd probably have to build the packs for the engines in addition to the packs for the main app in a build script, but I assume that's not a big deal, since there has to be a build script anyway, right?

I'll generate a PR for the webpacker change that permitted this to work. It may or may not be accepted, depending on whether it breaks something else. As I mentioned before, the tests still pass.

Pull request: #875

Nice! So... I guess there are two use-cases for engines here which have different solutions:

1. I want to use an Engine to provide uncompiled modules for use in a parent app's pack(s)
2. I want to use an Engine to provide one or more compilable packs to a parent app

In case 2, that's what react_on_rails seems to be doing - it provides a rails task for compiling the engine's pack, and then you can use it like any other asset. @lazylester and @chimame's solutions are nicer ways to be able to do that without having to run a separate task, by including the engine's entry points in the webpacker sources of the main app. ⭐

The use case I'm looking to solve for my personal use is case 1 though. And now that I think about it, we don't even need to include the engine's entry points in the webpacker sources to enable that, we just need to depend on it as a local dependency in package.json right? If that's true, it makes this all very clean, since there are orthogonal/independent solutions to each use case, and each can be employed depending on the purpose of the engine. (I still have to try the local dependency thing though for case 1, and the specifics of automating case 2 still need to be ironed out, since neither suggestion currently works with webpacker)

Since they seem to be orthogonal concerns, I suggest we open a new ticket for one of these use cases, and clarify the title of this ticket to represent the other - but I'm not sure which one corresponds best to the original post here... @fiedl ?

#348 (comment)

I honestly will very much like to see engines supported by webpacker out of the box @dhh. It surely will make engines more modular or self-reliant. It's really sleek.

Here's how I got this working, loading Stimulus controllers from within a gem / engine and having them compile for Stimulus within the main Rails app.

# main_app/packages.json
{
  "dependencies": {
    "@rails/webpacker": "^3.2.1",
    "my_gem": "../../gems/my_gem", # Or you can point to a GitHub repo, etc.
    ...
  }
}

// main_app/application.js

import { Application } from "stimulus"
import { definitionsFromContext } from "stimulus/webpack-helpers"

// The usual stuff
const application = Application.start()
const context = require.context("./controllers", true, /\.js$/)
application.load(definitionsFromContext(context))

// Pointed at my gem
const gem_context = require.context('my_gem/app/javascript/packs/controllers', true, /\.js$/)
application.load(definitionsFromContext(gem_context))

Note: I had some trouble trying to figure out how to register these. Using the following made it easy to see what was registered when:

console.log("controller definitions:", definitionsFromContext(context))
console.log("controller definitions:", definitionsFromContext(gem_context))

Then in the gem:

# package.json
{
  "name": "my_gem",
  "version": "0.0.1",
  "description": "testing...",
  ...
}

Finally, I created my Stimulus controller(s) as usual:

import { Controller } from "stimulus"
export default class extends Controller {
  connect() {
    console.log("It's working...")
  }
  ...
}

Seems to do what I need. I'm happy for any feedback as to whether I've missed something, though..

I kind of got it working with @andrewhaines approach, but it worked only if I know the path to the engine. Which works only if the engine is in the host_app/gems/ folder, but this does not work when the engine path to the filesystem is unknown

The problem of course is that the relative/absolute path of an engine in the file system will often differ between development and production, or possibly on different systems; the bundler/rubygems system is not meant for this to be fixed and unchanging from system to system.

I kind of got my answers - I don't think I really like them, but I guess I would have to accept them.

https://discuss.rubyonrails.org/t/webpacker-presents-a-more-difficult-oob-experience-for-js-sprinkles-than-sprockets-did/75345/13

https://discuss.rubyonrails.org/t/how-to-pack-js-from-a-different-gem-engine-in-rails-6-webpacker-it-seems-frustrating/75364/8

My conclusion for the moment seems to be - if you have a non trivial .js feature you'd need a npm package and get the dependencies in a "webpackER/webpack" standard way.

So I would "just" split my engines in two - leave the controllers in the engine, move the js to npm packages, maintain two separate stacks, build procedures and versioning. No irony or sarcasm here, just listing what would probably be involved if others come to this conclusion also.

Yep. I think separate npm package is going to remain the only reasonable solution. (POSSIBLY changing if yarn plugins become a real thing).

I think we could really use tooling to make managing this easier.

Even Rails doesn't actually use separate versioning. There are npm packages for eg actioncable, activestorage, and even ujs-rails that are always released with same version numbers as the corresponding (and same-named) rails gems. If I understand right, every single ruby gem release gets a corresponding npm release with the same version number -- even if there have been no changes to the JS code in the npm package. The intent is that an app will always use the npm packages with the identical version to the ruby gem in use. And of course, all the code lives in the same repo for Rails.

One problem is that I'm not sure this is intent is well-documented, and there aren't (to my knowlege) any tools available to the developers of the app using these dependencies to make sure the npm and gem dependency versions are kept in lockstep sync. I suspect there are a lot of apps out there using mismatched versions -- which works, unless/until it someday doesn't, and when it doesn't it could hypothetically be very confusing to debug what's going on if you don't think to consider the issue. What if there were a tool that would ensure that your app had sync'd versions of the npm packages and the rubygems, and errored or warned you if it didn't? (ideally at bundle install and/or yarn install time, less ideally at app boot time, less ideally of all at JS use time).

I believe Rails does have tools on the side of Rails publishing itself to always and automatically publish the npm package(s) whenever rails ruby gems are published. But this tooling is tied into Rails custom code, it isn't easily available to someone else who wants to do similar. What if there were a tool that added on to the bundler-provided rake release task to make sure a corresponding npm package in the same repo had the correct lockstep sync'd version (a bit tricky because npm versioning uses slightly different 'syntax' then rubygems, especially when it comes to prereleases), and were automatically released?

I think with the right tooling, there'd be much minimized pain of "splitting engines in two". But as it is, there is significant increased challenges and more importantly scope for easy to make (and routinely made) mistakes, both to dependency maintainers and dependency users.

It might be more profitable to focus on such tooling to reduce the pain of a dependency split between a rubygem and an npm package, rather than working out what seems like inevitably hacky and unreliable methods of avoiding the split.

Thanks @jrochkind. I had this internal problem with versions many times, where the same "feature" is separated in different projects that have different versioning. My example is with a "feature" in the platform separated in three stacks - js front end+rails server+python data processing. I came to the conclusion that I don't need to keep them in sync. It's nice to have, but all I needed was a well maintained changelog and a couple of tests to make sure each version works. We later found out we are doing a completely different release cycles for the three parts and syncing them only on a major release which was working like a charm.

That being said I would of course be interested to use any tooling that connects npm and ruby closer. What I was missing, but kind of understood from this tread and a few others is sprockets is there to stay, but webpacker is the way to go as there as npm packages for actioncable, activestorage, rails-ujs and probably others.

Yes, I also believe separate NPM packages are the way to go for engines.

I just switched Alchemy today from the hack mentioned in the engines.md file to a separate NPM module and some Rails installer magic. This is a very clean approach that I am very happy with. I think other engines will follow.

Just have a look into that diff https://github.com/AlchemyCMS/alchemy_cms/pull/1853/files

🎉

@tvdeyen are you choosing to release npm packages with same versions as your ruby gems, with every release, as rails does? If so, do you have any tooling to make that harder to mess up?

Do you have anything in place to ensure the user is using an appropriate or compatible version of the npm gem, corresponding to the ruby gem version they are using? Or if not "ensure", to even let them figure out what versions are compatible?

These are the two areas I could see leading to problems, and could really use some 'best practices' or examples or recommendations on.

@jrochkind No, not yet. But I could imagine some kind of .alchemy-version.yml file (You will actually remember the famous spree_version.yml file aren't you 😆?

This is something that could probably live in Rails::Engine and be provided by Rails

@dhh how do you manage this in Rails gems and NPM modules? Is there something that could be make accessible for Engine authors? Would be great.

@jrochkind something like this? 🤔

Edit: Version that uses the actual resolved npm package version

initializer "alchemy.check_package_version" do
  yarn_list = `yarn list --silent --flat --depth 0 --pattern alchemy_cms`
  package_version = yarn_list.match(/\d\.\d\.\d/).to_s
  if Gem::Version.new(package_version) < Alchemy.gem_version 
    abort "Your Alchemy npm package is outdated, please run `yarn upgrade @alchemy_cms/admin`"
  end
end

There are still a couple of caveats here:

1. It assumes you use a triple digit version number (The regex can easily be adopted to use 2 or 4 digits)
2. It is pretty slow to run the yarn list command in a ruby process. This adds a couple of milliseconds (~1000ms on my machine) to your app start up time. Maybe we only make this check in Rails.env.development?
3. Assumes your npm package version string is compatible with Gem::Version
4. Assumes that the released npm package has always the same version of the engine. Maybe the gem author holds a list of compatible npm package versions in a separate list and compares against it?

WDYT?

Yeah, I think this approach makes sense, but I suspect there will be all sorts of edge case details to get right, it'll have to be worked out. I believe for pre-releases, different version strings are required.

I think it would be nice if there were an out of the box maintained solution to this, perhaps from webpacker or rails. I think it's currently these issues are currently the biggest additional pain to separating JS into an npm package; with them solved it could seem no more challenging than the old sprockets-focused solution of packaging your JS in the gem.

This is not perfect, but works pretty well.

It figures out what the paths are to the engines on the file system, and then adds them as entry points to the parent Webpacker.

Get paths to engines on file system

#! /usr/bin/env ruby
# file: get_engine_paths

require 'bundler'
require 'json'

# gem names in Gemfile
engine_names = ['my-gem-name']

engine_paths = Bundler.load.specs
  .select{ |dep| engine_names.include?(dep.name) }
  .map{ |dep| dep.to_spec.full_gem_path }

puts engine_paths.to_json

Needs to be executable (chmod +x get_engine_paths)

Make Webpacker find packs in engines

// file: webpack/environment.js
const { environment } = require("@rails/webpacker");

const { execSync } = require("child_process");
const { basename, resolve } = require("path");
const { readdirSync } = require("fs");

const babelLoader = environment.loaders.get("babel");

// Get paths to all engines' folders
const scriptPath = resolve(__dirname, "./get_engine_paths");
const buffer = execSync(scriptPath);
const enginePaths = JSON.parse(buffer);

enginePaths.forEach((path) => {
  const packsFolderPath = `${path}/app/javascript/packs`;
  const entryFiles = readdirSync(packsFolderPath);

  entryFiles.forEach((file) => {
    // File name without .js
    const name = basename(file, ".js");
    const entryPath = `${packsFolderPath}/${file}`;

    environment.entry.set(name, entryPath);
  });

  // Otherwise babel won't transpile the file
  babelLoader.include.push(`${path}/app/javascript`);
});

module.exports = environment;

Run yarn install in engines when yarn install is run in parent

Add to package.json "postinstall": "./engines_yarn_install.js"

#!/usr/bin/env node
// file: engines_yarn_install.js

const { spawn, execSync } = require("child_process");
const { resolve } = require("path");

// Get paths to all engines
const buffer = execSync(
  resolve(__dirname, "./get_engine_paths")
);
const enginePaths = JSON.parse(buffer);

enginePaths.forEach(enginePath => {
  spawn("yarn", ["install"], {
    env: process.env,
    cwd: enginePath,
    stdio: "inherit"
  });
});

@valterkraemer does that solution assume the path is always the same in every environment, or does it actually look it up "live"? At asset compile time? Asset delivery time?

Of course the path is usually diferent between a dev and production deploy, and sometimes can be different between different production deploys.

It looks it up at asset build time, so it should have no problem with changing paths.

@valterkraemer I have a similar approach, but it uses bundle info on the CLI to get a particular gem's information rather than relying on a pre-built JSON. It works pretty well on my own test app but I haven't actually done anything with it:

master...tubbo:add-installed-engines-to-additional-load-paths

You use it by calling the gem() function with the name of your gem, and adding to the module resolve paths in env config:

const { environment, gem } = require("@rails/webpacker")

environment.config.resolve.modules = [ gem("my-engine") ]

module.exports = environment

Then in your JS code:

import MyEngine from "my-engine"

This does require your code to be well namespaced, there's no automatic namespacing...it's just as if the files lived side by side in your app/javascript folder. So files in different gems with the same name would conflict with one another, and I suppose whichever file Webpack found first would be the one it used. But I think that's a decent trade-off for now.

@tubbo that's exciting. Do you think it would be feasible to turn it into a shareable package so other people can use this technique with share code? I guess it's functionality spans both JS and ruby, so maybe not?

Thanks @tubbo. Weren't aware of bundle show, can now use that to get rid of the get_engine_paths file!

Not sure if this'll help anyone but I had good success with the following. In my setup I have a folder components within the root of my Rails application that then contains a bunch of engines.

1. In your engine, create a directory: app/javascripts/engine_name (e.g. components/blogs/app/javascripts/blogs
2. Within that directory you can do what you want. In my case I had a controllers directory. Within that I had the typical Stimulus.js loader (index.js):

// Load all the controllers within this directory and all subdirectories.
// Controller files must be named *_controller.js.

import { Application } from "stimulus"
import { definitionsFromContext } from "stimulus/webpack-helpers"

const application = Application.start()
const context = require.context(".", true, /_controller\.js$/)
application.load(definitionsFromContext(context))

A slight difference in this one is the require.context(".") instead of require.context("controllers") as otherwise this will look at the main app's controllers directory.

3. In config/webpacker.yml add a resolved_paths (or additional_paths if you use a newer version of webpacker).

resolved_paths: ['components/blogs/app/javascripts']

4. Now in your main app you can import your engine's controllers like so:

import 'controllers' // Should already be here
import 'blogs/controllers'

5. It's important to namespace your engine's javascript folder so that things don't collide.

Hopefully it's easy enough to see how this could be expanded for uses other than Stimulus.js.