Why we need a new liferay-npm-bundler (2 of 3)

How we are fixing the problems learned in bundler 2.x

This is the second of a three articles series motivating and explaining the enhancements we have done to Liferay's npm bundler. You can read the first article to find out the motivation for taking the actions explained in this one.

To solve the problems we saw in the previous article we followed this process:

  1. Go one step back in deduplication and behave like standard bundlers (webpack, Browserify, etc.).
  2. Fix peer dependencies support.
  3. Provide a way to deduplicate modules.

Step 1 gets us to a point where we have repeatable builds that mimick what the developer has in his node_modules folder when the projects are deployed to the server, and run in the browser. We implement it by isolating project dependencies (namespacing the packages) so that the dependencies of each project are isolated between them.

With that done, we get 2 nearly for free, because we just need to inject virtual dependencies in package.json files when peer dependencies are used. But this time we know which version constraints to use, because the project's whole dependency tree is isolated from other projects. That is, we have something similar to a node_modules folder available in the browser for each project.

Finally, because we have lost deduplication with steps 1 and 2 and that leads to a solution which is equivalent to standard bundlers, we define a way to deduplicate packages manually. This new way of deduplication is not automatic but leads to full control (during build time) of how each package is resolved.

Let's see the steps in more detail...

Isolation of dependencies

To achieve the isolation the new bundler just prefixes each package name with the name of the project and rewrites all the references in the JS files. For example, say you have our old beloved my-portlet project:


 package.json
    {
        "name": "my-portlet",
        "version": "1.0.0",
        "dependencies": {
            "isarray": "^1.0.0"
        }
    }
 node_modules/isarray
     package.json
        {
            "name": "isarray",
            "version": "1.0.1",
            "main": "index.js"
        }
 META-INF/resources
     view.jsp
        <aui:script require="my-portlet@1.0.0/js/main"/>
     js
         main.js
            require('isarray', function(isarray) {
                console.log(isarray([]));
            });


When we bundle it with bundler 1.x we get something like:


 META-INF/resources
     package.json
        {
            "name": "my-portlet",
            "version": "1.0.0",
            "dependencies": {
                "isarray": "^1.0.0"
            }
        }
     view.jsp
        <aui:script require="my-portlet@1.0.0/js/main"/>
     js
         main.js
            require('isarray', function(isarray) {
                console.log(isarray([]));
            });
     node_modules/isarray
         package.json
            {
                "name": "isarray",
                "version": "1.0.1",
                "main": "index.js"
            }


But if we use bundler 2.x it changes to:


 META-INF/resources
     package.json
        {
            "name": "my-portlet",
            "version": "1.0.0",
            "dependencies": {
                "my-portlet$isarray": "^1.0.0"
            }
        }
     view.jsp
        <aui:script require="my-portlet@1.0.0/js/main"/>
     js
         main.js
            require('my-portlet$isarray', function(isarray) {
                console.log(isarray([]));
            });
     node_modules/my-portlet$isarray
         package.json
            {
                "name": "my-portlet$isarray",
                "version": "1.0.1",
                "main": "index.js"
            }


As you see, we just needed to prepend my-portlet$ to each dependency package name and that way, each project will load its own dependencies and won't collide with any other project. Easy.

If we did now the same test of deploying my-portlet and his-portlet, each one would get its own versions simply because we have two different isarray packages: one called my-portlet$isarray and another called his-portlet$isarray.

Peer dependency support

Because we have isolated dependencies per portlet, we can now honor peer dependencies perfectly. For example, remember the Diverted peer dependencies section in the previous article: with bundler 1.x, there was only one a-library package available for everybody. But with the new namespacing technique, we have two a-librarys: my-portlet$a-library and his-portlet$a-library.

Thus, we can resolve peer dependencies exactly as stated in both projects because their names are prefixed with the project's name:

my-portlet@1.0.0
    ➥ my-portlet$a-library 1.0.0
    ➥ my-portlet$a-helper 1.0.0

his-portlet@1.0.0
    ➥ his-portlet$a-library 1.0.0
    ➥ his-portlet$a-helper 1.2.0

And in this case, my-portlet$a-library will depend on a-helper at version 1.0.0 (which is namespaced as my-portlet$a-helper) and his-portlet$a-library will depend on a-helper at version 1.2.0 (which is namespaced ashis-portlet$a-helper).

How does all this magic happen? Easy: we have just created a new bundler plugin named liferay-npm-bundler-plugin-inject-peer-dependencies that scans all JS modules for require calls and injects a virtual dependency in the package.json file when a module from an undeclared package is required.

So, for example, let's say you have the following project:


 META-INF/resources
     package.json
        {
            "name": "my-portlet",
            "version": "1.0.0",
            "dependencies": {
                "isarray": "^1.0.0"
            }
        }
     js
         main.js
            require(['isarray', 'isobject', function(isarray, isobject) {
                console.log(isarray([]));
                console.log(isobject([]));
            });
     node_modules
         isarray
             package.json
                {
                    "name": "isarray",
                    "version": "1.0.1",
                    "main": "index.js"
                }
         isobject
             package.json
                {
                    "name": "isobject",
                    "version": "1.1.0",
                    "main": "index.js"
                }


As you can see, there's no dependency to isobject in the package.json file.

However, if we run the project through the bundler configured with the inject-peer-dependencies plugin, it will find out that main.js is requiring isobject and will resolve it to version 1.1.0 which can be found in the node_modules folder.

After that, the plugin will inject a new dependency in the output package.json so that it looks like this:

{
    "name": "my-portlet",
    "version": "1.0.0",
    "dependencies": {
        "isarray": "^1.0.0",
        "isobject": "1.1.0"
    }
}

Note how, being an injected dependency, isobject's version constraints are its specific version number, without caret or any other semantic version operator. This makes sense, as we want to honor the exact peer dependency found in the project and thus we cannot inject a more relaxed semantic version expression because it could lead to unstable results.

Also keep in mind that these transformations are made in the output files (the ones in your build directory), not on your original source files.

Deduplication of packages (imports)

As we said before, the problem with namespacing is that each portlet is getting its own dependencies and we don't deduplicate any more. If we always used the bundler this way, it won't make too much sense, because we could obtain the same functionality with standard bundlers like webpack or Browserify and wouldn't need to rely on a specific tool like liferay-npm-bundler.

But, being Liferay a portlet based architecture, it would be quite useful if we could share dependencies among different portlets. That way, if a page is composed of five portlets using jQuery, only one copy of jQuery would be loaded by the JS interpreter to be used by the five different portlets.

With bundler 1.x that deduplication was made automagically, but we had no control over it. However, with version 2.x, we may now import packages from an external OSGi bundle, instead of using our own. That way, we can put shared dependencies in one project, and reference them from the rest.

Let's see an example: imagine that you have three portlets that use our favorite Non Existing Wonderful UI Components framework (WUI). Suppose this quite limited framework is composed of 3 packages:

  1. component-core
  2. button
  3. textfield

Now, say that we have these three portlets:

  1. my-toolbar
  2. my-menu
  3. my-content

Which we use to compose the home page of our site. And the three depend on the WUI framework.

If we just use the bundler to create three OSGi bundles, each one will package a copy of WUI inside it and use it when the page is rendered, thus leading to a page where your browser loads three different copies of WUI in the JS interpreter.

To avoid that, we will create a fourth bundle where WUI is packaged and import the WUI packages from the previous three bundles. This will lead to an structure like the following:


 my-toolbar
     .npmbundlerrc
        {
            "config": {
                "imports": {
                    "wui-provider": {
                        "component-core": "^1.0.0",
                        "button": "^1.0.0",
                        "textfield": "^1.0.0"
                    }
                }
            }
        }
 my-menu
     .npmbundlerrc
        {
            "config": {
                "imports": {
                    "wui-provider": {
                        "component-core": "^1.0.0",
                        "button": "^1.0.0",
                        "textfield": "^1.0.0"
                    }
                }
            }
        }
 my-content
     .npmbundlerrc
        {
            "config": {
                "imports": {
                    "wui-provider": {
                        "component-core": "^1.0.0",
                        "button": "^1.0.0",
                        "textfield": "^1.0.0"
                    }
                }
            }
        }
 wui-provider
     package.json
        {
            "name": "wui-provider",
            "dependencies": {
                "component-core": "^1.0.0",
                "button": "^1.0.0",
                "textfield": "^1.0.0"
            }
        }


As you can see, the three portlets declare the WUI imports in the .npmbundlerrc file. They would probably also be declared in the bundles' package.json files though they can be omitted too and it will still work in runtime because they are being imported.

So, how does this work? The key concept behind imports is that we switch the namespace of certain packages thus pointing them to an external bundle.

So, say that you have the following code in my-toolbar portlet:

var Button = require('button');

This would be transformed to the following when run through the bundler unconfigured:

var Button = require('my-toolbar$button');

But, because we are saying that button is imported from wui-provider, it will be changed to:

var Button = require('wui-provider$button');

And also, a dependency on wui-provider$button at version ^1.0.0 will be introduced in my-toolbar's package.json file so that the loader may look for the correct version.

And that's enough, because once we require wui-provider$button at runtime, we will jump to wui-provider's context and load the subdependencies from there on, even if we are executing code from my-toolbar.

If you give it a thought, that's logical because wui-provider's modules are namespaced too and once you load a module from it, it will keep requiring wui-provider$ prefixed modules all they way down.

So, that's pretty much the motivation and implementation of bundler 2.x. Hope it will shed some light on why we needed these changes and how we are now founding the npm SDK on much more stable roots.

You can now read the last article of the series analyzing a real life example of using bundler 2.0 within an Angular portlet.