Build node packages into deployable applications

Overview

strong-build

Build a node application package, preparing it for deploy to production.

It is useful standalone, but is commonly used to build applications for deployment to the StrongLoop process manager, strong-pm.

For more details, see http://strong-pm.io.

Installation

sl-build is made available through the strongloop tool as slc build, and works well with the StrongLoop Process Manager, strong-pm.

sl-build can be installed standalone with:

npm install -g strong-build

Overview

The purpose of building a node application is to bundle its dependencies so that there:

  • are no deploy-time dependencies on external services
  • it is in a deployable format

The build process is implemented as four commands:

  • sl-build --install: the core of the build, it installs dependencies, runs custom build steps, and prunes development dependencies
  • sl-build --bundle: modify the npm package.json and .npmignore configuration files so dependencies will be packed
  • sl-build --pack: create an npm package of the build
  • sl-build --commit: commit the build onto a git branch

The default behaviour of sl-build depends on whether the current directory is a git repository.

  • In a git repository: the default is sl-build --install --commit, to build onto a git branch.
  • Otherwise: the default is sl-build --bundle --install --pack, to build an npm package.

Both npm packages and git branches can be deployed to the StrongLoop process manager using the StrongLoop deploy tool.

Specifying any command disables the default, and allows any mix of commands to be run, either singly, or all at once.

Also, note that builds should be done in a clean working copy. You don't build deployment artifacts out of a possibly dirty working copy if you want reproducible builds. You can clean your working copy using git clean -x -d -f. This is too destructive for the build tool to do, but doing a build in an unclean working repository may trigger an error in a future version of the tool.

install command

Installation automates the common work flow for building application dependencies:

  • npm install --ignore-scripts: Install dependencies without running scripts. Scripts can be run optionally with --scripts.
  • npm run build: Custom build steps such as grunt build or bower can be specified in the package's scripts.build property, since front-end code served by node commonly requires some amount of preparation.
  • npm prune --production: Remove development-only tools (such as bower, or grunt) that may have been required by the package's build scripts, but should not be deployed.

Note that compilation and install scripts should be run on the deployment server using:

  • npm rebuild: Compile add-ons for current system.
  • npm install: Run any install scripts (not typical, but if they exist they may be required to prepare the application for running).

If builds are done on the same system architecture as the deploy, it is possible to compile and package the add-ons, and avoid the presence of a compiler on the deployment system. This is recommended when possible, but is not the default assumption of strong-build.

bundle command

Bundling configures the package.json and .npmignore so deployment (not development) dependencies as well as any 'build' script output will not be ignored by npm pack.

This is unnecessary when using git to deploy, but mandatory when creating npm packages!

package.bundleDependencies

Bundling requires that the bundleDependencies property in the package.json file is configured to include all non-development dependencies, including optional dependencies, which are often overlooked.

Its important that you remember to add every new production dependency to the bundleDependencies property, if you don't, npm will try and install them after deploy, creating unexpected and fragile dependencies on npmjs.org.

Since keeping this up-to-date manually is likely to go wrong, we recommend allowing the bundle command to do this for you. However, the bundleDependencies property is not modified if present, so you are free to maintain it yourself, if you wish (or to not just the bundle command).

.npmignore

Setting bundle dependencies is insufficient to get the output of build tools.

Both build output and project ephemera such as test output is usually ignored using .gitignore, as it should be. However, if npm does not find a .npmignore configuration file, it uses .gitignore as a fallback. This means that if you have custom build output, such as minimized JavaScript, it will be treated as project ephemera, and not be packed by npm.

The bundle command will create an empty .npmignore file if there is a .gitignore file but there is no .npmignore file. This will work for clean repositories, but if you have any project ephemera, they will get packed.

We recommend you write and maintain you own .npmignore file unless your build process guarantees a clean working repository.

pack command

Pack output is a tar file in the format produced by npm pack and accepted by npm install and strong-deploy.

The pack file is placed in the parent directory of the application being packed, to avoid the pack file itself getting packed by future builds, and to allow the working repository to be cleaned.

If a .npmignore file was created by the bundle command, check the pack file contents carefully to ensure build products are packed, but project ephemera are not.

commit command

Committing build products into git provides the most robust tracking and storage, including versioning of deployments.

This is often done by committing both build products and dependencies (node_modules) into git where they pollute the source branches, create massive git commits and huge churn on the development branches and repositories.

The commit command does not do this.

It commits an exact replica of current branch source and build products onto a deployment branch. After the commit, the deployment branch tip shows as a merge of the deployment and source branches. This allows a complete history of deployment builds to be kept in git, but separated from the development branches. Deployment branches can be pushed to the same repository as the development branches, or not.

Note that branches prepared like this can also be pushed to platforms such as OpenShift and Heroku.

The default name of the deployment branch is "deploy", but is configurable with the --onto BRANCH modifier to the commit command.

Usage

usage: sl-build [options]

Build a node application package.

With no options, the default depends on whether a git repository is
detected or not.

If a git repository is detected the default is `--git`: install and commit the
build results to the "deploy" branch, which will be created if it does not
already exist.

If no git repository is detected the default is `--npm`: bundle, install, and
pack the build results into a `<package-name>-<version>.tgz` file.

Options:
  -h,--help       Print this message and exit.
  -v,--version    Print version and exit.
  -n,--npm        Same as `--install --bundle --pack`.
  -g,--git        Same as `--install --commit`.
  -i,--install    Install dependencies (without scripts, by default).
  --scripts       If installing, run scripts (to build addons).
  -b,--bundle     Modify package to bundle deployment dependencies.
  -p,--pack       Pack into a publishable archive (with dependencies).

Git specific options:
  -c,--commit     Commit build output (branch specified by --onto).
  --onto BRANCH   Branch to commit build results to, created if
                  necessary ("deploy", by default).

License

strong-build uses a dual license model.

You may use this library under the terms of the Artistic 2.0 license, or under the terms of the StrongLoop Subscription Agreement.

Comments
  • Feature/build a package

    Feature/build a package

    /to @rmg @bajtos

    See TODO.txt for tracking of WIP (replaces the old gist https://gist.github.com/sam-github/8acbde690ef6b5554e91).

    I think this does all the mandatory features, remaining work before release is:

    • the git-related manipulations (will do in another PR)
    • talk to @ritch and get his help testing the approach against a significant example repo, such as strongloop/loopback-example-full-stack (I had some difficulties with install and run, probably I'm doing it wrong).
    opened by sam-github 12
  • tarball packed by slc build -ibp --scripts does not include the build/ subdirectory

    tarball packed by slc build -ibp --scripts does not include the build/ subdirectory

    I did

    cd /tmp/
    npm install statsd
    cd node_modules/statsd
    slc build -ibp --scripts
    tar -tf ../statsd-0.7.2.tar.gz | grep build  # nothing!
    find . -print | grep build  # lots!
    

    And the packfile produced did not include node_modules/node-syslog/build/....

    #fib-1 
    opened by sam-github 10
  • Update docs for --npm and --git options

    Update docs for --npm and --git options

    PR #25 adds new options to the slc build command.

    At a minimum, update the command ref. Also review to see if any changes are needed in these (and child articles):

    • http://docs.strongloop.com/display/SLC/Building+applications+with+slc
    • http://docs.strongloop.com/display/SLC/Deployment+best+practices

    NOTE: We should not make the doc changes until the change is published to npm.

    opened by crandmck 9
  • slc build --install should force a reinstall of deps.

    slc build --install should force a reinstall of deps.

    I've been running into an issue where every second time I run this command slc build -ibp the npm run build command, which is automatically invoked, fails. I get the following output:

    $ slc build -ibp --scripts
    Running `npm install`
    Running `npm run build`
    Error on `npm run build`:
    Failed to run `npm run build`:
    
    > <my project>
    > gulp default
    
    
    Error: Cannot find module '<path_to_my_project>/node_modules/gulp/node_modules/v8flags/3.14.5.9.flags.json'
    

    I know this looks like an issue with v8flags, but I'm not so sure that it is. When I rm -rf my node_modules and then do a fresh npm install I can invoke npm run build to my hearts content; but, when I invoke slc build -ibp, it will work only the first time.

    Any thoughts??

    opened by thedodd 7
  • Bundle private / scoped dependencies with slc build --npm

    Bundle private / scoped dependencies with slc build --npm

    Hello,

    I'm not sure if this issue belongs here or within npm, but I'll start here and see if there is a solution within strongloop / slc.

    I am using slc build --npm to package my node app into a .tgz that I can slc deploy to my production web server. This was working well when all of my dependencies were public npm modules.

    Recently I've started using private npm modules that are scoped and hosted on nppmjs.org (https://www.npmjs.com/private-modules)

    slc build -b appropriately adds my scoped module (e.g. @barwin/my-private-module) to the bundleDependencies array in package.json, but slc build --npm does not ultimately include the scoped module in the resulting .tgz file. All of my non-scoped (public) npm dependencies are included in the .tgz as expected. It's only an issue with the scoped modules.

    I found a snippet in npm's faq about scoped dependencies:

    https://docs.npmjs.com/misc/faq Unscoped packages can only depend on other unscoped packages. Scoped packages can depend on packages from their own scope, a different scope, or the public registry (unscoped).

    Based on that, I tried renaming my top level project to be under the same scope as the private modules that I'm trying to bundle, but that had no effect.

    Looking for any suggestions on how to accomplish this. Thanks!

    bug 
    opened by barwin 6
  • Fails when app has 0 deps

    Fails when app has 0 deps

    Here is my app:

    require('net').createServer().listen(0);
    

    Here is what I get from slc build

    Running `npm install --ignore-scripts`
    Running `npm prune --production`
    shell.js: internal error
    Error: ENOENT, no such file or directory 'node_modules'
        at Error (native)
        at Object.fs.statSync (fs.js:801:18)
        at /usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/shelljs/src/find.js:42:12
        at Array.forEach (native)
        at Object._find (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/shelljs/src/find.js:39:9)
        at Object.find (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/shelljs/src/common.js:186:23)
        at Object.doNpmPack [as func] (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/index.js:290:29)
        at Immediate._onImmediate (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/vasync/lib/vasync.js:213:20)
        at processImmediate [as _immediateCallback] (timers.js:358:17)
    

    @kraman

    bug 
    opened by ritch 6
  • Fix npm2 support

    Fix npm2 support

    Problem: npm run build fails on npm2 if there is no build run-script defined. Fix: Only run npm run build step if there is a build run-script defined.

    The fix is easy, but I'm going to spend a few minutes trying to get CI green if I can.

    opened by rmg 6
  • Construct packfile name from package

    Construct packfile name from package

    The approach of using the npm pack output assumes that there is no other output from npm, which isn't true when there are build scripts.

    connected to strongloop-internal/scrum-nodeops#766

    replaces https://github.com/strongloop/strong-build/pull/30

    opened by sam-github 5
  • update dependencies to latest major versions

    update dependencies to latest major versions

    Should squelch the warnings about json-file-plus being deprecated as well as reduce the number of lodash copies installed in a full strongloop installation.

    opened by rmg 5
  • replace 'npm pack' with strong-pack

    replace 'npm pack' with strong-pack

    • [x] failing test for verifying problem
    • [ ] more complete tests
      • ensure we ignore files key in package.json
      • ensure .npmignore is ignored
      • ensure packages not listed in bundledDependencies are included
    • [x] use custom tgz generator instead of npm pack
    • [ ] annotate package.json with metadata about the state of the bundle

    Connect to strongloop-internal/scrum-nodeops#1015

    opened by rmg 4
  • Clean up git behaviour

    Clean up git behaviour

    • [x] detect git repository
    • [x] make --onto <branch> a modifier of --install instead of checking out branch
    • [x] make --onto BRANCH default to "deploy"
    • [x] when git detected, use --install --commit as default operation (implicit --onto deploy)
    • [x] when committing, create branch if it doesn't exist
    • [x] merge HEAD into deploy branch as separate commit before committing artifacts
    • [x] skip merge if destination branch already up to date
    • [x] don't commit build artifacts if they are identical to the target branch's content
    • [x] update existing tests to match new behaviour
    • [x] add tests to verify new behaviour
    opened by rmg 4
  • Security Fix for Remote Code Execution - huntr.dev

    Security Fix for Remote Code Execution - huntr.dev

    https://huntr.dev/users/alromh87 has fixed the Remote Code Execution vulnerability πŸ”¨. alromh87 has been awarded $25 for fixing the vulnerability through the huntr bug bounty program πŸ’΅. Think you could fix a vulnerability like this?

    Get involved at https://huntr.dev/

    Q | A Version Affected | ALL Bug Fix | YES Original Pull Request | https://github.com/418sec/strong-build/pull/2 Vulnerability README | https://github.com/418sec/huntr/blob/master/bounties/npm/strong-build/1/README.md

    User Comments:

    Bounty URL: https://www.huntr.dev/bounties/1-npm-strong-build/

    βš™οΈ Description *

    strong-build was vulnerable against arbitrary command injection cause some user supplied inputs were taken and composed into string to be executed without prior validation. After update Arbitary Code Execution is avoided

    πŸ’» Technical Description *

    Shelljs is used to execute command so param is sanitized with regexp, whitelisting alphanumeric characters

    πŸ› Proof of Concept (PoC) *

    1. Install the package
    2. Check there aren't files called HACKED
    3. Execute the following commands:
    bin/sl-build.js --onto '";touch HACKED; #' #  Run the PoC
    
    1. It will create a file HACKED in the working directory.

    πŸ”₯ Proof of Fix (PoF) *

    After fix no file is created

    πŸ‘ User Acceptance Testing (UAT)

    Commands can be executed normally

    opened by huntr-helper 0
  • Can we have the --no-optional flag?

    Can we have the --no-optional flag?

    Can we add a --no-optional so slc build will install dependencies with npm install --no-optional? Our company's policy doesn't allow internet access on the build server, which slows down the build a lot when it tries to install an optional dependency sl-blip. Thanks!

    #community 
    opened by javefang 2
  • Support .npmignore when committing to deploy branch in git?

    Support .npmignore when committing to deploy branch in git?

    It seems that the .npmignore file is only observed by the --pack option, not the --commit option. I woud prefer to use git as the mechanism to store our versioned builds, but obviously the git repository for my project contains a test folder for my tests. If I add 'test' to .npmignore and pack a build into tgz then the test folder (and anything else in .npmignore) is not included... alas if I commit a build to a deployment branch and then git clone/pull from this branch on the production machine, the test folder comes along for the ride.

    I suppose there is perhaps no harm in these things hanging around on the production machine, but I'd prefer that they weren't there. (This also goes for things like IDE project files which are committed to git, but I would also add to a .npmignore file to have them excluded from the package or deploy branch commit).

    opened by jimmcslim 1
  • find workaround for warnings about CRLF to LF conversions

    find workaround for warnings about CRLF to LF conversions

    http://stackoverflow.com/questions/1967370/git-replacing-lf-with-crlf

    core.autocrlf set to false guarantees what gets checked out is what got checked in, this is appropriate for our use case

    as-is, since npm (bless it) uses UNIX conventions for most of the files it writes, including the ones from most package.tgz files from npmjs.org, the "warning: LF will be replaced by CRLF in ...." will be spewed endlessly to the screen.... for files that the user has no control over.

    We have to make sure its only temporarily false, though, and not mess with the user's defaults.

    #fib-3 
    opened by sam-github 10
Owner
StrongLoop and IBM API Connect
API lifecycle from creation to management
StrongLoop and IBM API Connect
:fork_and_knife: Web applications made easy. Since 2011.

Brunch Web applications made easy. Since 2011. Fast front-end web app build tool with simple declarative config and seamless incremental compilation f

Brunch 6.8k Jan 2, 2023
Browser compilation library – an asset pipeline for applications that run in the browser

Broccoli A fast, reliable asset pipeline, supporting constant-time rebuilds and compact build definitions. Comparable to the Rails asset pipeline in s

Broccoli 3.3k Dec 30, 2022
This template is designed for compiling Rust libraries into WebAssembly and publishing the resulting package to NPM.

This template is designed for compiling Rust libraries into WebAssembly and publishing the resulting package to NPM.

Keith 2 Jul 5, 2022
Builder: A gulp-like build system with modern JavaScript

Builder: A gulp-like build system with modern JavaScript What is this? This is a build system meant to automate tasks. At this point it’s merely a con

Wladimir Palant 7 Mar 22, 2022
Simple build system and CLI for AMX Mod X projects

?? AMXXPack Simple build system and CLI for AMX Mod X projects. ?? About This system will be useful for projects with multiple plugins and assets. Usi

Hedgehog Fog 11 Nov 15, 2022
browser-side require() the node.js way

browserify require('modules') in the browser Use a node-style require() to organize your browser code and load modules installed by npm. browserify wi

null 14.3k Dec 29, 2022
A powerful and lightweight inversion of control container for JavaScript & Node.js apps powered by TypeScript.

InversifyJS A powerful and lightweight inversion of control container for JavaScript & Node.js apps powered by TypeScript. About InversifyJS is a ligh

inversify 9.5k Jan 4, 2023
:red_circle: Functional task runner for Node.js

start ⚠️ Project has been transferred to NexTools metarepo functional – in all senses fast – parallelism and concurrency shareable – presets as publis

Kir Belevich 477 Dec 15, 2022
Template of yew project, using tailwind and webpack for css, trunk for build and serve, deployable as is for github.io

Yew Template for Github.io Yew template that deployable as is for github.io (or as a normal yew template with css/scss stuffs without github.io), with

null 17 Dec 23, 2022
The full power of the Go Compiler directly in your browser, including a virtual file system implementation. Deployable as a static website.

Static Go Playground Features Full Go Compiler running on the browser. Supports using custom build tags. Incremental builds (build cache). Supports mu

null 25 Jun 16, 2022
Demonstration of how you build full-stack, typed, NPM packages, the right way

NPM Packages - The Right way This repository aims to demonstrate how to build NPM packages the right way. my-package should import the shared code, bu

VulcanJS 61 Nov 27, 2022
A collection of Aurelia 2 example applications showcasing how to build Aurelia 2 applications and other tasks.

Aurelia 2 Examples A monorepository of a treasure trove of Aurelia 2 example applications you can use as a guide to help you build your own applicatio

aurelia 12 Dec 29, 2022
PackageInfo - chrome-extension which provides information about node packages, whenever you select the package name using mouse

PackageInfo chrome-extension which provides information about node packages, whenever you select the package name using mouse Whenever you surf throug

subrahmanya  s hegade 3 Feb 12, 2022
An npm package for demonstration purposes using TypeScript to build for both the ECMAScript Module format (i.e. ESM or ES Module) and CommonJS Module format. It can be used in Node.js and browser applications.

An npm package for demonstration purposes using TypeScript to build for both the ECMAScript Module format (i.e. ESM or ES Module) and CommonJS Module format. It can be used in Node.js and browser applications.

Snyk Labs 57 Dec 28, 2022
Call Python packages in JavaScript.

Introduction to Boa Boa is the Python Bridge Layer in Pipcook, it lets you call Python functions seamlessly in Node.js, it delivers any Python module

imgcook 64 Jan 5, 2023
An users NodeJS API without packages libs or frameworks!

NodeJS Users API - Without Frameworks And Packages ?? Table of Contents About Getting Started Usage Built Using Authors ?? About Purpose of this proje

Nathan Cotrim Lemos 31 Feb 7, 2022
Detect npm packages by author name in your package-lock.json or yarn.lock.

detect-package-by-author Detect npm packages by author name in your package-lock.json or yarn.lock. Install Install with npm: # Not Yet Publish # npm

azu 2 Jan 11, 2022
A set of javascript packages that generates fake data for you.

Faker A set of javascript packages that generates fake data for you. Install $ npm install --save @fakerjs/faker Usage import faker from '@fakerjs/fa

The New Faker for JavaScript 33 Apr 18, 2022