# node-pre-gyp #### node-pre-gyp makes it easy to publish and install Node.js C++ addons from binaries [![NPM](https://nodei.co/npm/node-pre-gyp.png?downloads=true&downloadRank=true)](https://nodei.co/npm/node-pre-gyp/) [![Build Status](https://api.travis-ci.org/mapbox/node-pre-gyp.svg)](https://travis-ci.org/mapbox/node-pre-gyp) [![Build status](https://ci.appveyor.com/api/projects/status/3nxewb425y83c0gv)](https://ci.appveyor.com/project/Mapbox/node-pre-gyp) [![Dependencies](https://david-dm.org/mapbox/node-pre-gyp.svg)](https://david-dm.org/mapbox/node-pre-gyp) `node-pre-gyp` stands between [npm](https://github.com/npm/npm) and [node-gyp](https://github.com/Tootallnate/node-gyp) and offers a cross-platform method of binary deployment. ### Features - A command line tool called `node-pre-gyp` that can install your package's c++ module from a binary. - A variety of developer targeted commands for packaging, testing, and publishing binaries. - A Javascript module that can dynamically require your installed binary: `require('node-pre-gyp').find` For a hello world example of a module packaged with `node-pre-gyp` see and [the wiki ](https://github.com/mapbox/node-pre-gyp/wiki/Modules-using-node-pre-gyp) for real world examples. ## Credits - The module is modeled after [node-gyp](https://github.com/Tootallnate/node-gyp) by [@Tootallnate](https://github.com/Tootallnate) - Motivation for initial development came from [@ErisDS](https://github.com/ErisDS) and the [Ghost Project](https://github.com/TryGhost/Ghost). - Development is sponsored by [Mapbox](https://www.mapbox.com/) ## FAQ See the [Frequently Ask Questions](https://github.com/mapbox/node-pre-gyp/wiki/FAQ). ## Depends - Node.js 0.12.x -> 0.8.x ## Install `node-pre-gyp` is designed to be installed as a local dependency of your Node.js C++ addon and accessed like: ./node_modules/.bin/node-pre-gyp --help But you can also install it globally: npm install node-pre-gyp -g ## Usage ### Commands View all possible commands: node-pre-gyp --help - clean - Remove the entire folder containing the compiled .node module - install - Install pre-built binary for module - reinstall - Run "clean" and "install" at once - build - Compile the module by dispatching to node-gyp or nw-gyp - rebuild - Run "clean" and "build" at once - package - Pack binary into tarball - testpackage - Test that the staged package is valid - publish - Publish pre-built binary - unpublish - Unpublish pre-built binary - info - Fetch info on published binaries You can also chain commands: node-pre-gyp clean build unpublish publish info ### Options Options include: - `-C/--directory`: run the command in this directory - `--build-from-source`: build from source instead of using pre-built binary - `--runtime=node-webkit`: customize the runtime: `node` and `node-webkit` are the valid options - `--fallback-to-build`: fallback to building from source if pre-built binary is not available - `--target=0.10.25`: Pass the target node or node-webkit version to compile against - `--target_arch=ia32`: Pass the target arch and override the host `arch`. Valid values are 'ia32','x64', or `arm`. - `--target_platform=win32`: Pass the target platform and override the host `platform`. Valid values are `linux`, `darwin`, `win32`, `sunos`, `freebsd`, `openbsd`, and `aix`. Both `--build-from-source` and `--fallback-to-build` can be passed alone or they can provide values. You can pass `--fallback-to-build=false` to override the option as declared in package.json. In addition to being able to pass `--build-from-source` you can also pass `--build-from-source=myapp` where `myapp` is the name of your module. For example: `npm install --build-from-source=myapp`. This is useful if: - `myapp` is referenced in the package.json of a larger app and therefore `myapp` is being installed as a dependent with `npm install`. - The larger app also depends on other modules installed with `node-pre-gyp` - You only want to trigger a source compile for `myapp` and the other modules. ### Configuring This is a guide to configuring your module to use node-pre-gyp. #### 1) Add new entries to your `package.json` - Add `node-pre-gyp` to `bundledDependencies` - Add `aws-sdk` as a `devDependency` - Add a custom `install` script - Declare a `binary` object This looks like: ```js "dependencies" : { "node-pre-gyp": "0.5.x" }, "bundledDependencies":["node-pre-gyp"], "devDependencies": { "aws-sdk": "~2.0.0-rc.15" } "scripts": { "install": "node-pre-gyp install --fallback-to-build", }, "binary": { "module_name": "your_module", "module_path": "./lib/binding/", "host": "https://your_module.s3-us-west-1.amazonaws.com" } ``` For a full example see [node-addon-examples's package.json](https://github.com/springmeyer/node-addon-example/blob/2ff60a8ded7f042864ad21db00c3a5a06cf47075/package.json#L11-L22). ##### The `binary` object has three required properties ###### module_name The name of your native node module. This value must: - Match the name passed to [the NODE_MODULE macro](http://nodejs.org/api/addons.html#addons_hello_world) - Must be a valid C variable name (e.g. it cannot contain `-`) - Should not include the `.node` extension. ###### module_path The location your native module is placed after a build. This should be an empty directory without other Javascript files. This entire directory will be packaged in the binary tarball. When installing from a remote package this directory will be overwritten with the contents of the tarball. Note: This property supports variables based on [Versioning](#versioning). ###### host A url to the remote location where you've published tarball binaries (must be `https` not `http`). It is highly recommended that you use Amazon S3. The reasons are: - Various node-pre-gyp commands like `publish` and `info` only work with an S3 host. - S3 is a very solid hosting platform for distributing large files, even [Github recommends using it instead of github](https://help.github.com/articles/distributing-large-binaries). - We provide detail documentation for using [S3 hosting](#s3-hosting) with node-pre-gyp. Why then not require S3? Because while some applications using node-pre-gyp need to distribute binaries as large as 20-30 MB, others might have very small binaries and might wish to store them in a github repo. This is not recommended, but if an author really wants to host in a non-s3 location then it should be possible. ##### The `binary` object has two optional properties ###### remote_path It **is recommended** that you customize this property. This is an extra path to use for publishing and finding remote tarballs. The default value for `remote_path` is `""` meaning that if you do not provide it then all packages will be published at the base of the `host`. It is recommended to provide a value like `./{name}/v{version}` to help organize remote packages in the case that you choose to publish multiple node addons to the same `host`. Note: This property supports variables based on [Versioning](#versioning). ###### package_name It is **not recommended** to override this property unless you are also overriding the `remote_path`. This is the versioned name of the remote tarball containing the binary `.node` module and any supporting files you've placed inside the `module_path` directory. Unless you specify `package_name` in your `package.json` then it defaults to `{module_name}-v{version}-{node_abi}-{platform}-{arch}.tar.gz` which allows your binary to work across node versions, platforms, and architectures. If you are using `remote_path` that is also versioned by `./{module_name}/v{version}` then you could remove these variables from the `package_name` and just use: `{node_abi}-{platform}-{arch}.tar.gz`. Then your remote tarball will be looked up at, for example, `https://example.com/your-module/v0.1.0/node-v11-linux-x64.tar.gz`. Avoiding the version of your module in the `package_name` and instead only embedding in a directory name can be useful when you want to make a quick tag of your module that does not change any C++ code. In this case you can just copy binaries to the new version behind the scenes like: ```sh aws s3 sync --acl public-read s3://mapbox-node-binary/sqlite3/v3.0.3/ s3://mapbox-node-binary/sqlite3/v3.0.4/ ``` Note: This property supports variables based on [Versioning](#versioning). #### 2) Add a new target to binding.gyp `node-pre-gyp` calls out to `node-gyp` to compile the module and passes variables along like [module_name](#module_name) and [module_path](#module_path). A new target must be added to `binding.gyp` that moves the compiled `.node` module from `./build/Release/module_name.node` into the directory specified by `module_path`. Add a target like this at the end of your `targets` list: ```js { "target_name": "action_after_build", "type": "none", "dependencies": [ "<(module_name)" ], "copies": [ { "files": [ "<(PRODUCT_DIR)/<(module_name).node" ], "destination": "<(module_path)" } ] } ``` For a full example see [node-addon-example's binding.gyp](https://github.com/springmeyer/node-addon-example/blob/2ff60a8ded7f042864ad21db00c3a5a06cf47075/binding.gyp). #### 3) Dynamically require your `.node` Inside the main js file that requires your addon module you are likely currently doing: ```js var binding = require('../build/Release/binding.node'); ``` or: ```js var bindings = require('./bindings') ``` Change those lines to: ```js var binary = require('node-pre-gyp'); var path = require('path'); var binding_path = binary.find(path.resolve(path.join(__dirname,'./package.json'))); var binding = require(binding_path); ``` For a full example see [node-addon-example's index.js](https://github.com/springmeyer/node-addon-example/blob/2ff60a8ded7f042864ad21db00c3a5a06cf47075/index.js#L1-L4) #### 4) Build and package your app Now build your module from source: npm install --build-from-source The `--build-from-source` tells `node-pre-gyp` to not look for a remote package and instead dispatch to node-gyp to build. Now `node-pre-gyp` should now also be installed as a local dependency so the command line tool it offers can be found at `./node_modules/.bin/node-pre-gyp`. #### 5) Test Now `npm test` should work just as it did before. #### 6) Publish the tarball Then package your app: ./node_modules/.bin/node-pre-gyp package Once packaged, now you can publish: ./node_modules/.bin/node-pre-gyp publish Currently the `publish` command pushes your binary to S3. This requires: - You have installed `aws-sdk` with `npm install aws-sdk` - You have created a bucket already. - The `host` points to an S3 http or https endpoint. - You have configured node-pre-gyp to read your S3 credentials (see [S3 hosting](#s3-hosting) for details). You can also host your binaries elsewhere. To do this requires: - You manually publish the binary created by the `package` command to an `https` endpoint - Ensure that the `host` value points to your custom `https` endpoint. #### 7) Automate builds Now you need to publish builds for all the platforms and node versions you wish to support. This is best automated. - See [Appveyor Automation](#appveyor-automation) for how to auto-publish builds on Windows. - See [Travis Automation](#travis-automation) for how to auto-publish builds on OS X and Linux. #### 8) You're done! Now publish your module to the npm registry. Users will now be able to install your module from a binary. What will happen is this: 1. `npm install ` will pull from the npm registry 2. npm will run the `install` script which will call out to `node-pre-gyp` 3. `node-pre-gyp` will fetch the binary `.node` module and unpack in the right place 4. Assuming that all worked, you are done If a a binary was not available for a given platform and `--fallback-to-build` was used then `node-gyp rebuild` will be called to try to source compile the module. ## S3 Hosting You can host wherever you choose but S3 is cheap, `node-pre-gyp publish` expects it, and S3 can be integrated well with [travis.ci](http://travis-ci.org) to automate builds for OS X and Ubuntu. Here is an approach to do this: First, get setup locally and test the workflow: #### 1) Create an S3 bucket And have your **key** and **secret key** ready for writing to the bucket. It is recommended to create a IAM user with a policy that only gives permissions to the specific bucket you plan to publish to. This can be done in the [IAM console](https://console.aws.amazon.com/iam/) by: 1) adding a new user, 2) choosing `Attach User Policy`, 3) Using the `Policy Generator`, 4) selecting `Amazon S3` for the service, 5) adding the actions: `DeleteObject`, `GetObject`, `GetObjectAcl`, `ListBucket`, `PutObject`, `PutObjectAcl`, 6) adding an ARN of `arn:aws:s3:::bucket/*` (replacing `bucket` with your bucket name), and finally 7) clicking `Add Statement` and saving the policy. It should generate a policy like: ```js { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1394587197000", "Effect": "Allow", "Action": [ "s3:DeleteObject", "s3:GetObject", "s3:GetObjectAcl", "s3:ListBucket", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::node-pre-gyp-tests/*" ] } ] } ``` #### 2) Install node-pre-gyp Either install it globally: npm install node-pre-gyp -g Or put the local version on your PATH export PATH=`pwd`/node_modules/.bin/:$PATH #### 3) Configure AWS credentials There are several ways to do this. You can use any of the methods described at http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html. Or you can create a `~/.node_pre_gyprc` Or pass options in any way supported by [RC](https://github.com/dominictarr/rc#standards) A `~/.node_pre_gyprc` looks like: ```js { "accessKeyId": "xxx", "secretAccessKey": "xxx" } ``` Another way is to use your environment: export node_pre_gyp_accessKeyId=xxx export node_pre_gyp_secretAccessKey=xxx You may also need to specify the `region` if it is not explicit in the `host` value you use. The `bucket` can also be specified but it is optional because `node-pre-gyp` will detect it from the `host` value. #### 4) Package and publish your build Install the `aws-sdk`: npm install aws-sdk Then publish: node-pre-gyp package publish Note: if you hit an error like `Hostname/IP doesn't match certificate's altnames` it may mean that you need to provide the `region` option in your config. ## Appveyor Automation [Appveyor](http://www.appveyor.com/) can build binaries and publish the results per commit and supports: - Windows Visual Studio 2013 and related compilers - Both 64 bit (x64) and 32 bit (x86) build configurations - Multiple Node.js versions For an example of doing this see [node-sqlite3's appveyor.yml](https://github.com/mapbox/node-sqlite3/blob/master/appveyor.yml). Below is a guide to getting set up: #### 1) Create a free Appveyor account Go to https://ci.appveyor.com/signup/free and sign in with your github account. #### 2) Create a new project Go to https://ci.appveyor.com/projects/new and select the github repo for your module #### 3) Add appveyor.yml and push it Once you have committed an `appveyor.yml` ([appveyor.yml reference](http://www.appveyor.com/docs/appveyor-yml)) to your github repo and pushed it appveyor should automatically start building your project. #### 4) Create secure variables Encrypt your S3 AWS keys by going to and hitting the `encrypt` button. Then paste the result into your `appveyor.yml` ```yml environment: node_pre_gyp_accessKeyId: secure: Dn9HKdLNYvDgPdQOzRq/DqZ/MPhjknRHB1o+/lVU8MA= node_pre_gyp_secretAccessKey: secure: W1rwNoSnOku1r+28gnoufO8UA8iWADmL1LiiwH9IOkIVhDTNGdGPJqAlLjNqwLnL ``` NOTE: keys are per account but not per repo (this is difference than travis where keys are per repo but not related to the account used to encrypt them). #### 5) Hook up publishing Just put `node-pre-gyp package publish` in your `appveyor.yml` after `npm install`. #### 6) Publish when you want You might wish to publish binaries only on a specific commit. To do this you could borrow from the [travis.ci idea of commit keywords](http://about.travis-ci.org/docs/user/how-to-skip-a-build/) and add special handling for commit messages with `[publish binary]`: SET CM=%APPVEYOR_REPO_COMMIT_MESSAGE% if not "%CM%" == "%CM:[publish binary]=%" node-pre-gyp --msvs_version=2013 publish If your commit message contains special characters (e.g. `&`) this method might fail. An alternative is to use PowerShell, which gives you additional possibilities, like ignoring case by using `ToLower()`: ps: if($env:APPVEYOR_REPO_COMMIT_MESSAGE.ToLower().Contains('[publish binary]')) { node-pre-gyp --msvs_version=2013 publish } Remember this publishing is not the same as `npm publish`. We're just talking about the binary module here and not your entire npm package. To automate the publishing of your entire package to npm on travis see http://about.travis-ci.org/docs/user/deployment/npm/ ## Travis Automation [Travis](https://travis-ci.org/) can push to S3 after a successful build and supports both: - Ubuntu Precise and OS X (64 bit) - Multiple Node.js versions For an example of doing this see [node-add-example's .travis.yml](https://github.com/springmeyer/node-addon-example/blob/2ff60a8ded7f042864ad21db00c3a5a06cf47075/.travis.yml). Note: if you need 32 bit binaries, this can be done from a 64 bit travis machine. See [the node-sqlite3 scripts for an example of doing this](https://github.com/mapbox/node-sqlite3/blob/bae122aa6a2b8a45f6b717fab24e207740e32b5d/scripts/build_against_node.sh#L54-L74). Below is a guide to getting set up: #### 1) Install the travis gem gem install travis #### 2) Create secure variables Make sure you run this command from within the directory of your module. Use `travis-encrypt` like: travis encrypt node_pre_gyp_accessKeyId=${node_pre_gyp_accessKeyId} travis encrypt node_pre_gyp_secretAccessKey=${node_pre_gyp_secretAccessKey} Then put those values in your `.travis.yml` like: ```yaml env: global: - secure: F+sEL/v56CzHqmCSSES4pEyC9NeQlkoR0Gs/ZuZxX1ytrj8SKtp3MKqBj7zhIclSdXBz4Ev966Da5ctmcTd410p0b240MV6BVOkLUtkjZJyErMBOkeb8n8yVfSoeMx8RiIhBmIvEn+rlQq+bSFis61/JkE9rxsjkGRZi14hHr4M= - secure: o2nkUQIiABD139XS6L8pxq3XO5gch27hvm/gOdV+dzNKc/s2KomVPWcOyXNxtJGhtecAkABzaW8KHDDi5QL1kNEFx6BxFVMLO8rjFPsMVaBG9Ks6JiDQkkmrGNcnVdxI/6EKTLHTH5WLsz8+J7caDBzvKbEfTux5EamEhxIWgrI= ``` More details on travis encryption at http://about.travis-ci.org/docs/user/encryption-keys/. #### 3) Hook up publishing Just put `node-pre-gyp package publish` in your `.travis.yml` after `npm install`. ##### OS X publishing If you want binaries for OS X in addition to linux you have two options: 1) [Enabling multi-OS](#enabling-multi-os) 2) [Using `language: objective-c` in a git branch](#using-language-objective-c). ##### Enabling multi-OS This requires emailing a request to `support@travis-ci.com` for each repo you wish to have enabled. More details at . Next you need to tweak the `.travis.yml` to ensure it is cross platform. Use a configuration like: ```yml language: cpp os: - linux - osx env: matrix: - NODE_VERSION="0.10" - NODE_VERSION="0.11.14" before_install: - rm -rf ~/.nvm/ && git clone --depth 1 https://github.com/creationix/nvm.git ~/.nvm - source ~/.nvm/nvm.sh - nvm install $NODE_VERSION - nvm use $NODE_VERSION ``` See [Travis OS X Gochas](#travis-os-x-gochas) for why we replace `language: node_js` and `node_js:` sections with `language: cpp` and a custom matrix. Also create platform specific sections for any deps that need install. For example if you need libpng: ```yml - if [ $(uname -s) == 'Linux' ]; then apt-get install libpng-dev; fi; - if [ $(uname -s) == 'Darwin' ]; then brew install libpng; fi; ``` For detailed multi-OS examples see [node-mapnik](https://github.com/mapnik/node-mapnik/blob/master/.travis.yml) and [node-sqlite3](https://github.com/mapbox/node-sqlite3/blob/master/.travis.yml). ##### Using `language: objective-c` If your repo does not have multi-OS enabled, an alternative method for building for OS X is to tweak your `.travis.yml` to use: ```yml language: objective-c ``` Keep that change in a different git branch and sync that when you want binaries published. Next learn about a few [Travis OS X Gochas](#travis-os-x-gochas). ##### Travis OS X Gochas First, unlike the Travis linux machines the OS X machines do not put `node-pre-gyp` on PATH by default. So to you will need to: ```sh export PATH=$(pwd)/node_modules/.bin:${PATH} ``` Second, the OS X machines doe not support using a matrix for installing node.js different versions. So you need to bootstrap the installation of node.js in a cross platform way. By doing: ```yml env: matrix: - NODE_VERSION="0.10" - NODE_VERSION="0.11.14" before_install: - rm -rf ~/.nvm/ && git clone --depth 1 https://github.com/creationix/nvm.git ~/.nvm - source ~/.nvm/nvm.sh - nvm install $NODE_VERSION - nvm use $NODE_VERSION ``` You can easily recreate the previous behavior of this matrix: ```yml node_js: - "0.10" - "0.11.14" ``` #### 4) Publish when you want You might wish to publish binaries only on a specific commit. To do this you could borrow from the [travis.ci idea of commit keywords](http://about.travis-ci.org/docs/user/how-to-skip-a-build/) and add special handling for commit messages with `[publish binary]`: COMMIT_MESSAGE=$(git show -s --format=%B $TRAVIS_COMMIT | tr -d '\n') if [[ ${COMMIT_MESSAGE} =~ "[publish binary]" ]]; then node-pre-gyp publish; fi; Then you can trigger new binaries to be built like: git commit -a -m "[publish binary]" Or, if you don't have any changes to make simply run: git commit --allow-empty -m "[publish binary]" Remember this publishing is not the same as `npm publish`. We're just talking about the binary module here and not your entire npm package. To automate the publishing of your entire package to npm on travis see http://about.travis-ci.org/docs/user/deployment/npm/ # Versioning The `binary` properties of `module_path`, `remote_path`, and `package_name` support variable substitution. The strings are evaluated by `node-pre-gyp` depending on your system and any custom build flags you passed. - `node_abi`: The node C++ `ABI` number. This value is available in Javascript as `process.versions.modules` as of [`>= v0.10.4 >= v0.11.7`](https://github.com/joyent/node/commit/ccabd4a6fa8a6eb79d29bc3bbe9fe2b6531c2d8e) and in C++ as the `NODE_MODULE_VERSION` define much earlier. For versions of Node before this was available we fallback to the V8 major and minor version. - `platform` matches node's `process.platform` like `linux`, `darwin`, and `win32` unless the user passed the `--target_platform` option to override. - `arch` matches node's `process.arch` like `x64` or `ia32` unless the user passes the `--target_arch` option to override. - `configuration` - Either 'Release' or 'Debug' depending on if `--debug` is passed during the build. - `module_name` - the `binary.module_name` attribute from `package.json`. - `version` - the semver `version` value for your module from `package.json` (NOTE: ignores the `semver.build` property). - `major`, `minor`, `patch`, and `prelease` match the individual semver values for your module's `version` - `build` - the sevmer `build` value. For example it would be `this.that` if your package.json `version` was `v1.0.0+this.that` - `prerelease` - the semver `prerelease` value. For example it would be `alpha.beta` if your package.json `version` was `v1.0.0-alpha.beta` The options are visible in the code at