Building fully-functional Mac, PC, & Linux apps in Javascript, Pt. 2: Native Addons with GYP, node-gyp, & NAN

Intro

This is the second in a series of posts that introduce essential tools & strategies when architecting a JS app for native desktop platforms.

This post is intended for developers hoping to use native OS functions from JS code, e.g. to pass webcam data through Snapchat-style lenses, or manage a system-wide VPN config (with the former likely leveraging DirectX or Metal APIs...).

The first section of this post will discuss GYP, node-gyp, and NAN in isolation; we will then use these tools together to build a very basic native plugin. By the end, you should understand enough of the ecosystem & process to write your own plugin (with a little SO grease, of course :).

Though we assume the addon is being built for an Electron + Ember.js environment, all concepts and code should be broadly applicable.

Series

Pt. 1 - Intro to Electron
Pt. 2 - Native Addons with GYP, node-gyp & NAN

Part 2: Native Addons with GYP, node-gyp, & NAN

Table of Contents

What are GYP, node-gyp, and NAN?

GYP, node-gyp, and NAN are the three essential tools for JS app developers looking to incorporate custom, platform-specific code in their apps.

GYP

GYP is a build language specification, parsed and actioned with Python, that uses a declarative syntax to process input files & options and spit out executables (apps) or libraries. It is described as "a Meta-Build system: a build system that generates other build systems" [1].

Developers use GYP on large, multi-platform projects, particularly when it is important to build the project with native build tooling (e.g. XCode, Visual Studio); doing so often simplifies app packaging & signing.

The project is notably tightly coupled to Chromium: while the team aims to be minimalist & broadly useful, points of contention are explicitly resolved in Chromium's best interests [2].

As a build system, GYP is very similar to CMake, although it was designed to use abstractions familiar to web devs (e.g. a clear dependency syntax, nested contexts, exported settings, easy passing of flags to native build systems...); these abstractions help minimize frustration and lethal muckups when hacking in less familiar territory. A balanced summary once shared by Bradley Nelson on the webkit-dev listserv is available on the GYP website [3].

All further discussion of GYP in this post assumes we are building a library for consumption by other code, namely a JS environment.

node-gyp

node-gyp is the Node.js binding lib for GYP projects [4]. It installs the headers & libs necessary to enable a C++ / JS interface, and builds the native module with a standard directory structure for easy requires.

You'll frequently see node-gyp bindings exposed via the bindings module [5]. bindings supports older build standards, and will likely remain more up-to-date with node-gyp than you will (and if it doesn't, at least now you know why it's there!).

If the name seems familiar, that's likely because node-gyp several popular modules depend on node-gyp, including node-sass, node-serialport, and node-ffi [6]. People on the interwebs also frequently complain about node-gyp for a number of reasons: poorly-documented build defaults [7, 8], Windows trouble [9], lack of familiarity with lower-level build tools [10], etc.

While many of its criticisms are deserved, I've still found using node-gyp a lot more ergonomic than using GYP without it.

NAN

NAN, aka Native Abstractions for Node.js, is a node module that decouples your plugin's code from Node.js versions [11]. It defines several C++ types, methods and macros that let you write a JS / C++ boundary without regard to which Node.js version your end-user is building with (i.e. which version's headers node-gyp imported for them).

NAN is maintained by the Node.js Github org, and keeps up with new Node versions. As of this writing, it is compatible with both Node.js Current & LTS releases, plus many other versions dating back to the 0.8 series.

How do they fit together?

Assume our app will call down to native OS functions that return a string, either synchronously or asynchronously. To do so, we will need to create an npm package called "node-gyp-hello-world", and wire hello.world() & hello.worldAsync() to their respective C++ integration points.

We will build our plugin in 4 steps:

  • Define our package.json;
  • Implement our main file (index.js) & test suite (test.js);
  • Define our native binding's build script (binding.gyp); and
  • Implement our native code.
Define our addon's package.json

index.js will be our main file; tests are very lightweight and only require running test.js. Since we are building an async interface and intend to wrap it as a promise, we declare an additional dependency on RSVP.

// package.json

{
  "name": "node-gyp-hello-world",
  "version": "0.0.1",
  "author": "@anulman",
  "description": "sync & async hello world implementation with node-gyp & NAN",
  "main": "index.js",
  "scripts": {
    "test": "node test.js",
    "install": "node-gyp rebuild"
  },
  "dependencies": {
    "bindings": "^1.2.1",
    "nan": "^2.3.5"
  },
  "devDependencies": {
    "rsvp": "^3.2.1"
  }
}

Notice that we've added an install hook. By calling node-gyp rebuild during install, we ensure our native plugin is built for the host platform when installing, vs. having to pre-build and distribute binaries for arbitrary platforms (which would increase both maintenance complexity and distro size).

Implement our addon's main file & test suite;

Since our addon is quite small, index.js needs only to require & export the bindings we'll be writing:

// index.js

module.exports = require('bindings')('hello-world');  

In test.js, we want to ensure neither function throws, and that the values returned by our sync & async variants are the same:

// test.js

var RSVP = require('rsvp'),  
    assert = require('assert'),
    hello = require('./index');

// sync test
var returned;

assert.doesNotThrow(function() { returned = hello.world(); });  
console.log(`hello.world() = ${returned}`);

// async test
var promiseTheWorld = RSVP.denodeify(hello.worldAsync);  
RSVP.on('error', function(reason) {  
  assert.fail(reason);
});

promiseTheWorld().then(_returned => {  
  console.log(`hello.worldAsync() = ${_returned}`);
  if (returned !== _returned) { throw new Error("incorrect value"); }
});
Define our native binding's binding.gyp

Now that our JS interface is defined, let's describe a simple binding.gyp:

// binding.gyp

{
  'targets': [{
    'target_name': 'hello-world',
    'sources': [
      'src/hello-world.cc'
    ],
    'include_dirs': [
      "<!(node -e \"require('nan')\")"
    ]
  }]
}

Here, we are exporting a target named hello-world (the same name we passed to require('bindings') in index.js), include the NAN headers, and compile our source file(s). Which leads us to...

Implement our native code.

For those less familiar with C: this first file, while not explicitly included in our binding.gyp, is included by the source file we'll see next. Header files define a public interface, and any imports required when working with it (in our case, NAN & C++ strings).

// src/hello-world.h

#ifndef SRC_HELLO_WORLD_H_
#define SRC_HELLO_WORLD_H_

#include <nan.h>
#include <string>

class Hello {  
public:  
  static void Init(v8::Handle<v8::Object> exports);
  static std::string _World();
  static NAN_METHOD(World);
};

class HelloWorker: Nan::AsyncWorker {  
public:  
  HelloWorker(Nan::Callback *callback);
  ~HelloWorker();

  void Execute();
  std::string _world;
  static NAN_METHOD(World);

protected:  
  void HandleOKCallback();
};

#endif

Here we've defined two classes: Hello and HelloWorker. The former declares several static methods available globally on the class (e.g. Hello::World); the latter is a class that inherits from Nan::AsyncWorker and defines variables & functions its instances may use.

This leaves us with hello-world.cc, our native implementation:

// src/hello-world.cc

#include "hello-world.h"
#include <time.h>

using std::string;

using Nan::AsyncWorker;  
using Nan::Callback;  
using Nan::GetFunction;  
using Nan::HandleScope;  
using Nan::New;  
using Nan::Null;  
using Nan::Set;

using v8::Function;  
using v8::FunctionTemplate;  
using v8::Local;  
using v8::String;  
using v8::Value;

NAN_MODULE_INIT(Hello::Init) {  
  Set(
    target,
    New("world").ToLocalChecked(),
    GetFunction(New<FunctionTemplate>(Hello::World)).ToLocalChecked()
  );

  Set(
    target,
    New("worldAsync").ToLocalChecked(),
    GetFunction(New<FunctionTemplate>(HelloWorker::World)).ToLocalChecked()
  );
}

NODE_MODULE(hello, Hello::Init);

// *****
// Hello
// *****
//
string Hello::_World() {  
  return string("howdy");
}

NAN_METHOD(Hello::World) {  
  string _world = Hello::_World();
  info.GetReturnValue().Set(New(_world).ToLocalChecked());
}

// ***********
// HelloWorker
// ***********
//
HelloWorker::HelloWorker(Callback *callback): AsyncWorker(callback) {};  
HelloWorker::~HelloWorker() {};

void HelloWorker::Execute() {  
  clock_t sleep = 600000 + clock();
  while (sleep > clock());

  _world = Hello::_World();
};

void HelloWorker::HandleOKCallback() {  
  HandleScope scope;
  Local<Value> argv[] = {
    Null(),
    New<String>(_world.c_str()).ToLocalChecked()
  };

  callback->Call(2, argv);
};

NAN_METHOD(HelloWorker::World) {  
  Callback *callback = new Callback(info[0].As<Function>());
  HelloWorker *worker = new HelloWorker(callback);
  AsyncQueueWorker(worker);
};

For those less-familiar with C, let's quickly describe each of the 4 logical sections above:

  1. A preamble, where we import our header file plus any other headers required for implementation, and pre-declare a number of functions & types we'll be using in this file (this lets new devs understand dependencies right away, and keeps the code that follows nice and terse);
  2. Node bridging, where we define our C++ / JS boundary. In our case, we use the NAN_MODULE_INIT macro [12] to init our Hello::Init function with its JS target, alias target.world & target.worldAsync to their respective functions, and export the init function via the NODE_MODULE macro [13];
    https://github.com/nodejs/node/blob/master/src/node.h#L465-L466
  3. Base (sync) implementation, where we define the Hello::_World function (i.e. where you do your work), plus the Hello::World function wrapped as a NAN_METHOD that passes Hello::_World()'s return value back to JS-land [14]; and
  4. Async implementation, where we implement the HelloWorker constructor(s), a long-running HelloWorker::Execute function that caches the value returned by Hello::_World on the HelloWorker object, the HelloWorker::HandleOKCallback implementation which sanitizes and passes the cached value to JS by Calling the worker's callback, and wrap HelloWorker::World with NAN_METHOD to kick off this process.

Typically, steps 2-4 would take place in their own files, each with their own preamble; this was omitted due to the blog format.

While not demonstrated above, you may also call SetErrorMessage in an AsyncWorker subclass' Execute implementation—e.g. in a try / catch block—to pass an error to the callback, which will propagate up via your Node-style callback / denodeified promise interface [15].

Anything else?

If these tools fascinate you as much as they do me, I highly recommend digging further into their documentation, particularly the respective Hello Worlds [16].

Future posts in this series will explore how to use native libraries from within a GYP-generated build, and even build them from source if necessary.

If you're a keener and are looking for more until then, perhaps take a look at Node.js' own guide about addons for more helpful info [17], or revisit some Electron plugins [18] to check out their implementations!

Conclusion

  • GYP is a platform-independent build system that defines how a project is built with platform-specific build tooling;
  • node-gyp makes it easy to bridge projects with access to a Node.js runtime to libraries built with GYP;
  • NAN gives node-gyp developers a common API for relevant Node.js versions;
  • Writing an integration requires a little bit of boilerplate / glue, and a lot of knowledge of C / C++ / your host system's tools.

Links & References
  1. GYP - https://gyp.gsrc.io/index.md
  2. GYP: Language Spec - https://gyp.gsrc.io/docs/LanguageSpecification.md
  3. GYP: vs. CMake - https://gyp.gsrc.io/docs/GypVsCMake.md
  4. node-gyp - https://github.com/nodejs/node-gyp
  5. bindings - https://www.npmjs.com/package/bindings
  6. node-gyp: binding.gyp files out in the wild - https://github.com/nodejs/node-gyp/wiki/%22binding.gyp%22-files-out-in-the-wild
  7. "node-gyp builds crap binaries" - https://github.com/alexhultman/uWebSockets/issues/101
  8. "overriding default flags" - https://github.com/nodejs/node-gyp/issues/26
  9. "Windows users aren't happy" - https://github.com/nodejs/node-gyp/issues/629
  10. "Linking in node-gyp for non-C++ Programmers" - http://quaintous.com/2015/06/12/node-gyp-for-non-cpp-programmers/
  11. NAN - https://github.com/nodejs/nan
  12. NAN_MODULE_INIT - https://github.com/nodejs/nan/blob/master/doc/node_misc.md
  13. NODE_MODULE - https://github.com/nodejs/node/blob/master/src/node.h#L465-L466
  14. NAN_METHOD - https://github.com/nodejs/nan/blob/master/doc/methods.md#api_nan_method
  15. AsyncWorker - https://github.com/nodejs/nan/blob/master/doc/asyncworker.md#api_nan_async_worker
  16. Hello Worlds:
  17. Node.js: Addons - https://nodejs.org/api/addons.html
  18. Awesome Electron: Tools - https://github.com/sindresorhus/awesome-electron#tools

aidan nulman

partner @ isle of code. hacks with ember.js & btle (e.g. ibeacons).

About Isle of Code:

We’re a Javascript/Ember firm in Toronto + Chicago, est. 2012. We provide on-demand development for Mobile Apps, Websites, iBeacons/ hardware, existing codebases and team augmentation / Ember leadership.

read more