Typescript async/await with Node

Typescript 1.7 added async/await, which is going to be a game changer for node usability. To demonstrate how well it works, I picked out a few examples from a project I've been working on. To add more context to these: I started out writing this project with callbacks, later switched to promises, and then added typescript.

A simple function

This is a function to walk a directory tree and return a list of files. It includes calls to the node functions stat and readdir, which I've promisified using when.js. Although an improvement over callbacks, the promises still suffer from poor readability.

var stat = nodefn.lift(fs.stat);
var readdir = nodefn.lift(fs.readdir);

export function walkDir(d) {
    return stat(d).
        then(function(stats) {
            if (stats.isDirectory())
                return readdir(d).
                    then(function (files) {
                        return when.map(files, function (file) {
                            return walkDir(path.join(d, file));
                        }).
                            then(flatten);
                    });
            else
                return [d];
        });
}

Here is the async/await version written in a more readable imperative style.

export async function walkDir(d) {
    var stats = await stat(d);
    if (stats.isDirectory()) {
        var files = [];
        for (let file of await readdir(d)) {
            files = files.concat(await walkDir(path.join(d, file)));
        }
        return files;
    } else {
        return [d];
    }
}

One important point is that async/await and native ES6 promises remain interoperable with other A+ implementations. So you can promisify node calls using Bluebird or something similar and they become awaitable. Its easy to refactor promise based code by following a few simple rules:

  • Anything returning a promise is await-able.
  • Any function using await must be async.
  • All async functions return a promise.

Loops

The next example demonstrates something promises have difficulty with: async inside a loop of unbounded size. This function finds a unique slug for a new blog post by appending an incrementing ID. After a lot of effort, this was solved this by creating an inner recursive function.

getNextAvailable(n, prefix) {
    var that = this;
    function chain() {
        return that.publisher.exists(prefix + n).
            then(function (exists) {
                if (exists) {
                    n++;
                    return chain();
                } else {
                    return prefix + n;
                }
            })
    }
    return chain();
}

Anyone who hasn't struggled with this problem would probably be confused by that mess. This is where async/await really excels:

async getNextAvailable(n, prefix) {
    while (await this.publisher.exists(prefix + n)) {
        n++;
    }
    return prefix + n;
}

Concurrency

Finally, one last example. This function has four async calls, but already the readability is getting out of hand.

reIndex() {
    return this.publisher.list().
        then(keys => {
            return when.map(keys, key => {
                var u = url.resolve(this.config.url, key);
                return this.publisher.get(key).
                    then(obj => {
                        if (obj.ContentType == 'text/html')
                            return microformat.getHEntryWithCard(obj.Body, u).
                                then(entry => {
                                    if (entry != null && (entry.url === u || entry.url + '.html' === u))
                                        return this.db.storeTree(entry).
                                            then(() => debug('indexed ' + entry.url));
                                });
                    });
            });
        }).
        then(() => debug('done reindexing'));
}

That looks terrible, but take a look at the async version. Its probably not hard to figure out what this does now.

async reIndex() {
    var keys = await this.publisher.list();
    for (var key of keys) {
        var u = url.resolve(this.config.url, key);
        var obj = await this.publisher.get(key);
        if (obj.ContentType == 'text/html') {
            var entry = await microformat.getHEntryWithCard(obj.Body, u);
            if (entry != null && (entry.url === u || entry.url + '.html' === u)) {
                await this.db.storeTree(entry);
                debug('indexed ' + entry.url);
            }
        }
    }
    debug('done reindexing');
}

What if publisher.get() has 20ms of latency and there are 1000 objects? In this scenario, the for loop could be orders of magnitude slower since its fetching objects serially. To get concurrent behavior, we can await when.map, and continue using async/await in the lambda passed to it:

async reIndex() {
    var keys = await this.publisher.list();
    await when.map(keys, async (key) => {
        var u = url.resolve(this.config.url, key);
        var obj = await this.publisher.get(key);
        if (obj.ContentType == 'text/html') {
            var entry = await microformat.getHEntryWithCard(obj.Body, u);
            if (entry != null && (entry.url === u || entry.url + '.html' === u)) {
                await this.db.storeTree(entry);
                debug('indexed ' + entry.url);
            }
        }
    });
    debug('done reindexing');
}

A couple things to note:

  • You can mark arrow functions async too.
  • An arrow function is preferable here, so this doesn't need to be captured in a closure.

 #code#typescript
Have you checked out typescript? I posted some thoughts on how it alleviates callback hell: http://notenoughneon.com/2016/3/25/typescript-async-await-with-node