17 06 2015
Using generators for node.js style callbacks
Embellishment of a story
Recently I had to setup a new mini webserver. The functionality of this server was not very complex:
1. Fetch JSON data from [Jenkins](http://jenkins-cli.com) server
2. Transform JSON into another format
3. Read files from the file system
4. Return the result to the user
I took express.js and node-jenkins package as server stack. Express was handling http/https requests and node-jenkins package was responsible for making requests to the Jenkins server.
To structure asynchronous code I’ve used async.js library. This is because jenkins package and fs module of node.js are both based on error first callbacks.
When everything was done, I suddenly realized that world is almost up to the next version of the JavaScript specs – ECMAScript-2015 and that there are generators available to us. Generators allow halting the execution flow inside the function and resume it later. This gives the possibility to use generators as an alternative for handling asynchronous code in a synchronous manner.
var result = yield asyncMethodOne(); var result2 = yield asyncMethodTwo();
Support of the generators is already added to node.js by using –harmony flag and I could make use of that to restructure code of the webserver.
I’ve opened up Google and looked at the existence of the available libraries for using as generators runners. I’ve found a nice overview http://blog.vullum.io/nodejs-javascript-flow-fibers-generators/ which shows different approaches to the same problem.
After looking at all these packages, a couple of things still bothered me.
First of all it appears that the current node code is not really suitable for all this generators stuff, unless it’s written with Promises. Which means that to make my current code work I will have to wrap all the code with promises, e.g with promisify:
var processedData1 = yield Promise.promisify(fs.readFile)(inputFile);
There was also another approach to use generators and “old-school-node-js” functions. In this case I had to use bind for every function and pass it to the yield statement, so that the generator runner could pass itself as a callback to yielded function.
var data = yield fs.readFile.bind(fs, inputFile);
I didn’t like both solutions. The first one does not look very clean and put an extra abstraction which I don’t like. The second one gives wrong expectation to the reader of the code, because executing function fs.readFile(inputFile)
means something different than binding the function fs.readFile.bind(fs, inputFile)
.
Yielding the yield
From the beginning of time was node.js implemented using error-first callbacks. It is so fundamental to node.js, that a major part of libraries in the npm registry works this way and every developer who uses node.js knows how to apply technique of error-first callbacks.
When using yield constructs within the asynchronous code you have to choose whether you use promises or wrap around your calls. My idea was to create a library which would work with error-first callback functions in a natural way, without wrappers and promises. I called it yield-yield
To show how it is working, let’s walk through a sample code which reads a file from the file-system.
var fs = require('fs'); var inputFile = '/etc/hosts'; var callback = function (err, content) { if (err) { console.error('Error when opening file: ' + err.message); } /* code */ }; fs.readFile(inputFile, 'utf8', callback);
In order to use yield-yield library, yield statement have to be passed to fs.readFile
function instead of a usual callback.
fs.readFile(inputFile, 'utf8', yield);
After that, to get the content from the fs.readFile
second yield statement must be used at the beginning of the call:
var fs = require('fs'); var inputFile = '/etc/hosts'; var content = yield fs.readFile(inputFile, 'utf8', yield);
The content which is normally passed by the fs.readFile
as the second argument will returned instead. If for some reason fs.readFile
will pass a first argument, in cases of error, then this yield will throw an Error
. In this way it can be caught using standard try {} catch (e) {}
syntax.
var fs = require('fs'); var inputFile = '/etc/hosts'; var content; try { content = yield fs.readFile(inputFile, 'utf8', yield); } catch (e) { console.error('Error when opening file: ' + err.message); }
Because yield statement is used, this code must be placed inside a generator function with an asterisk.
var fs = require('fs'); var readHostsFile = function *() { var inputFile = '/etc/hosts'; var content; try { content = yield fs.readFile(inputFile, 'utf8', yield); } catch (e) { console.error('Error when opening file: ' + err.message); } };
Still, it’s not enough to put an asterisk in a function definition.
Generators are supposed to be handled differently, using .next()
method of the generator. In order to use this generator yield-yield must be used as a runner for this generator.
To keep yield-yield compatible with exciting code, yield-yield runner will transform any given generator into the error-first callback function, so that it can be executed as if it is not a generator:
var sync = require("yield-yield"); var readMyFile = sync(function* () { try { var content = yield fs.readFile(inputFile, options, yield); } catch (e) { console.log('error detected'); } }); readMyFile(function () { console.log('Function is done'); });
In this way, yield-yield is compliant with existing code and can be used as an enrichment instead of a full replacement.
All the possible execution flows of the yield-yield can be found in the documentation.
Using yield-yield in real-world examples
Even if you don’t plan to transform your code to the generators ready solution, you can start applying yield-yield to different parts of your operations code. For example inside tests, or building tools or anything else.
Mocha
Mocha is one of the libraries which can run tests using error-first callback mechanism. Let’s see how asynchronous test can be translated into the yield-yield code.
First of all, code before transformation:
describe('file', function() { test('should do something async', function(done) { methodone(function() { methodTwo(function() { return done(); }); }); }); });
Now we can wrap testing code into the yield-yield runner and use yield statements to do async execution:
var sync = require('yield-yield'); describe('file', function() { test('should do something async', sync(function* () { yield methodOne(yield); yield methodTwo(yield); }); });
If methodOne
or methodTwo
will throw an exception or will pass an error as the first parameter, then test will fail.
x file should do something async
Also you can see, that there is no done()
call at the end. Because mocha passes done
as an argument to the test function, yield-yield will execute it automatically at the end of the code flow.
PostCSS-cli
I took another library postcss-cli and looked what can be changed. Postcss-cli is used as a command line tool for postcss. Inside index.js there is enough processing using async.js
I took method processCSS
and translated it into the yield-yield construction without changing all the other code, and it stil works.
GitHub
Code is available on GitHub: https://github.com/nemisj/yield-yield
How to test React.js components Obscure way to repeat string ( till 32 times )