Refactoring Node with Higher-Order Functions

Written by Sean May

Refactoring Node with Higher-Order Functions Prior to the holidays, my colleague Nick wrote an awesome post on getting better at functional programming by stepping out of your comfort zone, and burning the boats upon the shores of strange new languages. If you did find yourself conquering the lands of Erlang, Elm, Haskell, or the isles of Akka/Scala, my hat’s off to you.

This time, I’d like like to bring the battle a little closer to home, and show you how you can use higher-order functions to clean up one of the more prominent battlegrounds in JavaScript: Node library callbacks and framework (Express, etc...) routes.

Let’s assume, for a moment, that we’re dealing with a very simple path that just returns a statically-chosen file.

import { readFile } from "fs";

router.get("/some-file", function (req, res) {  
  const url = "./data/some-file.ext";

  readFile(url, function (err, file) {
    if (err) {
      res.writeHead(404);
      res.end();
    } else {
      res.writeHead(200, { "Content-Type": "application/octet-stream" });
      res.write(file);
      res.end();
    }
  });
});

That's not a terrible amount of code to have to deal with, and it’s not as bad as it could be.
There are dozens (perhaps hundreds) of libraries we could lean on to write less code. But if you look at it, there really isn’t a lot of code there to begin with. What we do have is nesting, with the promise of more nesting, and that immediately makes certain things more difficult.

I’m typically a fan of Uncle Bob, and his suggestion might be to separate the code out, along the lines of intent. Some of this code is handling the negotiation of the request, and some of this code is about handling strategies.
This might be a little preemptive, but code in JS ought to be small and readable regardless.

router.get("/some-file", function (req, res) {  
  const url = "./data/some-file.ext";

  function handleError (err) {
    res.writeHead(404);
    res.end();
  }

  function handleSuccess (file) {
    res.writeHead(200, { "Content-Type": "application/octet-stream" });
    res.write(file);
    res.end();
  }

  readFile(url, function (err, file) {
    if (err) {
      handleError(err);
    } else {
      handleSuccess(file);
    }
  });
});

We’ve gone about as far as we can or should go for now. And we’ve pulled those behaviours out of the logical flow. Perhaps this is preemptive, and there are good arguments for leaving big goopy messes alone until they can be refactored to be made of reusable behaviours (rather than just breaking an algorithm into sequential phases; don’t make Uncle Bob cry). But, the goal is to prevent it from happening in the first place.

To that end, we can see that this pattern will quickly become boilerplate.

function handleError (err) { /* ... */ }  
function handleSuccess (x) { /* ... */ }

function nodeCallback (err, thingIWant) {  
  if (err) {
    handleError(err);
  } else {
    handleSuccess(thingIWant);
  }
}

someNodeFunction(arg, nodeCallback);  

It’s way less than what you might have in a language like Java, but it’s there, nonetheless. Also, all three functions are still cursed with the need to be kept inside of the route.

// PROBLEM 1:
// This will blow up, because handleSuccess / handleError don't have access to `res`
function handleSuccess (data) {  
  res.writeHead(200, { "Content-Type": "application/octet-stream" });  // *BOOM*
  res.write(data);
  res.end();
}
function handleError (err) { /* ... */ } // *ALSO BOOM*

router.get(path, function (req, res) {  
  readFile(filePath, function (err, file) {
    if (err) {
      handleError(err);
    } else {
      handleSuccess(file);
    }
  });
});


// PROBLEM 2:
// This will blow up, because nodeCallback has no access to handleError or handleSuccess
function nodeCallback (err, data) {  
  if (err) {
    handleError(err); // *BOOM*
  } else {
    handleSuccess(data); // *ALSO BOOM*
  }
}

router.get(path, function (req, res) {  
  function handleSuccess (file) { /* ... */ }
  function handleError (err) { /* ... */ }

  readFile(filePath, nodeCallback);
});

Because they’re so tightly tied together, this pattern is suitable for one-offs, but it’s still not really unit-testable or reusable. This is bleak, because even if our code gets a little clearer, we still have a lot of stuff buried inside of that route.

But we have a secret weapon in JS, and functional languages in general. We’ve been hurt by it thus far, but it’s a tool we’ve simply misunderstood.

Higher Order Functions

A higher order function is simply a function which takes a function as an argument to be used, or is a function returned as a result (or both).
In most languages, even if you could return a function from a function, it wouldn't do you much good. But in most functional languages, thanks to the concept of Lexical Scoping, we have access to Closure.

You’ve been using it all along, in the router callback, and in the Node readFile callback.

We can take this to the next level, to make this code cleaner and easier to test. First, we need to make sure we understand what we’re really dealing with, and our desired outcome.

function rememberX (x) {  
  return function getX () {
    return x;
  };
}

// ES6 version might look like the following
// const rememberX = x => () => x;

const get42 = rememberX(42);  
const get36 = rememberX(36);  
const get18 = rememberX(18);  

Let that marinate for a second. We’re passing a value of x and we’re getting a function back on the other side.
But, why would we want to do that?

let x = 96;  
get42(); // 42  
get36(); // 36  
x = 12;  
get18(); // 18  
x; // 12  

Note that the values of x have all been protected from the changing value of x on the outside. That’s because the x which is referenced is the one that is visible at the time the function is created, and not at the time the function is called. So by passing that newly created function back to the outside world, we remember the value which was passed into the outer function as an argument.

function multiply (multiplier) {  
  return function (x) {
    return x * multiplier;
  };
}

// ES6:
// const multiply = x => y => x * y;

const triple = multiply(3);  
const quadruple = multiply(4);

const numbers = [1, 2, 3];  
numbers.map(triple); // [3, 6, 9]  
numbers.map(quadruple); // [4, 8, 12]  
numbers.map(triple).map(quadruple); // [12, 48, 108]  

There’s a lot of power in using these types of functions in order to preserve references to configuration, but pass the function around to be used in multiple places.
Knowing this, we should be able to find a way to build a function which holds some information, and returns a preconfigured function ready to be used by something else.

function higherOrderNodeCallback (onSuccess, onError) {  
  function nodeCallback (err, thingIWant) {
    if (err) {
      onError(err);
    } else {
      onSuccess(thingIWant);
    }
  }
  return nodeCallback;
}

If you look at the function that gets returned, it should be clear as to what it is doing. That is, it’s doing exactly what the nodeCallback was doing, previously.

The function on the outside, however, is taking in a success callback and an error callback.

It’s configuring that instance of the inner function to refer to those things, so that when the returned function is called, it can still access them, the same way rememberX worked.

router.get(url, function (req, res) {  
  function handleSuccess (file) {
    res.writeHead(200, { "Content-Type": "application/octet-stream" });
    res.write(file);
    res.end();
  }
  function handleError (err) {
    res.writeHead(404);
    res.end();
  }
  const callback = higherOrderNodeCallback(handleSuccess, handleError);
  readFile(filePath, callback);
});

We’ve successfully removed the Node boilerplate, by putting it in a configurable higher-order function. If we wanted to take that even further, we could create similar configurations for the success and error handlers.

function closeWithStatus (response, status, headers) {  
  const data = headers || {};
  function end () {
    response.writeHead(status, data);
    response.end();
  }
  return end;
}

function closeWithContent (response, status, headers) {  
  const data = headers || {};
  function endWith (content) {
    response.writeHead(status, data);
    response.write(content);
    response.end();
  }
  return endWith;
}

function higherOrderNodeCallback (onSuccess, onError) {  
  return function nodeCallback (err, content) {
    if (err) {
      onError(err);
    } else {
      onSuccess(content);
    }
  };
}

router.get(url, function (req, res) {  
  const headers = { "Content-Type": "application/octet-stream" };
  const onSuccess = closeWithContent(res, 200, headers);
  const onError = closeWithStatus(res, 404);
  const callback = higherOrderNodeCallback(onSuccess, onError);

  readFile(filePath, callback);
});

Our router logic is super clean, and it should be pretty clear what everything is doing, now that we have a grasp of the how and why.
If your team was comfortable with functional programming, perhaps just this, instead:

readFile(  
  filePath,
  higherOrderNodeCallback(
    closeWithContent(res, 200, headers),
    closeWithStatus(res, 404)
  )
);

You don’t have to write your JavaScript like a LISP, of course; but if it works for your team...

This pattern has some fun properties. By assuming nothing of the outside world, except what it’s given, we’ve created code that is happily:

  • Testable
  function passTest () { /* ... */ }
  function failTest () { /* ... */ }

  const errorCallback = higherOrderNodeCallback(failTest, passTest);
  errorCallback(errorObj, null);

  const successCallback = higherOrderNodeCallback(passTest, failTest);
  successCallback(null, successObj);
  • Reusable
  export const doSomething = higherOrderNodeCallback(doA, doB);

  import doSomething from "./do-something";
  fstat("./some-file", doSomething);
  • nearly SOLID, already. Yes, okay. Uncle Bob never meant for his principles to apply to functional programming, and it almost feels like cheating here, because getting most of the way there was so easy... and we aren't using inheritance, so getting the rest of the way there is wholly unnecessary.

That’s not the end, of course. You can happily use your composed functions inside of other higher-order functions.

function runAsPromised (task, ...params) {  
  return new Promise((resolve, reject) =>
    task(...params, higherOrderNodeCallback(resolve, reject))
  );
}

runAsPromised(readFile, filePath)  
  .then(convertToContent)
  .then(JSON.stringify)
  .then(closeWithContent(res, 200, jsonHeaders))
  .catch(closeWithStatus(res, 404));

“But why would I want to use this technique, when there are dozens of libraries that can do this for me?”

The reality is that those libraries consistently make code cleaner and easier to write. If you have them and they make life simpler, then use them. That said, if there was a library to solve every problem, and a library to tie all libraries together, to produce business value, then we would likely not have jobs.

This pattern, however, can be applied at any level, and is particularly good at separating code that provides value from code that keeps the system running.

const displayItems = storeItems  
  .filter(filterBy(systemCriteria))
  .filter(filterBy(customerCriteria))
  .sort(sortBy(primaryDimension, secondaryDimension));

There are even techniques for building these configurable functions automatically; “Partial Application” and “Currying”. But that’s a talk for another day.

Learn More

If you're looking for more in-depth tutorials and learning opportunities, check out our various JavaScript training options. Better yet, inquire about custom training to ramp up your knowledge of the fundamentals and best practices with custom course material designed and delivered to address your immediate needs.