We were having serious memory issues on the Heroku hosted NodeJS backend for our game [BattleTabs](https://battletabs.com) when we decided to apply "this one little trick"...



# TLDR;

Use [Jemalloc](https://github.com/jemalloc/jemalloc).

# The Problem

For those that aren't regular readers of this blog, for the past year my main focus has been a chrome extension game called [BattleTabs](https://battletabs.com). The game is based around the classic board-game BattleShips except you can play it in a Chrome Extension. This makes it rather unique as the game is immediately available every time you open a new tab in your browser. You can take a turn and move on with what you were doing or you can sit and play a longer session.

Because of the way the game works we keep a persistent socket open to our backend at all times. This allows us to keep the user's state in sync by pushing down updates from the server.

The game has grown considerably over the past 12 months and as a result the demands on the server have continued to grown. That coupled with the increased feature set has seen more and more open sockets and more required memory per socket.

# Vertical to Horizontal

We follow a pragmatic development strategy for the game, meaning that we only spend the time working on improving the infrastructure if its necessary. This meant that for 9 months we weren't really worried about server scaling or performance. We knew the issue was there in the background but whenever we needed more capacity we just bumped up the server size in Heroku:

![](./heroku-dyno-sizes.png)

Well unfortunately the day game when we were up to the largest size dyno and we were starting to see some issues again, it was time to do something about it.

The issue wasn't really that of memory at this stage, instead it was the number of simultaneous open sockets to a dyno instance. As far as I am aware there is no docs on what the max number of sockets is per dyno but [this](https://stackoverflow.com/questions/25118381/websocket-concurrent-connections-limit-on-heroku) stack overflow answer seems to suggest 6000.

We were definitely right up against that but at the same time because we are on node we are single threaded so we were also up against a CPU bottleneck trying to handle all the requests when the dynos go through their daily restart.

The time had come to go from a horizontal scale strategy to a vertical one.

We had been putting this off because horizontally scaling a websocket server isn't trivial. Fortunately we didn't have that much local-state and we were using [Socket.io](https://socket.io/) which has some good plugins to help with scaling. I wont detail everything we did here, feel free to contact me if you want details.

![](./socketio.jpg)

# Memory Issues

Now we could horizontally scale our server out to as many dynos as we wanted we should in theory be able to drop our dyno size back down to the smallest size (512mb RAM) and then just add more dynos as we grew.

The problem is, we seemed to still be using far more memory than we should have been given we were only running less than 1000 sockets per dyno. Worse still the memory seemed to be increasing over time indicating that there might be a memory leak somewhere.

After days of experiments and searching I decided to look a little closer at what socket-io was doing, thats when I came across this Github issue: https://github.com/socketio/socket.io/issues/3477

![](./mem-leak.png)

Others on the issue had done the hard work for me and instead of a memory leak they suggest memory fragmentation was the root cause of the problem.

This was a big "Ahhh!" moment for me as memory fragmentation would look a lot like a memory leak as your available memory over time would decrease as the memory became more and more fragmented.

There were two suggested solutions on the thread. I tried the first which was to disable "perMessageDeflate" on SocketIO. This had some effect but still we seemed to be using far more memory than we should have been.

So I decided to try the second solution too which was something I would never have considered; replace the system memory allocator in Node.

# Jemalloc

The other users on the Github Issue suggested trying [Jemalloc](https://github.com/jemalloc/jemalloc). I personally had never heard of it before but apparently its well known and popular with 5,500+ stars on Github. It describes itself as:

> jemalloc is a general purpose malloc(3) implementation that emphasizes fragmentation avoidance and scalable concurrency support.

Sounds like its exactly what we need, but how do I get node to use that? As with most things Heroku, someone has thankfully trodden this path before me and written a "Buildpack" which I can just plug into my app: https://elements.heroku.com/buildpacks/gaffneyc/heroku-buildpack-jemalloc

And the effect?

![](./half-mem.png)

Wow! It halved our memory usage! Incredible!

# Conclusion

We were able to scale our heroku server out horizontally and overcome a memory fragmentation issue in socket-io by replacing the memory allocator with Jemalloc.

We have been running like this for a couple of weeks now and have yet to see any issues.

I haven't tested this on non-SocketIO servers but I suspect so I cant say for sure but I suspect it would have a similar magical effect. I would be very interested to hear your results if you tried this!

Wow its taken a little longer than we would have liked but finally we have finally released the completely redesigned and rewritten [Markd](https://markd.co) website, extension and server!



# Design

So we teamed up with an old colleague of mine [Alan Macfarlane](https://www.melbournemonsters.com/) to help us with the designs and branding. We wanted to go for something that would make our brand stand out, not feel too formal and perhaps have a little "gamey" feel to it as both Brandon and I are long time game developers.

Alan went away and came up with a bunch of awesome ideas for characters which we then used as the basis for many of the pages.

![](./different characters.png)

In the end we went with a very simple and stylized version that we thought suits the brand we want, is easily extensible and hopefully easy to animate in the future should we wish to.

![](./login-page.png)

Alan also was responsible for our new logo which I think looks awesome when shown off in its animated form:

<img alt="markd logo animated large" style="box-shadow: none !important; margin-bottom: 20px" src="./markd-logo-animated-large.gif" />

For this version of Markd we decided to split the website up into two parts;

"Static" pages live on https://markd.co and are designed in Webflow for fast tweaking and A / B testing by Brandon.

![](./markd.co.png)

"App" pages live on https://app.markd.co and is Typescript / React / Single Page Application (SPA) and is where I have spent most of my time.

![](./marks-page.png)

So thanks to Alans awesome designs and after a lot of hard work I think we have ended up with a really good looking and easy to use site for collecting and organizing marks.

Heres a short video showing some of the main features:

<iframe width="853" height="480" src="https://www.youtube.com/embed/k7Ks42OSyTI" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# Tech

Behind the scenes a lot of work has gone into ensuring the tech is in a good stable position for us to build future features on.

I decided that our [Parse-Server](https://github.com/parse-community/parse-server) based server that we have been using up till now just wasnt going to cut it for some of the future features we had planned.

So right at the start of the project we took (not lightly) the decision to completely rip out our backend and start again, then migrate the old database over.

Due to the highly document-orientated nature of our data we decided to stick with the no-sql database MongoDb and then interface with it via node. For the Client to Server querying and mutating we decided to depart from my usual choice of REST based API and instead decided to go with GraphQL.

GraphQL should in theory be much better for us in the long term, allowing us to develop APIs that can grow with us and be compatible without a huge parameter or endpoint explosion you might end up with REST. For now I am yet to be convinced, there is definitely some benefits but also quite a bit of extra overhead to deal with when using GraphQL, im just hoping that over time as things mature some of those issues will go away.

# Testing

As we are just a small 2-man team I knew testing such a large set of changes was going to be a challenge particularly when we had added a whole bunch of new features in the process.

Our strategy with testing is to test in depth. So firstly we have the automated unit and integration testing for the server-side code.

![](./server-integration-tests.png)

These tests hammer the actual API from the outside which then hits a local database. This gives us a high level of confidence that when the client uses the same API to call the server will get the expected results.

On the client side I actually have very few "unit" tests and instead opt for a few high-level End-To-End (E2E) tests which use [Google Puppeteer](https://github.com/GoogleChrome/puppeteer) to spin up an actual browser which then clicks on actual elements on the screen and observes the results match the expected results.

<iframe width="853" height="480" src="https://www.youtube.com/embed/iyl7XSTZE-I" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

I run these E2E tests twice, once for desktop sized resolutions and once for mobile sized resolutions.

All these automated tests are run on every single code push by the CI server which gives us a good level of confidence that the core functionality is working after every change.

Following that however we do a smoke-testing pass on our "staging" environment before we merge any new changes into the "master" environment.

For very large updates such as this complete redesign we knew we needed a little more testing so we contracted a freelance QA tester Seb to help us get the confidence up that everything works on a number of platforms.

# Conclusion

Time will tell if this was worth all the effort. Personally I think we have done a great job and im proud of the results. Im feeling much better about the foundations we have now to go ahead and build the next round of exciting features, so stay tuned more more!

Wow, well that took longer than expected! [44 days ago I blogged](/posts/tinkering-with-typescript/) that I had started work on a second version of my Chrome Crawler extension and have only just managed to get it to a state I was happy with enough to release it. To be fair I had been on [a trip to New York](/posts/new-york-new-york/) during that period so perhaps I can be excused. Having said that however I think the time has been well spent and am fairly proud of the result.



### TL;DR

Recursive is an experimental tool for visualising the world wide web. Given a URL it downloads the page search for links and then recursively downloads those. The information is then displayed in a node-based graph.



### The Name

So what's this all about? Why is it called 'Recursive', why not 'Chrome Crawler 2'?

Although I would like to have called the spiritual successor to ['Chrome Crawler'](/posts/chrome-crawler-a-web-crawler-written-in-javascript/), 'Chrome Crawler 2' Chrome's [branding guidelines](https://developers.google.com/chrome/web-store/branding) forbid using the Chrome name or logo (they brought this in since the launch of Chrome Crawler 1).

With that in mind I decided that rather bend Chrome Crawler's name and logo to fit the guidelines I would create a whole new logo, name and app. The app is a total rewrite from the previous iteration anyway so I thought it justified.

According to dictionary.com there is no definition for "Recursive" or "Recurse" but there is one for "Recursion":

> 2\. the application of a function to its own values to generate an infinite sequence of values.
> So a tool that downloads pages, follows the links on that page to download other pages seemed like a rather apt description of something that is "Recursive".

### Video

Before I go much further, I put together this little video demonstrating some of the extensions core functionality:

<iframe width="640" height="360" src="https://www.youtube.com/embed/oNdpoM5Vhsc" frameborder="0" allowfullscreen></iframe>

### Installing

Installing and upgrading is dead simple thanks to how Google Chrome's extension system works. Just head over to this link and hit install:

[https://chrome.google.com/webstore/detail/recursive/hbgbcmcmpiiciafmolmoapfgegbhbmcc](https://chrome.google.com/webstore/detail/recursive/hbgbcmcmpiiciafmolmoapfgegbhbmcc)

Then to launch it visit any website and hit the little icon in the Omnibox:

[![](/wp-content/uploads/2012/12/Screenshot_008.png "Screenshot_008")](/wp-content/uploads/2012/12/Screenshot_008.png)

### How it works

Recursive works by taking in a starting URL which it uses to download the page it points to:

[![](/wp-content/uploads/2012/12/03.jpg "03")](/wp-content/uploads/2012/12/03.jpg)

Once that page is downloaded Recursive parses it looking for links and files. If it finds things it thinks are files then it records them against that URL. It then proceeds to visit all the links in turn, downloading the page then parsing the for yet more files and links.

This cycle continues until a certain "depth" is reached which is the maximum number of links away from the starting URL. You can set the maximum depth allowed in the settings:

[![](/wp-content/uploads/2012/12/04.jpg "04")](/wp-content/uploads/2012/12/04.jpg)

One of the key improvements of Recursive over Chrome Crawler is the way it visualises the data as it is returned:

[![](/wp-content/uploads/2012/12/05.jpg "05")](/wp-content/uploads/2012/12/05.jpg)

Every page is grouped by its domain and is represented by a circular "node".

[![](/wp-content/uploads/2012/12/Screenshot_013.png "Screenshot_013")](/wp-content/uploads/2012/12/Screenshot_013.png)

So for example "/posts/tinkering-with-typescript/" would be grouped under the "mikecann.blog" domain. Any other pages found while running that match this domain are added as little page icons inside the host node.

[![](/wp-content/uploads/2012/12/Screenshot_014.png "Screenshot_014")](/wp-content/uploads/2012/12/Screenshot_014.png)

Any files that are found on a given page are given an appropriate icon and added to that page's domain node.

[![](/wp-content/uploads/2012/12/Screenshot_015.png "Screenshot_015")](/wp-content/uploads/2012/12/Screenshot_015.png)

As Recursive downloads pages and follows links it records the path it takes. It then draws lines between the nodes that are linked:

[![](/wp-content/uploads/2012/12/06.jpg "06")](/wp-content/uploads/2012/12/06.jpg)

### Interacting

Using the mouse wheel you can zoom in and out to get a better perspective. Click and drag to move about the recursive space. You can also run the app in fullscreen if you so desire.

[![](/wp-content/uploads/2012/12/Screenshot_017.png "Screenshot_017")](/wp-content/uploads/2012/12/Screenshot_017.png)

If you click on a node it tells Recursive to explore that node for one extra level depth.

Right clicking a node opens a menu that lets you either open all the pages contained in that node or view the files for that node.

[![](/wp-content/uploads/2012/12/Screenshot_019.png "Screenshot_019")](/wp-content/uploads/2012/12/Screenshot_019.png)

### Files

By using the context menu for a node you can checkout all the files that Recursive found for that node. The files are separated into various categories which you can toggle on or off:

[![](/wp-content/uploads/2012/12/Screenshot_020.png "Screenshot_020")](/wp-content/uploads/2012/12/Screenshot_020.png)

Then if you wish you can download all the files as a zip.

### More

If you would like to read more on recursive, then checkout these posts:
[catlist id=30 numberposts=100]