
So the big day is finally here! The moment we have been working on since we decided to bring our little [Chrome extension / Web game](https://battletabs.com/) to mobile some 2 years ago.

We have finally made the game globally available on both iOS and Android!

[![](./download-on-app-store.png)](https://apps.apple.com/us/app/battletabs/id6476166902)
[![](./get-on-google-play.png)](https://play.google.com/store/apps/details?id=io.battletabs)

Don't know what I'm talking about? Then check out our trailer:

<iframe width="853" height="480" src="https://www.youtube.com/embed/wGgJjEvNgt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

If you are interested in a little more detail about the history of the game and where that funny name comes from then you can check out [another video we put together here](https://mikecann.blog/posts/battletabs-in-7-minutes).

# Since Soft Launch

Over 6 months ago [we announced](https://mikecann.blog/posts/battletabs-enters-soft-launch) we were entering Soft Launch for mobile. Since then we have been iterating, adding features, tweaking features and fixing bugs. Things have taken a little more time than expected mainly because we decided to focus on our [Discord Activities launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities) which seemed like (and has been) a great opportunity for us.

With that behind us now we are back focused on mobile. The main priority has been performance and stability. One issue with building your mobile game using web technologies is that you unfortunately don't get as much control over resources such as GPU memory as you might do if you were going with a more native approach.

This means we have been at the whim of whatever browser technology is available on the platform. In the case of iOS this means WebKit. It has been known for a while now that WebKit on iOS isn't great, conspiracy theorists would say this is because Apple is intentionally trying to hobble web experiences on mobile in an effort to boost App Store sales. I don't know, all I know is that we have had a bit of a hard time with GPU memory crashes on iOS mobile.

These crashes can happen at just about any point in the game which makes it really tricky to replicate and thus debug the problem. The tooling that Apple provides (only available on Macs of course) isn't great and unfortunately crashes all the time when we try to use it.

We have narrowed down the bulk of the crashing is related to [browser layering](https://medium.com/masmovil-engineering/layers-layers-layers-be-careful-6838d59c07fa) particularly when combined with CSS animated elements. This seems to force a high GPU memory usage as WebKit attempts to hardware accelerate large layers.

Curiously Chrome (and thus Android) does not have this problem so I, as a Windows user, am not able to debug this problem from my PC.

We don't have a definitive solution for the problem, I suspect the issue is still there but we have tried our best to minimize it by removing most usages of CSS animations (on iOS) and changing the League Page over to a much simpler design as it was the culprit of many crashes.

[![](./new-league-page.png)](./new-league-page.png)

# Apple Feature

A few weeks back we excitedly received an email from Apple informing us that we may be featured on the App Store.

[![](./app-feature.png)](./app-feature.png)

We submitted all the media information required so fingers crossed we do. Apple declined to comment on likelihood or timelines so we will see. If we do get a feature then it should hopefully be a big boost to game traffic. In preparation we have scaled up our servers just in case, I don't want another repeat of the [Discord Activities global launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities) ;)

*EDIT*: [We were indeed featured in the US](https://www.linkedin.com/feed/update/urn:li:activity:7258089646545334273/)!

[![](./were-featured.jpeg)](./were-featured.jpeg)

# Conclusion

So it's been a long time in the making but we are super excited to have the game finally available to (just about) everyone's mobile device and tablet. I'm looking forward to battling with you guys from around the world soon!

[![](./download-on-app-store.png)](https://apps.apple.com/us/app/battletabs/id6476166902)
[![](./get-on-google-play.png)](https://play.google.com/store/apps/details?id=io.battletabs)



For the [longest time](https://mikecann.blog/posts/battletabs-6-months-later), Gangbusters has been a Discord-focused company. We use it as our primary method of interacting with our community, receiving bug reports, and gathering feature suggestions.

So when [we heard about](https://venturebeat.com/games/discord-launches-activities-with-light-games-and-a-new-app-store/) Discord's intention to make games playable within the app itself, we knew it was a strong opportunity for our game [BattleTabs](https://battletabs.com/).

Discord is a game-focused social platform where many players are hanging out with friends and looking for something to play together. Having our social multiplayer game embedded right there was a match made in heaven.

![](./battletabs-in-the-app-drawer.png)

In addition to reducing the discovery friction for BattleTabs, Discord also offers a number of other compelling advantages. One of these is the ability to directly message players (we had early access to this feature). We currently use it as another Push Notification channel, but it’s so much more powerful than that—we’ve barely scratched the surface of what’s possible.

![](./discord-dms.png)

Super-fast and friction-free payments are another strong benefit of using Discord, particularly over the web platform where the friction for paying for IAPs or subscriptions makes it super challenging to monetize games like ours. Discord even provides an app "store" where you can see all products that the game offers and purchase them directly without even opening the game.

![](./showing-the-store.png)

There are many other strong reasons to be excited about the Discord platform for games, especially when you read about the [incredible success](https://mojiworks.com/news/why-were-all-in-on-discord-activities) stories [from](https://a16z.com/discord-activities-social-gaming/) other [developers](https://discord.com/build-case-studies/frvr). You can see why we were keen to get on there.

Fast forward a year or two, and after much hard work by [Brandon](https://www.gangbusters.io/about), we found ourselves in the inner circle as one of Discord’s Developer Partners. The process of getting BattleTabs on Discord had begun.

The first step was a "geo test" or soft launch, focusing on Canada. This was an opportunity for Discord to see how the game performed before deciding whether to move forward with a global launch.

![](./happy-meme.png)

After two weeks of the game being available in Canada, we sat down with Discord again, and they gave us the very exciting news that the game had done exceedingly well. As a result, they wanted to take us to a global launch.

We scheduled the official global launch for 10 a.m. my time (Western Australia) on Tuesday, September 10th.

# Tuesday

After a brief call with Discord, we went live! We immediately started to see our usual traffic numbers get obliterated.

![](./initial-spike-graph.png)

And this is where the fun started. To explain the issues we were soon to face, I first have to explain the architecture for BattleTabs a bit.

![](./architecture1.png)

You can see from the above diagram that we have two different kinds of servers. Users connect to the "Web" servers via WebSockets, these scale out horizontally and generally aren’t a problem.

The other kind is the "worker" server. This is a singleton server and was serving many roles (foreshadowing). Some of its responsibilities include:

+ Listening for "app events" and performing various tasks like updating challenge progression, sending push notifications, awarding medals, updating league positions, etc.
+ Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.
+ Listening to database events and performing logic, such as alerting web servers to update subscribed users when their currency values change (due to rewards or other events).
+ Handling tasks like cheat detection, server metrics reporting, Discord server bot management, Discord live reporting, and more.

I was aware that the worker was a bottleneck in the architecture, but up to now, it hadn’t been an issue. The worker usually sat at a very low CPU and memory level.

Regardless, I beefed the worker up to the largest size that [our hosting supported](https://fly.io/docs/about/pricing/).

![](./worker-sitting-okay.png)

It was sitting at about 50% utilization for most of the day, which was a bit high but still within the bounds of comfort. So I went to bed at about 9 p.m., feeling good about the launch so far... little did I know what was coming.

![](./bed-time.png)

# Wednesday

I woke up to an absolute nightmare. The servers had clearly been having a bad time while I was asleep. I hopped on [our Discord Server](https://discord.gg/fTdxf4y9) and saw that many players were complaining about the game not working correctly.

I took a look at the app event queue and, to my horror, saw that there were over 3 million events in the queue waiting to be processed.

![](./3m-queued-events.png)

The number was decreasing, but even after much tweaking of the various settings, I calculated it would take 26 hours for my poor single worker server to clear them all down, on top of everything else it was dealing with.

Before I could make any architectural changes to the servers, I needed to clear this queue to avoid losing data or causing more problems by handling events out of order.

![](./slack-to-brandon.png)

So although the game was in a semi-broken state, I went to bed thinking that at least the queue would be mostly cleared by morning, allowing me to focus on fixing everything properly.

# Thursday

![](./4m.png)

... well sh*t ... this isn’t good...

It turns out that while I was sleeping, we had some "good" news from our friends at Discord...

![](./moving-you-up-to-rank-4.png)

Ah, crap... I knew at this point I had to take action. This queue wasn’t going to clear itself.

I asked Discord to drop us back down the list. As painful as it was to request LESS traffic, I needed to buy us some breathing room.

The next step was to parallelize that event queue processing.

So I created a new kind of server called an "eventsWorker." This server took much of the event processing work from the worker, and critically, it could be parallelized.

![](./architecture2.png)

There was a risk of out-of-order event processing, but I carefully examined each event and realized there wasn’t much event-order dependence, particularly between users. Anything that was dependent, I could likely fix later.

This realization gave me the confidence to scale up the number of eventsWorker instances. However, this put a huge strain on the database, so I scaled it up to the largest size that Fly would allow.

![](./16-performance-core-db-pricing.png)

This allowed the eventsWorkers to scale to 15 instances, ripping through the queue at over 1 million events per hour. The entire queue should be cleared by late afternoon.

Based on that assumption, I got to work preparing the update that would go out once the queue was resolved.

I decided to reduce as many places as possible that depended on app events, thus reducing pressure on the background workers.

After thinking it over, I realized that most events could be processed "locally" instead of being sent to the background.

To explain what I mean, let's take an example event, `matchFinished`. (By the way, if you're curious, [here are all the app events](https://gist.github.com/mikecann/c7ad434ffcf94eb4e5c0d0b30f7967e1) in the game.) Here's where this event is used:

+ BattleEventsService - to send push notifications to users about the battle outcome
+ BattleWebhooksService - to send API messages to our Tournaments bot for advancing tournaments
+ BattleTabsLiveService - to update the #battletabs-live channel in Discord
+ HighScoresService - to update highscore stats
+ MedalAwardingService - to potentially award a user a new medal
+ UserChatDetectionService - to check for abuse if a user plays the same person multiple times
+ UserStatsService - to update a user’s stats
+ ChallengeUpdatingService - to progress any challenges the user might be on
+ TVChannelsService - to update any playing BattleTabsTV instances

Each of these is a totally different feature in the game and independent of each other, which is why they are handled as events rather than inline function calls in the first place.

When a battle was finished, I would publish a `matchFinished` event to the queue ([managed by pgboss](https://github.com/timgit/pg-boss)). The event would then be picked up by a background worker to handle all that work mentioned above.

I realized, though, that instead of sending it to the background, I could simply emit a "server event" in addition to the "app event". These server events would only be listenable by services running on that one server.

This shifted the computational burden from the background workers to the web workers, which could easily scale horizontally.

It seems obvious now, but hindsight is 20/20. I had been so focused on client-side improvements that I missed these obvious server-side optimizations. To be fair, it was only the increased traffic that necessitated the change.

So, shortly after 4 p.m., the events were all cleared. I scaled the eventsWorkers down from 15 to 3, then pushed my "server events" change.

This immediately reduced worker load but increased web server load as expected. Web servers could [auto-scale](https://fly.io/docs/reference/autoscaling/) up however so I wasn't worried.

I went to bed feeling pretty good about things. I did tell Brandon (who is based in the UK) to call me if the servers went down.

I soon fell asleep, fairly confident I would get a good night’s rest...

# Friday

3:14 a.m. – I got a call from Brandon letting me know that the game was down.

I jumped out of bed and hopped on my PC. Sure enough, things were not looking good and hadn’t been for a couple of hours.

![](./friday-morning-graph.png)

This time, it wasn’t immediately obvious what was wrong. It appeared that the worker process was running out of memory. I attempted some desperate fixes but wasn’t sure about the underlying reason.

![](./desperate-commit-log.png)

By 4:30 a.m. my brain really wasn’t working properly. I felt like the game was stable, so I decided to get an hour of sleep.

### Later on Friday

Awake and rested, I realized what the problem might be.

As previously mentioned, one of the worker server’s responsibilities was:

> Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.

For example, matchmaking works by requesting a list of open battle proposals every few seconds, then looping over that list and creating battles as needed.

Turn timeouts, AI turn-taking, and a bunch of other systems worked similarly.

The issue was that this polling was effectively another kind of queue. If the number of rows returned from the database during each poll was too small, that queue would build up.

Unfortunately, I had no metrics around these polling "queues," so my first task was to get some visibility into them.

![](./poll-queue-growth-chart.png)

Sure enough, those queues had grown to huge levels.

Much like my app events problem earlier, I knew the solution would be parallelization.

So, I carefully moved the polling code over to the eventsWorker instances. I was very nervous about this, as I knew that allowing multiple servers to poll the database simultaneously risked race conditions and other problems due to polling overlaps.

My solution was to carefully place [distributed locks](https://github.com/mike-marcacci/node-redlock) to ensure that two different workers couldn’t work on the same item from a polling queue simultaneously.

This solution worked well, and I soon had the polling queues back under control.

![](./polling-under-control-graph.png)

That’s when disaster struck...

![](./slack-to-brandon-wild-db-growth.png)

I had been keeping a close eye on our database’s growth, knowing that if we ran out of space, it would put a hard stop to the game.

Well, it turns out that with all the chaos of the past 12 hours, I forgot to check the DB size, and it had grown wildly. It finally ran out of space and started refusing connections.

Fortunately, I was looking at the logs when I saw this message go by:

![](./jobs-cannot-archive-log-message.png)

It turns out that pgboss, our app events queuing and jobs system, was timing out when trying to archive completed jobs. I ran a quick query against the database and saw over 7 million event rows waiting to be archived, with another 3 million waiting to be deleted.

It seems like pgboss wasn’t designed to handle this scenario very well and was erroring out during its archival and deletion routines.

My solution was to manually delete all 10 million rows from the database immediately. This instantly freed up 70 GB of space. I then ran a [Postgres VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html) to clear up any deleted tuples.

Instantly, the database came alive again, and disaster was averted.

Phew! It had been a long day, but I was finally feeling good about things. I went to bed certain that everything was all good.

# Saturday

I woke up, hopped on my PC, and to my dismay, saw that my "worker stopped" alert had triggered.

![](./slack-message-alert-worker-stopped.png)

Fortunately, it didn’t take long to track down that this time the worker hadn’t actually gone down. The alert had triggered because [Axiom](https://axiom.co/) (our log aggregation tool) had stopped receiving log messages. This happened because our "[log shipper](https://github.com/superfly/fly-log-shipper)" instance had died.

I still don’t know why or how this happened, but I quickly gave it a poke, and things were up and running again.

From the Grafana logs on the servers, it appeared that everything had been fine overnight, much to my relief.

I think I’m going to leave the day-by-day commentary here, but I believe you’ve got a good sense of the firefight I was putting up at this point.

# Lessons

So what lessons did I learn from all this?

1. **Be wary of queues**. Anytime you’re building something that could get backed up if you 10x or 100x your traffic, it’s wise to deal with it now or, at the very least, get some metrics around it so you can keep an eye on it.

2. **Single points of failure are ticking time bombs**. For the same reasons as above, be cautious of any places that cannot scale horizontally.

3. **Use Axiom**. It’s fast, cheap, and powerful. Log everything, then build graphs.

![](./axiom-pricing.png)

500GB on the free tier? That’s just insane.

To put it into perspective: despite all the spam we produced over the past week, we’ve amassed 54 million log lines, amounting to just 17GB of space, which is only 1GB compressed!

![](./axiom-dataset-logs.png)

4. **Load test if you can** before you expect a large increase in traffic. Even if you can’t easily hit the expected numbers on your local machine, it should at least hint at where bottlenecks might occur.

# Conclusions & Thanks

It’s still a bit early to talk about the results of all this traffic on the business, but things are going well.

I would like to thank the special people at Discord who took a chance on us and were very patient while we fought through these problems. :)



Phew its been a long long journey for our little game [BattleTabs](https://battletabs.com/) ([watch a summary video here](https://mikecann.blog/posts/battletabs-in-7-minutes)) but finally we are entering the phase that is a culmination of all the hard work, creativity, and perseverance: [the soft launch!](https://play.google.com/store/apps/details?id=io.battletabs)

# The Trailer

Before I start woffling on and loose you, heres some eye-candy:

<iframe width="853" height="480" src="https://www.youtube.com/embed/wGgJjEvNgt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Its almost not cringe! :D

# What Is a Soft Launch?

For those who might be wondering, a "soft launch" is a strategy used in the gaming industry where a new game is released to a limited audience or in specific regions before its official worldwide release.

In our case we are launching today [just on Android and just in the Philippines](https://play.google.com/store/apps/details?id=io.battletabs).

This phase allows us to gather valuable feedback, fine-tune gameplay, and ensure that everything is as perfect as it can be.

It’s a critical step in ensuring that BattleTabs delivers the best possible experience to its players.

# The Last 18 months

For the past year and half we have been beavering away at breakneck pace converting our web and chrome extension game over to work on mobile. Along the way we have added many new features and refined many of the existing.

Many parts of the game were changed in particular the UI has undergone a massive upgrade to make sure that the game works on small handheld devices in portrait mode.

That last part is worth underscoring. BattleTabs works in both Landscape (on the web or tablets) and Portrait (on mobile). This was no small task but I think we have pulled it off and im pretty proud of it.

[![](./layout_1.png)](./layout_1.png)

[![](./layout_2.png)](./layout_2.png)

Its also worth noting that, as often get asked this, the game is entirely Typescript using React for the UI and BabylonJS for the 3D rendering. I don't think there are too many games out there that have taken this path not sure if that is a good thing or not but it certainly sets us apart.

# What’s Next?

During the soft launch, we’ll be closely monitoring every aspect of the game, from user experience to the technical performance, particularly server performance.

We still have some important changes we want to make before final release such as a new First Time User Experience (tutorial) as that is super critical for players who are quick to dismiss something if they are confused or frustrated.

Post launch we have a number of exciting improvements and features planned but I might have to leave that for a future post :)

# How You Can Get Involved

Right now the game is only available on Android and in the Philippines but we are going to slowly be expanding the number of countries and platforms we are on.

We invite you to join us in this exciting phase. The best place to stay up to date is to [join the 16,000 other people on our Discord Server](https://discord.gg/battletabs). We post to the [#announcements](https://discord.com/channels/770955477215084554/770963532455477310) channel regularly about everything thats going on with the game.

Your feedback will be invaluable in shaping the final version of BattleTabs. Whether it’s a bug report, a suggestion, or even just sharing your favorite moments, every bit of communication helps us make BattleTabs better for everyone!

# Wrapping Up

The soft launch is just the beginning of the next chapter in the BattleTabs saga.

We’re incredibly proud of what we’ve achieved so far and even more excited about what the future holds.

Stay tuned, and [let the battles begin](https://play.google.com/store/apps/details?id=io.battletabs)!



As we close in on our year-long goal of bringing [BattleTabs](https://mikecann.blog/posts/battletabs-in-7-minutes) to mobile the time has come for some long-overdue performance improvements.

I have been putting it off as performance improvements are always a bit of a project-black-hole sucking all time, space and energy. Fortunately I was able to escape this relativistic singularly and come out of the other side with some sizeable performance gains and interesting insights in tuning a BabylonJS game that I thought I would share here.

# Before Getting Started

First thing's first. When attempting any sort of performance optimization pass its critical to lay a good foundation down first. We need to first get some good diagnostics info from the game. This allows us to see which parts of our game are causing the performance drops.

Fortunately BabylonJS has some pretty good metrics built into the engine that you can checkout in the inspector:

[![](./babylon-inspector-stats.png)](./babylon-inspector-stats.png)

It unfortunately doesnt show me those numbers as a percentage of frametime and it also takes up quite a bit of the screen. So I decided to make a little stats counter in the top left that shows me the critical numbers and their percentage of the total frame.

[![](./my-stats-counter.png)](./my-stats-counter.png)

With this in place I was able to start to narrow down what I think I can spend some time looking to a little more deeply.

One other quick tool I like to use is Chrome's CPU throttling feature:

[![](./chrome-cpu-throttle.png)](./chrome-cpu-throttle.png)

I develop the game on a powerful desktop PC which is likely more powerful than many of our users' devices. So using the CPU throttle it lets me simulate what its like for them a little more closely.

Arguably this slants things more towards a CPU bottleneck rather than a GPU bottleneck but lets just ignore that for now, ill discuss it more a bit later.

# Problem Areas Identified

I quickly noticed that there were three large contributors towards the total frame time: "Mesh Selection", "Animations" and "Render".

## Render

Render is a pretty obvious one, it's the time for the GPU to get the pixels on the screen. There was likely some stuff I could do here to speed things up but I decided not to tackle this one for now as it was likely going to involve some quality tradeoffs or digging through shader code which I would rather not do just yet.

## Mesh Selection

I wasn't too sure what this meant but after reading [Babylon's performance documentation](https://doc.babylonjs.com/features/featuresDeepDive/scene/optimize_your_scene#freezing-the-active-meshes) I had a better idea. As far as I understood it, it is the time taken for Babylon to choose which meshes it should send to the GPU for rendering. This is dependant mainly whether the mesh is within the camera view area (Frustrum).

BattleTabs doesn't really have many meshes and most of them are on screen most of the time so I wasnt really sure why exactly this number was so high.

I decided to [raise this issue on the BabylonJS forums](https://forum.babylonjs.com/t/why-is-mesh-selection-time-so-high-and-how-to-reduce-it/47069).

[![](./forum-post.png)](./forum-post.png)

After a bit more experimentation I added a "FREEZE" button to my stats which called the `scene.freezeActiveMeshes` which was also mentioned in the Babylon performance docs. This had a dramatic effect on the Mesh Selection time, effectively removing it.

<iframe width="100%" height="480" src="https://www.youtube.com/embed/HtVbtbm8I5w" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

As far as I understand it, freezing active meshes means that whatever is currently "active" (viewable within the Camera Frustrum) will ALWAYS be incuded in the list of meshes to be rendered.

This is great as it removes all that frustum culling time but it has a nasty side effect. If anything is added or removed from the scene, or its visibility changes, then it wont be reflected on the screen. This is because the mesh selection is "frozen". Animations also froze (im not exactly sure why this is).

All of these issues were possible to work around but it was going to be a bit of a pain going through the code with a fine-toothed-comb looking for places that might cause problems. Not only that it was likely to cause future bugs as its a nasty cross-cutting concern you can often forget.

I decided to sit on this one for now and look into the final large chunk of frame time I had yet to explore.

## Animations

Animations were taking over a third of frame time, this was a surprise to me. We only have 4-8 ships animating at any one time and [I was under the impression that babylon's skeleton animations were computed on the GPU](https://doc.babylonjs.com/features/featuresDeepDive/mesh/bonesSkeletons#picking-a-mesh-attached-to-a-skeleton) so there was no reason why they should be so slow.

I had read others having issues with this in Babylon and they suggested [that one could bake their animations](https://doc.babylonjs.com/features/featuresDeepDive/animation/baked_texture_animations) into whats know as "Vertex Animation Textures (VAT)".

These are basically pre-computed positions for each (bone? vertex?) each frame. These positions are stored in a giant array then loaded a runtime, passed to the GPU then used in a shader to update the positions of each bone / vertex each frame.

Doing this was going to be a bit of a pain as it meant we would have to add an additional design-time step of producing these VATs for each ship (and skin) and keeping them updated any time a change is made. It also precluded us from using blended animations, but we werent using those so I wasnt too bothered about that.

As with all performance work I did the bare minium of absolutely the nastiest-hackyiest code I could to test the theory and was surprised that VATs actually worked:

<iframe width="100%" height="480" src="https://www.youtube.com/embed/gIfKTqxTS4o" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

VATs meant slower game loads as each animation for each ship now had to load in another texture. It likely also meant more VRAM usage as these new textures had to be sent to the GPU but I suspected this tradeoff was going to be an acceptable one for BattleTabs so I ploughed ahead.

# Putting it all together

With the theories in place and experiments conducted I reset my git branch and started again, this time properly coding it up. This was an arduous process as it meant handling lots of edge-cases that the experiments didn't cater.

Eventually I thought I had it all looking pretty good. To make sure I had the problem solved I put together a torture test scene:

<iframe width="100%" height="480" src="https://www.youtube.com/embed/xJ8DGv_NzAE" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

The above shows that even with 100 ships animating I wasnt spending any time in Mesh Selection or Animations, huzzah!

# Conclusion

So in total I saw about a 70-100% performance improvement once I set my CPU throttle to 6x slowdown. This is quite a nice gain and makes me question why BabylonJS as an engine doesnt put more effort into reducing CPU load as JS is inherantly single-threaded and GPUs are so powerful these days it seems to me that the CPU is almost always going to be the bottleneck.

When I quizzed Evengi_Popov one of Babylons core-contributors he agreed:

[![](./jedi.png)](./jedi.png)

So who knows, perhaps we will see some of this stuff built into the engine in the future.

For now im happy with the results, theres always more that could be done but I think its taken a little bit of pressure off of many of our users' poor devices and given them a better gaming experience.

Now onto the last few things before the mobile release!



As you may be aware we have been working really hard lately on the next version of our game [BattleTabs](https://mikecann.blog/tags/battletabs).

In an effort to broaden the game's reach for the upcoming mobile release, we decided to invest some time in producing "devlogs".

If you're not familiar with devlogs, they are a popular format on YouTube where developers discuss their past and current progress with a game.

We hope that viewers will find the video entertaining and informative about the game. Ideally, they will subscribe and follow along as the game progresses and future devlogs are released.

With that in mind, we created our first devlog. This one is longer than future ones because I was eager to tell the story up to this point. Watch it and let us know what you think!

<iframe width="853" height="480" src="https://www.youtube.com/embed/kDlEoLjflGw" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

We were initially hesitant about investing the time (about 5 days) to prepare and create this video, as that's time I could have spent working on the game.

However, we justified the time investment because marketing is an area where none of us on the team are experts. We feel the need to explore every possible avenue to spread the word about the game and attract new users.

Speaking of which, we are taking [BattleTabs to the Perth Pixel Expo](https://www.pixelexpo.org.au/game-devs) this year, where we have a double-wide booth.

[![](./expo.png)](./expo.png)

I'm looking forward to showcasing the game in person to a larger audience, running tournaments, and giving out stickers and t-shirts!

We plan to continue our devlog videos roughly monthly, depending on other commitments and the content we have to show off.

Anyway, let us know what you think in the comments. If there are things we can improve for future videos and, importantly, other communities you think might be interested in this video, we'd love to hear from you!



This month we completed probably the single largest change in the history of our game [BattleTabs](https://battletabs.io), we converted our 2D game to run completely in 3D. This was obviously a massive undertaking for a 3 year old game but I personally think the results are fantastic and unlock a whole bunch more capabilities for us.

# History

Ever since the very earliest version of [BattleTabs](https://battletabs.io) the game has been rendered using a 2D canvas (PhaserJS & PixiJS).

[![](./old-game1.jpg)](./old-game1.jpg)

This has been a surprise to some as they assumed the game was 3D but in actuality the art was authored in 3D then simply rendered out to 2D sprite sheets.

[![](./spritesheet.png)](./spritesheet.png)

This looked great and worked just fine at first but then we started to add more and more ships and started to run into issue with our process. I have previously [discussed this in detail here](https://mikecann.blog/posts/introducing-battletabs) but the short version is "isometric is hard".

<iframe width="853" height="480" src="https://www.youtube.com/embed/ivIm2bZNQ3Q" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Not only did the 2D pipeline cause us issues in art output it also limited us design wise.

For example, we would liked to have done 4-way rotations, but that would have meant we would have to render out the ships facing the opposite direction then sliced them up, increasing the art load significantly. We also would have loved to have more than one animation per ship.

# Mobile Woes

There was a bigger looming issue on the horizon however; our port to Mobile. With mobile we knew that the smaller screen size was going to be a big problem for the game as there simply wasnt going to be enough space to fit both boards on screen in portrait or landscape.

[![](./mobile1.png)](./mobile1.png)

We could have done some clever scaling and blurring to give the illusion of depth:

[![](./battle-screen.png)](./battle-screen.png)

This however would limit our future plans for improving the scenery aesthetics and features:

[![](./scene-mock.jpg)](./scene-mock.jpg)

What we needed really was a way to rotate and move the camera in 3D space so we could shift the players perspective around the world.

[![](./perspective.png)](./perspective.png)

# Going 3D

So for these reasons (and others) we decided to bite the bullet and rewrite the game in 3D.

We did some preliminary investigations into various game engines:

[![](./editor-research.png)](./editor-research.png)

We really liked the look of Godot and Unity. We have lots of experience with Unity as as team but worried that its Web output file size would have been too large for users to download.

In the end the thing that decided it for us was the mountains of existing shared code between the server and client and we really didn't want to have to rewrite that in another language (such as c#) and then potentially have to maintain two codebases.

We decided to stick with our existing codebase and replace the [PixiJS](https://pixijs.com/) 2D engine and replace it with a [BabylonJS](https://www.babylonjs.com/) 3D engine. We went for Babylon because of the great documentation, feature set and rave reviews from other developers.

Because we were using a custom ECS implementation for the 2D version in theory I should have been able to simply swap out the 2d rendering components and systems and replace them with 3D versions. In reality there was too many other things that required changes that I decided to start from scratch with a brand new ECS architecture.

After a lengthy evaluation process I ended up going with hman's excellent [miniplex](https://github.com/hmans/miniplex). Miniplex is so cool I think I will devote an entire post to it in the future but ill just say if you need an ECS engine in JS and dont need vast numbers of entities then I strongly recommend miniplex.

<iframe width="853" height="480" src="https://www.youtube.com/embed/hbAWQYpoL-c" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

As this post is getting quite long already ill skip a lot of the middle ground and just say that our artist, Sophie, did an incredible job converting the existing ships and assets over to work in 3D. She slogged through all 30 ships 3 times to get the number of draw-calls down and performance under control but I think the end result is fantastic.

For this first release we decided to keep the game looking as close as we can to the 2D version. This reduced the number of design-related changes so we could simply focus on the bare essentials of the port. This also reduced the list of changes for our existing user base.

# Conclusion

Since we released the 3D update earlier in the month, we have have done a number of follow-up updates including the first feature we were unable to do in 2D; full 4-direction ship rotation!

<iframe width="853" height="480" src="https://www.youtube.com/embed/BtXFBOO-Gk8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

I feel like we now have a really firm footing which to build on for our mobile port and upcoming set of exciting features.

Stay tuned as I think 2023 is going Gangbusters!



Its been a while since I have posted about BattleTabs but things most definitely haven't been standing still, so I think its time for an update!

# Background

Just quickly; [BattleTabs](https://battletabs.com/) is the premier game from [Gangbusters](https://www.gangbusters.io/), the company that im a co-founder of.

I have [posted quite a few times in the past](https://mikecann.blog/tags/battletabs) about BattleTabs so checkout those posts for more background on the game and how it works.

# Whats Changed

[![](./update-notes.jpg)](./update-notes.jpg)

Since November last year there have been over 40 updates to the game. Some of the major ones include:

- Rivalry Feature - challenge others to "just one more battle"
- Huge change to the battles architecture allowing for more complex ship-ship interactions
- New content - ships, emotes, etc
- Lots of changes to support Manifest v3 Chrome Extensions
- Added new sound and visual effects
- Added push notifications to the web versions of the game
- Added "Daily Deals" feature to the shop
- AI turn taking for players that go offline
- Board Ping feature
- TONs of bug fixes

In addition to the above we also started working on our long-term strategy for the company.

# Cards & Web3

We strongly believe in the power of community driven games. We think that a game can be good but the community is what makes it great. This is certainly the case for BattleTabs.

Over the almost 3 years of running the game we have amassed over a million downloads, millions of battles played and over 15,000 members on discord. 24 hours a day players on discord discuss the game, create content, share ideas or just generally hang out.

Given this, we started to think how we could empower our community to go further and at the same time develop a strategy that allows us to continue to develop the game and company.

This is where BattleTabs Cards comes in.

From our [community page](https://battletabs.com/battletabs-cards) on the topic:

> BattleTabs Cards are digital collectible cards that you can trade with other players. They will be released in Sets, where each set will feature a particular theme or art style. Every Set will be limited in quantity, and cards will have different rarities.

<iframe width="853" height="480" src="https://www.youtube.com/embed/LFV1ugBAb1M?playlist=LFV1ugBAb1M&loop=1;rel=0&autoplay=1&controls=0&showinfo=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

> Outside of their intrinsic value in looking amazing cards will also provide unique benefits in BattleTabs and other experiences we have in production. For example, you may craft a special cosmetic item or a ship skin in BattleTabs using a combination of cards, then display or trade the crafted treasure inside your own 3D sandbox! Watch as the newly crafted item comes alive with special animations and behaviours.

[![](./storyboard_c.png)](./storyboard_c.png)

> Additionally, BattleTabs Cards can be freely built-upon without any intervention or restrictions from us. Say you have a fantastic idea for a new game, you can team up with your friends, use the exact same "shared card database" that we use!

[![](./storyboard_a.png)](./storyboard_a.png)

> You could host tournaments with these cards, perhaps charging players a Common card to enter the tournament and setting Rare cards as a reward for the winners.

> Say you have a vast stack of cards saved up and want to give some to helpful members in the community, you can do that with cards!

We think it an exciting direction for the company and community. To kick things off we are running a tournament on [Fractal.is](https://fractal.is).

# Fractal Tournament

Our first card is going to be special. Rather than simply selling or giving them away we thought it would be more fun if we rewarded top players and community members by giving them preferential access.

So every day that the tournament runs (2 weeks), the top 20 players that participate will earn a silver token:

[![](./silver-token.png)](./silver-token.png)

The silver token along with the golden token which we awarded to a selection of of most loyal players will grant free minting rights to the Seed Card.

[![](./gold-token.png)](./gold-token.png)

The tournament starts tomorrow, monday the 15th of August. Head over to [https://www.fractal.is/battletabs/tournaments](https://www.fractal.is/battletabs/tournaments) for all the details!

# Fractal Interview

In preparation we sat down with Justin Kan who co-founded JustinTV (Twitch) and a number of other companies including [Fractal.is](https://Fractal.is) where the tournament is being held.

<iframe width="853" height="480" src="https://www.youtube.com/embed/nRbUc9N9hGY" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

We talk about the game, the tournament, web3 in general and some future projects. Check it out!

# Conclusion

The past 9 months have been a total rollercoaster of ups and downs (no one ever said running a startup would be easy huh?). The ever-changing funding landscape in particular has been a source of continual headaches.

It does however feel great to wake up each morning and work on something that brings obvious joy to thousands of people daily.

I hope we can continue to grow and improve our game, business and community for many years to come!

BTW: I generated the header image using [Dalle](https://openai.com/dall-e-2/).. so cool :)


Well its been years in the planning and months in the development but the "Big Summer Update" / "Seasons" / V1 update for our game [BattleTabs](https://battletabs.com) is finally out!

This is by far our biggest update yet. We generally don't like making large changes to the game like this, preferring small frequent incremental changes but it was necessary however as there are lots of exciting new overlapping systems and features in this update!



# Progression

For ages now there has been a large missing feature in the game one so obvious it stands as a testament to the [enjoyability and popularity](https://twitter.com/theBrandonWu/status/1458058505072652300) of the game that we were able to get this far without it. That missing feature was progression.

Sure we have had a rudimentary sort of progression via preset fleet unlocks but this was rather crude and players tend to whip through that part of the game:

[![](./preset-fleet-unlocks.png)](./preset-fleet-unlocks.png)

The new update however brings a whole new dimension to progression in the game. Players now earn XP for various actions during a battle which contributes to their season level.

[![](./game-over-modal.png)](./game-over-modal.png)

# Season

Seasons are a feature common to Free-2-Play games and exist as a mechanism to keep the game feeling fresh as the developer continually works on new content. The season is periodically (every few months) refreshed and the player's level resets. This gives new players to the game a way to enter the game without feeling like they will never catch up with the players that joined before them.

The XP earned contributes to their season level:

[![](./progress-scroller-1.png)](./progress-scroller-1.png)

Each level the player can claim various rewards:

[![](./reward1.png)](./reward1.png)

There a lots of different kinds of rewards: Avatar Cosmetics, Custom Emotes, Ship Blueprints, Gold, Gems etc

There are 60 levels and two "tiers" of rewards. Theres the free tier, that anyone can claim rewards from, then there is the premium "Battle Pass" tier:

[![](./bp-modal.png)](./bp-modal.png)

The Battle Pass costs 2000 gold to unlock (1000 with Battle Club, our subscription product) and gives the player 2x XP rewards and substantially better rewards. Battle Pass also resets at the end of each season.

The player can grind enough gold during the free tier to unlock the Battle Pass for free eventually but they will loose all the double XP bonus they could have achieved along the way if they do that, so we are hoping this will be a strong incentive to visit our new Shop.

# Shop

[![](./shop1.png)](./shop1.png)

Another common Free-2-Play mechanic that this update brings is In App Purchases (IAP) via the Shop. The player can spend fiat on in game currency in the form of Gold and Gems.

Gold can be used to unlock the Battle Pass but Gold and Gems can also be used to rent and buy ships.

# Ship Blueprints

Ships are the core game play feature for the game and can be thought of as cards in a traditional Collectable Card Game (CCG). You collect Ships (Cards) and combine them together in Fleets (Decks) and fight other players.

This update brings the concept of "Ship Blueprints" to game.

[![](./blueprint-states.jpg)](./blueprint-states.jpg)

The player can "discover" blueprints during the season then rent the ship for a few days for some gold. Once rented the ship can be used in a fleet. If it expires it can no longer be used and will have to be rented again. If the player get tired of renting a ship they can alternatively buy it outright with some gems. Once bought / owned they will never have to rent it again.

This Ship Rental concept gives the player something to spend their gold on and forces them into some interesting cost / benefit choices as they play.

# Content

The update is jam-packed with new content including:

- 6 New Ships
- 29 New Avatar Cosmetics
- 24 New Emotes
- 5 New Medals

The key is; once the season is over this content will no longer be unlockable. This incentives players to unlock everything before the season ends (FOMO). We may bring back some of the rewards in the future in the shop or through other rewarding mechanisms.

# Gangbusters

[![](./logo_full.png)](./logo_full.png)

This update brings a name-change for the company.

[3 years ago](https://mikecann.blog/posts/farewell-bamboo-hello-markd) Brandon and I started working on a SaaS product called "Markd.co" and so we named the company "Markd Ltd".

Some time later we decided to pivot the company and focus on BattleTabs. For for some time now we have felt that the name no longer suits what we are doing and have been searching for a new name.

We traded different ideas until we hit on "Gangbusters". We like it as it hints at excitement, growth and energy, all things we want to instil in the company.

"Gangbusters" as a word also has some utility as it can be split into "Gang" and "Busters" so employees or fans of the company could be called "Gang Members" or perhaps "Busters". Its also a word in common parlance that hasnt been taken, which helps :P

# Conclusion

I hope this post has given you a little taste of this update. We have worked really hard on it and I am personally quite proud of the results. We are due to be featured on the GooglePlay ChromeOS store soon and the hope is this will increase the number of users and excitement around the game significantly.

If you are in interested in checking out out head over to [https://battletabs.io/](https://battletabs.io/) and get started. Ill see you there!


It's been 6 months since [I last wrote about BattleTabs](/posts/battletabs-ships-and-fleets-update) and so much has happened and so many things have changed I felt it was time to do an update.

# 6 Months Ago..

.. We had [just released](/posts/battletabs-ships-and-fleets-update) the ships and fleets update which was by far our biggest update to that point. For the first time we introduced the concept of ships that have abilities and are grouped into fleets. This fundamentally changed the game from what was essentially just Battleships into its own unique experience.

[![](./select-a-fleet.png)](./select-a-fleet.png)

In the same update we added a proper tutorial and gave a UI a much needed overhaul.

[![](./new-sidebar.png)](./new-sidebar.png)

# Whats New

Since then we released a whole heap of new features and improvements over a total of 35 updates!

Some of the biggest new features include:

- New version for web (https://battletabs.io)
- Battle Club, our paid subscription feature giving the player access to exclusive ships fleets, avatars backgrounds and lots more.
- BattleTabs Live, a discord bot that announces major events in the game.
- A user profile modal for viewing states and auth data
- A new authentication system so you can signup and login with socials
- A new "Update Available" / auto update feature
- Private proposals so you can send invites to each other via links
- A bunch of new fleets and ships

[![](./fleets-list.jpg)](./fleets-list.jpg)

# Some Numbers

Players seem to enjoy the regular updates and improvements. The number of matches created per week has steadily increased to the point where almost 70,000 matches are created per week:

[![](./matches-created.png)](./matches-created.png)

Thats over 300,000 turns taken per day!

[![](./turns-taken.png)](./turns-taken.png)

# Discord

We also launched a [Discord Server](https://discord.com/invite/kvJKXsu) in the last 6 months which has seen surprising growth to over 8,000 members.

[![](./discord-memberships.png)](./discord-memberships.png)

At any one time we have hundreds of members online chatting away in the various channels. Its awesome to see such a fantastic community grow around our little game.

Brandon, my co-founder, has been busy running community tournaments which have proved incredibly popular garnering far more sign-ups than there is space for.

[![](./tourny.png)](./tourny.png)

# Under The Hood

<div style='position:relative; padding-bottom:calc(50.42% + 44px)'><iframe src='https://gfycat.com/ifr/redevenbedlingtonterrier' frameborder='0' scrolling='no' width='100%' height='100%' style='position:absolute;top:0;left:0;' allowfullscreen></iframe></div>

As you can imagine with all this growth comes some technical challenges.

## Scaling

In the past 6 months we have had our work cut our for us scaling the backend. Its a challenge as the game uses stateful sockets for communication which complicates the usual "stateless" servers mantra.

Our solution has been to use a combination of careful horizontal scaling of client facing nodes which communicate via a shared custom redis pub-sub mechanism.

Since launch we have run most of our infrastructure on Heroku which has been great for for rapidly getting us to were we are today but its scaling is a little lacking. For now it works but theres plenty room for improvement in the future.

## Client State

Recently we also changed the way state is managed in the client over from [MobX State Tree](https://github.com/mobxjs/mobx-state-tree) (MST) to just [MobX](https://mobx.js.org/README.html). Although [I am a fan of MST](https://mikecann.blog/posts/mst-libs) we have found that it unfortunately doesn't play very well with Typescript once your store reaches a certain size.

MST relies on a lot of type inference and union types and if you have been around Typescript for a while you will know that this specific combination of characteristics can quickly lead to a combinatorial explosion of types causing slow compile times and a poor IDE experience.

With MobX things are simpler as state is split between multiple stores defined as classes. This means Typescript doesn't have to work so hard calculating what the type of something should be at any given time.

We do give up some nice MST features such as time travel debugging, undo-redo and state managed purely in the background page but we either weren't using those features or we would have had to stop using them for [manifest v3](https://developer.chrome.com/docs/extensions/mv3/intro/) regardless.

## Monorepo

[![](./monorepo.png)](./monorepo.png)

We changed the way we code to move from a multi-repo approach to a single monorepo approach. This was done to simplify dependency management between the various parts of the system and simplify the versioning story.

This definitely makes things much simpler to understand when you have more than one person working on the codebase at once.

# Future

Theres so much more we have planned both from a feature standpoint and technically. I cant share too much right now but its super exciting so stay tuned!

If you want to play the game you can [download the extension](https://chrome.google.com/webstore/detail/battletabs/mjcklhnhfiepmofggcoegkmkokbljmjd) or play right now on http://battletabs.io/


We were suffering from slow integration tests on our server when I discovered something that sped up performance by 15 times!



# TLDR;

Set `fsync=off` in your postgres config.

# The Problem

For our game [BattleTabs](http://battletabs.com/) run a custom sockets based API server. To ensure that the server is working as intended I like to write integration tests that hit the database. This gives a good level of confidence that calling that endpoint from the game client is going to work as expected.

The issue is that the tests were taking quite a while to run:

[![](./before.png)](./before.png)

I narrowed the issue down to where most of the time was being spent:

[![](./trunc-before.png)](./trunc-before.png)

We must truncate the database before each test runs so that there is no state from the previous test that can affect the results. The problem is that this truncate seems to be taking an inordinate amount of time.

# The Solution

After some googling I came across [this post](https://dev.to/thejessleigh/speed-up-your-postgresql-unit-tests-with-one-weird-trick-364p#:~:text=fsync,are%20physically%20written%20to%20disk.) which seemed to suggest that there was a magical config value `fsync` that I could disable which would speed things up.

So I set `fsync=false` in my postgres config and restarted my docker container and..

[![](./trunc-after.png)](./trunc-after.png)

Wow! Thats a 15x speedup! Incredible.

So whats the downside? Well `fsync` ensures that changes are written (synced) to disk when Postgres makes a transaction. So by disabling it you are disabling protection from crashes power-cuts and whatnot so its [definitely not recommended in production](https://www.2ndquadrant.com/en/blog/postgresql-fsync-off-warning-in-config-file/) but when running tests we dont really care about data integrity so its safe to disable it.

# Github Actions

I wondered if it was possible to apply this sort of speedup to the CI which runs all the integration tests on each push.

To cut a long story short it appears you [cant supply config params to the standard postgres image while running in Github Actions](https://stackoverflow.com/questions/65438055/how-to-add-config-args-to-postgres-service-container-in-github-action/65444387?noredirect=1#comment115705280_65444387) but fortunately we can use another postgres image from Bitnami which does let us disable fsync.

```yaml
jobs:
  test-and-build-pr:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: "bitnami/postgresql:latest"
        env:
          POSTGRESQL_DATABASE: postgres
          POSTGRESQL_USERNAME: postgres
          POSTGRESQL_PASSWORD: postgres
          POSTGRESQL_FSYNC: "off"
        options: >-
          --health-cmd "pg_isready -d postgres -U postgres -p 5432"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
```

The result is a speedup of 3x, not quite as dramatic as the local speedup (probably because im running postgres in a linux docker container on windows locally and Github is running it on linux) but an impressive performance boost nonetheless!

We were having serious memory issues on the Heroku hosted NodeJS backend for our game [BattleTabs](https://battletabs.com) when we decided to apply "this one little trick"...



# TLDR;

Use [Jemalloc](https://github.com/jemalloc/jemalloc).

# The Problem

For those that aren't regular readers of this blog, for the past year my main focus has been a chrome extension game called [BattleTabs](https://battletabs.com). The game is based around the classic board-game BattleShips except you can play it in a Chrome Extension. This makes it rather unique as the game is immediately available every time you open a new tab in your browser. You can take a turn and move on with what you were doing or you can sit and play a longer session.

Because of the way the game works we keep a persistent socket open to our backend at all times. This allows us to keep the user's state in sync by pushing down updates from the server.

The game has grown considerably over the past 12 months and as a result the demands on the server have continued to grown. That coupled with the increased feature set has seen more and more open sockets and more required memory per socket.

# Vertical to Horizontal

We follow a pragmatic development strategy for the game, meaning that we only spend the time working on improving the infrastructure if its necessary. This meant that for 9 months we weren't really worried about server scaling or performance. We knew the issue was there in the background but whenever we needed more capacity we just bumped up the server size in Heroku:

[![](./heroku-dyno-sizes.png)](./heroku-dyno-sizes.png)

Well unfortunately the day game when we were up to the largest size dyno and we were starting to see some issues again, it was time to do something about it.

The issue wasn't really that of memory at this stage, instead it was the number of simultaneous open sockets to a dyno instance. As far as I am aware there is no docs on what the max number of sockets is per dyno but [this](https://stackoverflow.com/questions/25118381/websocket-concurrent-connections-limit-on-heroku) stack overflow answer seems to suggest 6000.

We were definitely right up against that but at the same time because we are on node we are single threaded so we were also up against a CPU bottleneck trying to handle all the requests when the dynos go through their daily restart.

The time had come to go from a horizontal scale strategy to a vertical one.

We had been putting this off because horizontally scaling a websocket server isn't trivial. Fortunately we didn't have that much local-state and we were using [Socket.io](https://socket.io/) which has some good plugins to help with scaling. I wont detail everything we did here, feel free to contact me if you want details.

[![](./socketio.jpg)](./socketio.jpg)

# Memory Issues

Now we could horizontally scale our server out to as many dynos as we wanted we should in theory be able to drop our dyno size back down to the smallest size (512mb RAM) and then just add more dynos as we grew.

The problem is, we seemed to still be using far more memory than we should have been given we were only running less than 1000 sockets per dyno. Worse still the memory seemed to be increasing over time indicating that there might be a memory leak somewhere.

After days of experiments and searching I decided to look a little closer at what socket-io was doing, thats when I came across this Github issue: https://github.com/socketio/socket.io/issues/3477

[![](./mem-leak.png)](./mem-leak.png)

Others on the issue had done the hard work for me and instead of a memory leak they suggest memory fragmentation was the root cause of the problem.

This was a big "Ahhh!" moment for me as memory fragmentation would look a lot like a memory leak as your available memory over time would decrease as the memory became more and more fragmented.

There were two suggested solutions on the thread. I tried the first which was to disable "perMessageDeflate" on SocketIO. This had some effect but still we seemed to be using far more memory than we should have been.

So I decided to try the second solution too which was something I would never have considered; replace the system memory allocator in Node.

# Jemalloc

The other users on the Github Issue suggested trying [Jemalloc](https://github.com/jemalloc/jemalloc). I personally had never heard of it before but apparently its well known and popular with 5,500+ stars on Github. It describes itself as:

> jemalloc is a general purpose malloc(3) implementation that emphasizes fragmentation avoidance and scalable concurrency support.

Sounds like its exactly what we need, but how do I get node to use that? As with most things Heroku, someone has thankfully trodden this path before me and written a "Buildpack" which I can just plug into my app: https://elements.heroku.com/buildpacks/gaffneyc/heroku-buildpack-jemalloc

And the effect?

[![](./half-mem.png)](./half-mem.png)

Wow! It halved our memory usage! Incredible!

# Conclusion

We were able to scale our heroku server out horizontally and overcome a memory fragmentation issue in socket-io by replacing the memory allocator with Jemalloc.

We have been running like this for a couple of weeks now and have yet to see any issues.

I haven't tested this on non-SocketIO servers but I suspect so I cant say for sure but I suspect it would have a similar magical effect. I would be very interested to hear your results if you tried this!

After months of hard work we are excited to announce that our latest and largest ever update to [BattleTabs](https://www.battletabs.com/) is finally live! Read on to find out about some of the gory details.



# TLDR;

The video below gives you a taste of whats changed.

<iframe width="100%" height="400" src="https://www.youtube.com/embed/JNCD3PhP08w" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

# Ships and Fleets

The most important change in this update is the addition of Ships and Fleets to the game.

Previously the game was pretty much just [standard battleships](<https://en.wikipedia.org/wiki/Battleship_(game)>) except perhaps with the addition of the distance indicators on the cells if you miss:

<iframe src='https://gfycat.com/ifr/DesertedHonoredFluke?hd=1' frameborder='0' scrolling='no' width='100%' height='450px'  allowfullscreen></iframe>

<iframe src='https://gfycat.com/ifr/BlackandwhiteLoathsomeDairycow?hd=1' frameborder='0' scrolling='no' height='450px' width='100%'   allowfullscreen></iframe>

This was enjoyable enough but we always felt it lacked a little depth so after a lot of brainstorming and hard work from our designers we came up with the concept of fleets.

[![](./select-a-fleet.png)](./select-a-fleet.png)

A fleet contains 4 ships which each have a [special ability](https://www.battletabs.com/how-to-play-battletabs-v2/) that they can use.

Some ships live in multiple fleets but they are carefully tuned so that each fleet has an play style.

For example the "Reveal & Snipe" fleet is high risk vs high reward fleet that revolves around discovering the location of the enemy ships with "Reveal" abilities then taking them out in one hit with the Coracle.

[![](./reveal-and-snipe.png)](./reveal-and-snipe.png)

This adds some much needed diversity to the battles as you learn how each fleet works and what its strengths and weaknesses are against other fleets.

[![](./example-battle.png)](./example-battle.png)

One other strong part of this design is that it is an obvious point of extension where we can add more fleets and ships as time goes on, keeping the game interesting to loyal players.

# Tutorial

Because the Ships and Fleets is now quite a bit more complex we also decided to take the time and build a proper tutorial into the game.

[![](./tutorial.png)](./tutorial.png)

Previously we simply popped up a modal with a couple of short gif animations that gave you the general gist of how to play but we always knew that was a temporary measure.

The tutorial now guides you step by step through how to play the game.

<div style='position:relative; padding-bottom:calc(50.42% + 44px)'><iframe src='https://gfycat.com/ifr/ValuableIncomparableAfricangoldencat' frameborder='0' scrolling='no' width='100%' height='100%' style='position:absolute;top:0;left:0;' allowfullscreen></iframe></div>

# Sidebar

We know from our analytics that a large percentage of our users are playing on chromebooks or small windows laptops both of which have very limited vertical height. This meant that the play area they had was small and made smaller thanks to a bunch of vertical space eaten by our UI at the bottom of the screen:

[![](./old-sidebar.png)](./old-sidebar.png)

So we decided as a part of this update to overhaul the UI and move all of this to a bar on the left hand side of the screen instead:

[![](./new-sidebar.png)](./new-sidebar.png)

This not only looks a whole lot better but it frees up vertical space so that the thing thats most important (the game) can take up as much space as possible and thus isnt scaled down.

# Other Stuff

We added a whole bunch of other stuff as part of this update that I wont detail but some of which includes:

- New avatars for players
- Larger maps
- New emote UI
- New Logo
- New info modal on pinning the extension
- Screen shake effects
- Better in-game UI
- Some nice CSS animations
- Faster load times
- Better ship placement and rotation
- Countdown sound effect and better UI

# Conclusion

We are really proud of this update as we think it brings some much needed depth and polish to the game. Hopefully it also sets us up nicely for a bunch of future features we have planned so stay tuned!

If you want to checkout the game head over to: [https://www.battletabs.com/](https://www.battletabs.com/)

This one has been quite a while in the making but im proud to announce this innovative new game we have been working on.



# TLDR;

<iframe width="853" height="480" src="https://www.youtube.com/embed/AEC2tPX6HdM" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# History

I have been building chrome extensions for [quite some time](/posts/my-first-chrome-extension-post-to-tumblr/) and have always considered them a powerful and often overlooked medium.

When I came across the [Tabby Cat](https://chrome.google.com/webstore/detail/tabby-cat/mefhakmgclhhfbdadeojlkbllmecialg) extension a few years back I realized that the New Tab page is a powerful space for content that gets short but frequent attention (every new tab thats opened). It got my brain-juices flowing and I began to wonder what else we could use the new-tab page for.

Obviously as a game developer my mind quickly turned to what games could be played there. It needed to be asynchronous and quick to take your turn so you can continue on with what you were doing. I thought about classic board games and the first one I thought about was Battleships.

I half-started to implement what would become "BattleTabs" some years ago but never completed it.

When I mentioned the idea to my partner-in-crime Brandon he agreed that its an interesting idea and so we started to explore the idea a little more.

We decided to enlist the skills of my ex-colleague and artist extraordinaire [Alan Macfarlane](http://www.melbournemonsters.com/alan-macfarlane). A month or so of development later and BattleTabs was born!

# How It Works

So what is BattleTabs? Well its "An Asyncronous New Tab Chrome Extension Game", a descriptive but mouthful of a name. I like to think of it as a "Nano Game". That is you can dip into the game for a second (or less) at a time.

You start the game by going to the Chrome Webstore and installing an extension: https://chrome.google.com/webstore/detail/battle-tabs/mjcklhnhfiepmofggcoegkmkokbljmjd

Once installed the extension will replace your current new-tab page. The first time you open it you will immediately be asked to start placing your ships for your first match:

[![](./first-match.png)](./first-match.png)

You can drag and rotate the ships to whatever you think are good positions or you can randomize the positions for quick action.

Once ready you give yourself a name and avatar (clicking the avatar changes it):

[![](./create-user.png)](./create-user.png)

And then you are straight into the action:

[![](./waiting-for-opponent.png)](./waiting-for-opponent.png)

If theres another player that is waiting for an opponent too you will instantly be matched up against them.

You can then start taking turns trying to sink each others ships.

[![](./taking-turns.png)](./taking-turns.png)

Because the game is designed for quick bursts of gameplay you can close the window at any time and the game will continue to periodically check in the background for turns from your opponent. It will let you know via a notification if they took a turn.

You can start a new match at any time, but you can only have 5 active matches at any one time.

[![](./new-match.png)](./new-match.png)

You have one day to take your turn otherwise you forfeit your turn. If you forfeit your turn three times in a row you automatically loose the match.

Scoring is pretty simple for now with a point ranking based on the number of games you have won or lost, but we plan on improving that later (see Future Plans).

[![](./simple-scoring.png)](./simple-scoring.png)

# Development

Development was quite an involved process as it often is with games. We first had to decide how we wanted to present the game, should it be 3D, topdown, isometric or something more exotic?

[![](./playing areas.png)](./playing areas.png)

We ended up with a simple isometric layout. We felt like it gave us the right combination of aesthetic and simplicity on art requirements as you can simply horizontal flip the sprite to make it look like its going in a different direction.

Once we had the general layout down we moved on to how do we present the ships. The talented Alan knocked out a bunch of different options for us to choose from:

[![](./theme1.png)](./theme1.png)

[![](./theme2.png)](./theme2.png)

[![](./theme3.png)](./theme3.png)

[![](./theme4.png)](./theme4.png)

[![](./theme5.png)](./theme5.png)

We decided that the Vikings theme would be best for the first release but we liked the idea some of the others so much we thought about future upgrades where the player can choose from different "themes" they can play with.

I got to work developing the actual game in isometric. I had built a few isometric games before and one effect I like from block based isometric is seeing the blocks move independently, it gives a sort of rippley water effect so I had a go at implementing that.

<iframe width="853" height="480" src="https://www.youtube.com/embed/AaU5sJXYWhs" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

I was originally using React to do all of this but I discovered that react is a little too slow to animate this many objects so I reluctantly changed over to using Phaser3 for the game engine. I really didnt enjoy using Phaser tho, I think I would probably just roll my own solution in the future for more control.

Talking about isometric we had some issues with the length of the ships and isometric depth sorting.

<iframe width="853" height="480" src="https://www.youtube.com/embed/ivIm2bZNQ3Q" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

The issue is that isometric doesnt work too well when you have long sprites. After much discussion and experimentation (and fighting with Phaser) we decided on a solution that split the sprites up so they could be depth sorted individually

[![](./depth-sorting-masks.png)](./depth-sorting-masks.png)

This finally solved the issue.

One other thing we fiddled around with for a while was how to indicate the various states that cells could be in.

[![](./blocks.png)](./blocks.png)

For example when you click a cell to make a guess, it first enters the "guessing" state to indicate that its waiting for a response from the server. It then transitions to either a "hit" or "miss" state depending on if there is a ship there or not. Then finally there is a the "destroyed" state to make it clear that you have destroyed a ship at that cell so you dont need to explore any more.

# Future Plans

So its obviously very early days for the project and to be honest we are really just testing the idea out at this stage. Is the new-tab page a good medium for games? We arent sure. We think it is but time will tell.

If people are interested in the game then we have a bunch of future plans for things we want to work on to improve the game.

[![](./trello.png)](./trello.png)

Our public [Trello Board](https://trello.com/b/EoQS8Hq2/battle-tabs) lists a bunch of the things we have planned. This lets people see what we want to work on next and they can vote on their favorite's or add a comment to suggest something thats not on the list.

One of our favorite upcoming features is per-match-chat. You can use it to send quick messages to your opponent. We are considering making it emoji only chat which would be rather novel. If they are only emoji-based then theres no real way to send abuse thus we dont have to spend time moderating it.

Currently the game is totally free but obviously we will need a way to pay for continued development so we are thinking that we could sell new "themes", "skins" and other content to enhance the game.

Anyways let us know what you think of the game, is it interesting? What do you think we could do to improve it?