
So the big day is finally here! The moment we have been working on since we decided to bring our little [Chrome extension / Web game](https://battletabs.com/) to mobile some 2 years ago.

We have finally made the game globally available on both iOS and Android!

[![](./download-on-app-store.png)](https://apps.apple.com/us/app/battletabs/id6476166902)
[![](./get-on-google-play.png)](https://play.google.com/store/apps/details?id=io.battletabs)

Don't know what I'm talking about? Then check out our trailer:

<iframe width="853" height="480" src="https://www.youtube.com/embed/wGgJjEvNgt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

If you are interested in a little more detail about the history of the game and where that funny name comes from then you can check out [another video we put together here](https://mikecann.blog/posts/battletabs-in-7-minutes).

# Since Soft Launch

Over 6 months ago [we announced](https://mikecann.blog/posts/battletabs-enters-soft-launch) we were entering Soft Launch for mobile. Since then we have been iterating, adding features, tweaking features and fixing bugs. Things have taken a little more time than expected mainly because we decided to focus on our [Discord Activities launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities) which seemed like (and has been) a great opportunity for us.

With that behind us now we are back focused on mobile. The main priority has been performance and stability. One issue with building your mobile game using web technologies is that you unfortunately don't get as much control over resources such as GPU memory as you might do if you were going with a more native approach.

This means we have been at the whim of whatever browser technology is available on the platform. In the case of iOS this means WebKit. It has been known for a while now that WebKit on iOS isn't great, conspiracy theorists would say this is because Apple is intentionally trying to hobble web experiences on mobile in an effort to boost App Store sales. I don't know, all I know is that we have had a bit of a hard time with GPU memory crashes on iOS mobile.

These crashes can happen at just about any point in the game which makes it really tricky to replicate and thus debug the problem. The tooling that Apple provides (only available on Macs of course) isn't great and unfortunately crashes all the time when we try to use it.

We have narrowed down the bulk of the crashing is related to [browser layering](https://medium.com/masmovil-engineering/layers-layers-layers-be-careful-6838d59c07fa) particularly when combined with CSS animated elements. This seems to force a high GPU memory usage as WebKit attempts to hardware accelerate large layers.

Curiously Chrome (and thus Android) does not have this problem so I, as a Windows user, am not able to debug this problem from my PC.

We don't have a definitive solution for the problem, I suspect the issue is still there but we have tried our best to minimize it by removing most usages of CSS animations (on iOS) and changing the League Page over to a much simpler design as it was the culprit of many crashes.

![](./new-league-page.png)

# Apple Feature

A few weeks back we excitedly received an email from Apple informing us that we may be featured on the App Store.

![](./app-feature.png)

We submitted all the media information required so fingers crossed we do. Apple declined to comment on likelihood or timelines so we will see. If we do get a feature then it should hopefully be a big boost to game traffic. In preparation we have scaled up our servers just in case, I don't want another repeat of the [Discord Activities global launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities) ;)

*EDIT*: [We were indeed featured in the US](https://www.linkedin.com/feed/update/urn:li:activity:7258089646545334273/)!

![](./were-featured.jpeg)

# Conclusion

So it's been a long time in the making but we are super excited to have the game finally available to (just about) everyone's mobile device and tablet. I'm looking forward to battling with you guys from around the world soon!

[![](./download-on-app-store.png)](https://apps.apple.com/us/app/battletabs/id6476166902)
[![](./get-on-google-play.png)](https://play.google.com/store/apps/details?id=io.battletabs)



I recently took a much-needed vacation to an exotic location. While I was away, I kept a close eye on the [BattleTabs](https://battletabs.com/) servers. Given [our recent global launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities), things were still a bit unstable, and traffic was continuing to increase.

One issue was that, despite all my best efforts, our worker server was still regularly shutting down and not coming back online.

![](./downtime.png)

This generally happened at points of peak load and usually when I was in bed (typical). The result was that chunks of the game wouldn't work or would work in unexpected ways.

Technically, this should not have been possible as Fly.io's machines have [an auto-start/stop feature](https://fly.io/docs/launch/autostop-autostart/) that should ensure that at least one machine is running at all times.

Since I was on vacation and didn’t have time to debug the root cause of the issue, I decided the simplest thing to do was to write a script that would continually check if the worker server was alive, and if it wasn't, start it up.

I knew that this script would need to be online all the time, so I started thinking about how to host it. I could have easily used my favorite tool, [Convex](https://mikecann.blog/posts/im-now-a-convex-developer-advocate), but I had heard about the recently released [Replit Agents](https://docs.replit.com/replitai/agent), so I thought—why not give that a go?

# Replit Agents

If you're not familiar with Replit Agents, here's a quick primer:

<iframe width="853" height="480" src="https://www.youtube.com/embed/IYiVPrxY8-Y" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

The idea was to use their AI to generate my script and host it on Replit. It should be a fairly simple script and, therefore, hopefully a straightforward, isolated task for the AI.

![](./initial-impressions.png)

I hadn’t used Replit before, and my initial impressions were good. I was very impressed with the UI. It felt like a cut-down code editor, but with the agent chat on the side, taking up a large chunk of the UI to let you know that this is a sort of "pair programming" type of workflow.

I started out with what I felt was a pretty clear prompt (see the image above) and was happy to see that the agent was able to easily scaffold out the start of the project with a bunch of files.

After it had generated the files, it asked me if it worked. Unfortunately, it didn’t, so I had to dig through the code and do a bit of guesswork to try to work out what was going on.

![](./env.png)

It probably would have been better if it worked out this issue by itself, but I was impressed to see that they had a little inline UI for adding an environment variable. I thought that was a nice touch.

This is where things started to go off the rails a bit. The AI continued to make a lot of mistakes, both architecturally and logically, which required quite a few prompts on my part to sort out.

Eventually, I had what I thought should work, but I was still getting errors. I eventually tracked down the problem—it stemmed right back to the first prompt, where the AI was using a very outdated endpoint URL for the API.

![](./tracked-down-the-error.png)

After letting it know, it was able to fix the problem, and I finally had a working API call.

As a side note, I was impressed that Replit has a Git integration, and the AI made regular "checkpoint" commits as it went along.

With it working, I now wanted to have it run itself every 5 minutes. This cloud hosting and cron functionality is one of the core features of Replit, so I thought it should be pretty straightforward for the Agent.

![](./always-on.png)

I was super confused at first as it directed me towards an "Always On" feature that doesn’t seem to exist in Replit (anymore). When I pointed it out to the Agent, it even suggested using another service to run the cron!

I had to manually read through the Replit docs before I worked out how to do it myself in the UI.

![](./deployments.png)

With that, my script was up and working, checking the worker every 5 minutes.

# Conclusions

I really like the Replit Agents UI. I think they have done a great job with how it all looks and works. The functionality of Replit itself is also great, and I will definitely keep it in mind next time I need a quick and dirty project.

The AI, however, just makes way too many mistakes right now for me to recommend. Even with my simple script, I had to go in and manually correct it on multiple occasions.

I will, however, continue to monitor how this develops, as this way of developing software projects feels very much like the future direction for coding.



For the [longest time](https://mikecann.blog/posts/battletabs-6-months-later), Gangbusters has been a Discord-focused company. We use it as our primary method of interacting with our community, receiving bug reports, and gathering feature suggestions.

So when [we heard about](https://venturebeat.com/games/discord-launches-activities-with-light-games-and-a-new-app-store/) Discord's intention to make games playable within the app itself, we knew it was a strong opportunity for our game [BattleTabs](https://battletabs.com/).

Discord is a game-focused social platform where many players are hanging out with friends and looking for something to play together. Having our social multiplayer game embedded right there was a match made in heaven.

![](./battletabs-in-the-app-drawer.png)

In addition to reducing the discovery friction for BattleTabs, Discord also offers a number of other compelling advantages. One of these is the ability to directly message players (we had early access to this feature). We currently use it as another Push Notification channel, but it’s so much more powerful than that—we’ve barely scratched the surface of what’s possible.

![](./discord-dms.png)

Super-fast and friction-free payments are another strong benefit of using Discord, particularly over the web platform where the friction for paying for IAPs or subscriptions makes it super challenging to monetize games like ours. Discord even provides an app "store" where you can see all products that the game offers and purchase them directly without even opening the game.

![](./showing-the-store.png)

There are many other strong reasons to be excited about the Discord platform for games, especially when you read about the [incredible success](https://mojiworks.com/news/why-were-all-in-on-discord-activities) stories [from](https://a16z.com/discord-activities-social-gaming/) other [developers](https://discord.com/build-case-studies/frvr). You can see why we were keen to get on there.

Fast forward a year or two, and after much hard work by [Brandon](https://www.gangbusters.io/about), we found ourselves in the inner circle as one of Discord’s Developer Partners. The process of getting BattleTabs on Discord had begun.

The first step was a "geo test" or soft launch, focusing on Canada. This was an opportunity for Discord to see how the game performed before deciding whether to move forward with a global launch.

![](./happy-meme.png)

After two weeks of the game being available in Canada, we sat down with Discord again, and they gave us the very exciting news that the game had done exceedingly well. As a result, they wanted to take us to a global launch.

We scheduled the official global launch for 10 a.m. my time (Western Australia) on Tuesday, September 10th.

# Tuesday

After a brief call with Discord, we went live! We immediately started to see our usual traffic numbers get obliterated.

![](./initial-spike-graph.png)

And this is where the fun started. To explain the issues we were soon to face, I first have to explain the architecture for BattleTabs a bit.

![](./architecture1.png)

You can see from the above diagram that we have two different kinds of servers. Users connect to the "Web" servers via WebSockets, these scale out horizontally and generally aren’t a problem.

The other kind is the "worker" server. This is a singleton server and was serving many roles (foreshadowing). Some of its responsibilities include:

+ Listening for "app events" and performing various tasks like updating challenge progression, sending push notifications, awarding medals, updating league positions, etc.
+ Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.
+ Listening to database events and performing logic, such as alerting web servers to update subscribed users when their currency values change (due to rewards or other events).
+ Handling tasks like cheat detection, server metrics reporting, Discord server bot management, Discord live reporting, and more.

I was aware that the worker was a bottleneck in the architecture, but up to now, it hadn’t been an issue. The worker usually sat at a very low CPU and memory level.

Regardless, I beefed the worker up to the largest size that [our hosting supported](https://fly.io/docs/about/pricing/).

![](./worker-sitting-okay.png)

It was sitting at about 50% utilization for most of the day, which was a bit high but still within the bounds of comfort. So I went to bed at about 9 p.m., feeling good about the launch so far... little did I know what was coming.

![](./bed-time.png)

# Wednesday

I woke up to an absolute nightmare. The servers had clearly been having a bad time while I was asleep. I hopped on [our Discord Server](https://discord.gg/fTdxf4y9) and saw that many players were complaining about the game not working correctly.

I took a look at the app event queue and, to my horror, saw that there were over 3 million events in the queue waiting to be processed.

![](./3m-queued-events.png)

The number was decreasing, but even after much tweaking of the various settings, I calculated it would take 26 hours for my poor single worker server to clear them all down, on top of everything else it was dealing with.

Before I could make any architectural changes to the servers, I needed to clear this queue to avoid losing data or causing more problems by handling events out of order.

![](./slack-to-brandon.png)

So although the game was in a semi-broken state, I went to bed thinking that at least the queue would be mostly cleared by morning, allowing me to focus on fixing everything properly.

# Thursday

![](./4m.png)

... well sh*t ... this isn’t good...

It turns out that while I was sleeping, we had some "good" news from our friends at Discord...

![](./moving-you-up-to-rank-4.png)

Ah, crap... I knew at this point I had to take action. This queue wasn’t going to clear itself.

I asked Discord to drop us back down the list. As painful as it was to request LESS traffic, I needed to buy us some breathing room.

The next step was to parallelize that event queue processing.

So I created a new kind of server called an "eventsWorker." This server took much of the event processing work from the worker, and critically, it could be parallelized.

![](./architecture2.png)

There was a risk of out-of-order event processing, but I carefully examined each event and realized there wasn’t much event-order dependence, particularly between users. Anything that was dependent, I could likely fix later.

This realization gave me the confidence to scale up the number of eventsWorker instances. However, this put a huge strain on the database, so I scaled it up to the largest size that Fly would allow.

![](./16-performance-core-db-pricing.png)

This allowed the eventsWorkers to scale to 15 instances, ripping through the queue at over 1 million events per hour. The entire queue should be cleared by late afternoon.

Based on that assumption, I got to work preparing the update that would go out once the queue was resolved.

I decided to reduce as many places as possible that depended on app events, thus reducing pressure on the background workers.

After thinking it over, I realized that most events could be processed "locally" instead of being sent to the background.

To explain what I mean, let's take an example event, `matchFinished`. (By the way, if you're curious, [here are all the app events](https://gist.github.com/mikecann/c7ad434ffcf94eb4e5c0d0b30f7967e1) in the game.) Here's where this event is used:

+ BattleEventsService - to send push notifications to users about the battle outcome
+ BattleWebhooksService - to send API messages to our Tournaments bot for advancing tournaments
+ BattleTabsLiveService - to update the #battletabs-live channel in Discord
+ HighScoresService - to update highscore stats
+ MedalAwardingService - to potentially award a user a new medal
+ UserChatDetectionService - to check for abuse if a user plays the same person multiple times
+ UserStatsService - to update a user’s stats
+ ChallengeUpdatingService - to progress any challenges the user might be on
+ TVChannelsService - to update any playing BattleTabsTV instances

Each of these is a totally different feature in the game and independent of each other, which is why they are handled as events rather than inline function calls in the first place.

When a battle was finished, I would publish a `matchFinished` event to the queue ([managed by pgboss](https://github.com/timgit/pg-boss)). The event would then be picked up by a background worker to handle all that work mentioned above.

I realized, though, that instead of sending it to the background, I could simply emit a "server event" in addition to the "app event". These server events would only be listenable by services running on that one server.

This shifted the computational burden from the background workers to the web workers, which could easily scale horizontally.

It seems obvious now, but hindsight is 20/20. I had been so focused on client-side improvements that I missed these obvious server-side optimizations. To be fair, it was only the increased traffic that necessitated the change.

So, shortly after 4 p.m., the events were all cleared. I scaled the eventsWorkers down from 15 to 3, then pushed my "server events" change.

This immediately reduced worker load but increased web server load as expected. Web servers could [auto-scale](https://fly.io/docs/reference/autoscaling/) up however so I wasn't worried.

I went to bed feeling pretty good about things. I did tell Brandon (who is based in the UK) to call me if the servers went down.

I soon fell asleep, fairly confident I would get a good night’s rest...

# Friday

3:14 a.m. – I got a call from Brandon letting me know that the game was down.

I jumped out of bed and hopped on my PC. Sure enough, things were not looking good and hadn’t been for a couple of hours.

![](./friday-morning-graph.png)

This time, it wasn’t immediately obvious what was wrong. It appeared that the worker process was running out of memory. I attempted some desperate fixes but wasn’t sure about the underlying reason.

![](./desperate-commit-log.png)

By 4:30 a.m. my brain really wasn’t working properly. I felt like the game was stable, so I decided to get an hour of sleep.

### Later on Friday

Awake and rested, I realized what the problem might be.

As previously mentioned, one of the worker server’s responsibilities was:

> Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.

For example, matchmaking works by requesting a list of open battle proposals every few seconds, then looping over that list and creating battles as needed.

Turn timeouts, AI turn-taking, and a bunch of other systems worked similarly.

The issue was that this polling was effectively another kind of queue. If the number of rows returned from the database during each poll was too small, that queue would build up.

Unfortunately, I had no metrics around these polling "queues," so my first task was to get some visibility into them.

![](./poll-queue-growth-chart.png)

Sure enough, those queues had grown to huge levels.

Much like my app events problem earlier, I knew the solution would be parallelization.

So, I carefully moved the polling code over to the eventsWorker instances. I was very nervous about this, as I knew that allowing multiple servers to poll the database simultaneously risked race conditions and other problems due to polling overlaps.

My solution was to carefully place [distributed locks](https://github.com/mike-marcacci/node-redlock) to ensure that two different workers couldn’t work on the same item from a polling queue simultaneously.

This solution worked well, and I soon had the polling queues back under control.

![](./polling-under-control-graph.png)

That’s when disaster struck...

![](./slack-to-brandon-wild-db-growth.png)

I had been keeping a close eye on our database’s growth, knowing that if we ran out of space, it would put a hard stop to the game.

Well, it turns out that with all the chaos of the past 12 hours, I forgot to check the DB size, and it had grown wildly. It finally ran out of space and started refusing connections.

Fortunately, I was looking at the logs when I saw this message go by:

![](./jobs-cannot-archive-log-message.png)

It turns out that pgboss, our app events queuing and jobs system, was timing out when trying to archive completed jobs. I ran a quick query against the database and saw over 7 million event rows waiting to be archived, with another 3 million waiting to be deleted.

It seems like pgboss wasn’t designed to handle this scenario very well and was erroring out during its archival and deletion routines.

My solution was to manually delete all 10 million rows from the database immediately. This instantly freed up 70 GB of space. I then ran a [Postgres VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html) to clear up any deleted tuples.

Instantly, the database came alive again, and disaster was averted.

Phew! It had been a long day, but I was finally feeling good about things. I went to bed certain that everything was all good.

# Saturday

I woke up, hopped on my PC, and to my dismay, saw that my "worker stopped" alert had triggered.

![](./slack-message-alert-worker-stopped.png)

Fortunately, it didn’t take long to track down that this time the worker hadn’t actually gone down. The alert had triggered because [Axiom](https://axiom.co/) (our log aggregation tool) had stopped receiving log messages. This happened because our "[log shipper](https://github.com/superfly/fly-log-shipper)" instance had died.

I still don’t know why or how this happened, but I quickly gave it a poke, and things were up and running again.

From the Grafana logs on the servers, it appeared that everything had been fine overnight, much to my relief.

I think I’m going to leave the day-by-day commentary here, but I believe you’ve got a good sense of the firefight I was putting up at this point.

# Lessons

So what lessons did I learn from all this?

1. **Be wary of queues**. Anytime you’re building something that could get backed up if you 10x or 100x your traffic, it’s wise to deal with it now or, at the very least, get some metrics around it so you can keep an eye on it.

2. **Single points of failure are ticking time bombs**. For the same reasons as above, be cautious of any places that cannot scale horizontally.

3. **Use Axiom**. It’s fast, cheap, and powerful. Log everything, then build graphs.

![](./axiom-pricing.png)

500GB on the free tier? That’s just insane.

To put it into perspective: despite all the spam we produced over the past week, we’ve amassed 54 million log lines, amounting to just 17GB of space, which is only 1GB compressed!

![](./axiom-dataset-logs.png)

4. **Load test if you can** before you expect a large increase in traffic. Even if you can’t easily hit the expected numbers on your local machine, it should at least hint at where bottlenecks might occur.

# Conclusions & Thanks

It’s still a bit early to talk about the results of all this traffic on the business, but things are going well.

I would like to thank the special people at Discord who took a chance on us and were very patient while we fought through these problems. :)



Phew its been a long long journey for our little game [BattleTabs](https://battletabs.com/) ([watch a summary video here](https://mikecann.blog/posts/battletabs-in-7-minutes)) but finally we are entering the phase that is a culmination of all the hard work, creativity, and perseverance: [the soft launch!](https://play.google.com/store/apps/details?id=io.battletabs)

# The Trailer

Before I start woffling on and loose you, heres some eye-candy:

<iframe width="853" height="480" src="https://www.youtube.com/embed/wGgJjEvNgt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Its almost not cringe! :D

# What Is a Soft Launch?

For those who might be wondering, a "soft launch" is a strategy used in the gaming industry where a new game is released to a limited audience or in specific regions before its official worldwide release.

In our case we are launching today [just on Android and just in the Philippines](https://play.google.com/store/apps/details?id=io.battletabs).

This phase allows us to gather valuable feedback, fine-tune gameplay, and ensure that everything is as perfect as it can be.

It’s a critical step in ensuring that BattleTabs delivers the best possible experience to its players.

# The Last 18 months

For the past year and half we have been beavering away at breakneck pace converting our web and chrome extension game over to work on mobile. Along the way we have added many new features and refined many of the existing.

Many parts of the game were changed in particular the UI has undergone a massive upgrade to make sure that the game works on small handheld devices in portrait mode.

That last part is worth underscoring. BattleTabs works in both Landscape (on the web or tablets) and Portrait (on mobile). This was no small task but I think we have pulled it off and im pretty proud of it.

![](./layout_1.png)

![](./layout_2.png)

Its also worth noting that, as often get asked this, the game is entirely Typescript using React for the UI and BabylonJS for the 3D rendering. I don't think there are too many games out there that have taken this path not sure if that is a good thing or not but it certainly sets us apart.

# What’s Next?

During the soft launch, we’ll be closely monitoring every aspect of the game, from user experience to the technical performance, particularly server performance.

We still have some important changes we want to make before final release such as a new First Time User Experience (tutorial) as that is super critical for players who are quick to dismiss something if they are confused or frustrated.

Post launch we have a number of exciting improvements and features planned but I might have to leave that for a future post :)

# How You Can Get Involved

Right now the game is only available on Android and in the Philippines but we are going to slowly be expanding the number of countries and platforms we are on.

We invite you to join us in this exciting phase. The best place to stay up to date is to [join the 16,000 other people on our Discord Server](https://discord.gg/battletabs). We post to the [#announcements](https://discord.com/channels/770955477215084554/770963532455477310) channel regularly about everything thats going on with the game.

Your feedback will be invaluable in shaping the final version of BattleTabs. Whether it’s a bug report, a suggestion, or even just sharing your favorite moments, every bit of communication helps us make BattleTabs better for everyone!

# Wrapping Up

The soft launch is just the beginning of the next chapter in the BattleTabs saga.

We’re incredibly proud of what we’ve achieved so far and even more excited about what the future holds.

Stay tuned, and [let the battles begin](https://play.google.com/store/apps/details?id=io.battletabs)!



As you may be aware we have been working really hard lately on the next version of our game [BattleTabs](https://mikecann.blog/tags/battletabs).

In an effort to broaden the game's reach for the upcoming mobile release, we decided to invest some time in producing "devlogs".

If you're not familiar with devlogs, they are a popular format on YouTube where developers discuss their past and current progress with a game.

We hope that viewers will find the video entertaining and informative about the game. Ideally, they will subscribe and follow along as the game progresses and future devlogs are released.

With that in mind, we created our first devlog. This one is longer than future ones because I was eager to tell the story up to this point. Watch it and let us know what you think!

<iframe width="853" height="480" src="https://www.youtube.com/embed/kDlEoLjflGw" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

We were initially hesitant about investing the time (about 5 days) to prepare and create this video, as that's time I could have spent working on the game.

However, we justified the time investment because marketing is an area where none of us on the team are experts. We feel the need to explore every possible avenue to spread the word about the game and attract new users.

Speaking of which, we are taking [BattleTabs to the Perth Pixel Expo](https://www.pixelexpo.org.au/game-devs) this year, where we have a double-wide booth.

![](./expo.png)

I'm looking forward to showcasing the game in person to a larger audience, running tournaments, and giving out stickers and t-shirts!

We plan to continue our devlog videos roughly monthly, depending on other commitments and the content we have to show off.

Anyway, let us know what you think in the comments. If there are things we can improve for future videos and, importantly, other communities you think might be interested in this video, we'd love to hear from you!



AI is all the rage at the moment and for good reason, its going to revolutionize many industries.

Since I first wrote about it in the context of Game Development back [in November](https://mikecann.blog/posts/ai-and-the-future-of-game-development) I have been thinking how we can apply some of the advances to [our](https://gangbusters.io/) game [BattleTabs](https://battletabs.io/).

# Problem

Since the first version of the game (3+ years ago) we allowed users to name their player. This, we feel, gives a better sense of ownership and attachment to the game.

![](./signup.png)

A year or so after launch [we introduced the "Custom Fleets" feature](https://mikecann.blog/posts/battletabs-ships-and-fleets-update). As part of this feature we also allowed the user to name their fleet.

![](./fleet-name.png)

Again we feel this creates a stronger feel of attachment to the game and their choices within it.

Both of these features however opened up an annoying issue for us: profanity.

We started to get reports of players being offended by the names of other players.

![](./early.png)

This was an issue as lots of children played BattleTabs so we felt like we had to do something about it.

# Local Filtering

So.. we started to implement a "naughty words" list.. which grew.. and grew and grew..

![](./list.png)

No matter how many words or phrases we put on this list, users always seemed to find ways to be offensive.

Did you know for example that "Tangina nyo" is a swear word in Fillipino? Nope, neither did I.

# Community Filtering

So after many months of this we decided to enlist some help.

We implemented a "Report Player" button on the player profile page.

![](./report.png)

This then sent a "Report" to a special channel in discord that only us and our Community Moderators could see.

![](./reports.png)

We then added special discord commands to our bot that allowed our moderators to rename the player and then potentially ban then from renaming themselves again (repeat offenders).

This worked.. kind of.

Obviously it relied on people reporting others and then out Community moderators being available to handle the report.

So it meant that some names slipped through the cracks or went days before being detected.

# AI Enters The Chat

Enter ChatGPT in late 2022 and suddenly every problems seems possible to solve with the hammer that is LLMs.

So we decided to do some quick experimentation and see if AI could detect profanity in a phase.

We created a sample spreadsheet of a sample of profanities and non-profanities and hooked up ChatGPT plugin to Google Sheets and tried out a few prompts.

![](./sheet.png)

It definitely _seemed_ to work. Enough that it was worth going ahead and implementing it in the code.

Ill save you all the iterations we did around the prompts and just show you what we ended up with:

```ts
{
  max_tokens: 20,
  temperature: 0,
  messages: [
    {
      role: "system",
      content: `I want you to act as a profanity checker. I will give you potentially profane word or phrase and you will tell me if it is profane or not. Do not provide personal opinions or assumptions. Do not respond asking how you can assist. Simply respond with the words 'profane' or 'not profane'. If it is profane you may also give a very short explanation of why you think it is profane.`,
    },
    {
      role: "user",
      content: `the phrase is: '${phrase}'`,
    },
  ],
  model: "gpt-3.5-turbo-0301",
}
```

We set the temperature to very low as we wanted it to give consistent outputs given the same inputs, but im not sure is that makes much difference.

We have quite a long prompt, this was based on recommendations from [awesome chat gpt prompts](https://github.com/f/awesome-chatgpt-prompts).

We also added the `the phrase is: '${phrase}'`, instead of simply providing the phrase because for some reason the AI would still get confused and try to respond to the phrase rather than the system message:

![](./whoops.png)

For example in the above you can see that if the player provides their name as "yes" then the AI responds with "Great, let's begin. What is the first word or phrase you would like me to check?" instead of the expected "not profane".

Its not perfect and we are still refining it but it generally works really well and has caught thousands and thousands of profane player names and fleet names.

![](./results.png)

# Conclusion

In the end im super happy with how this has turned out. For minimal effort and cost we were able to get a much more super profanity checker improving the experience for the 99.9% of our players who dont want to see profanity while playing [BattleTabs](https://mikecann.blog).



I have been a long fan of Heroku and we have been using it as the platform that powers [BattleTabs](https://battletabs.io/) for over 3 years and before that [Markd](https://markd.co) but they have long since stopped innovating and after a number of stressful events I decided it was time to break up.

# TLDR;

Ditch Heroku, move to fly.io, thank me later.

# Pain

Moving a Database to another platform is never an easy thing to do so I resisted it for the longest time despite knowing that we were paying way over the market price for the service. It just wasn't worth the developer time and potential risk of moving an unknown platform. Sometimes its "better the devil you know".

But then the issues started happening.

Every now and then, for no apparent reason, the database would grind to a crawl. Queries would start taking minutes to execute before eventually timing out. This then put backwards pressure on the web-servers as they buffered requests and started to suffer memory issues.

And of course, this all would happen just before I was going to bed at night.

My thinking at the time was perhaps we were outgrowing the database.

You see, Heroku doesnt really offer much in the way of metrics for your database. Sure they have the Postgres dashboard but that thing is pretty minimal at best and doesn't tell you anything about the CPU or memory pressures that the DB is under.

My solution each time this happened was to simply make the DB larger.

We started working our way through [the plans](https://elements.heroku.com/addons/heroku-postgresql#pricing) progressively getting larger and larger until we ended up paying $400 per month just for our Postgres instance.

I was getting more and more frustrated and nervous about this because I had no idea what the fundamental issue was.

Eventually I opened a support ticket with Heroku and they told me to try to use the `heroku pg:diagnose` tool, so I did and I got the following:

![](./diagnose.png)

Notice that one item in yellow there "Disk Burst".

What is disk burst I ask myself? I didnt see anything on Heroku's [pricing page](https://elements.heroku.com/addons/heroku-postgresql#pricing) about IO limits

![](./pricing.png)

So I ask the Heroku Support and well..

![](./support.png)

Okay so Heroku has "hidden" IO limits that I was totally unaware of before and we are hitting this on certain tables and it is silently throttling us with no errors or warning or anything.

So in the end I ended up bumping up the size of the DB again but by this point I had had enough, it was time to move on.

# Moving

I started looking around at options and oh boy there are a lot of options for Postgres.

I'll save you hours and hours of research and comparison and just say that I ended up landing on [Fly.io](http://fly.io/).

The [pricing](https://fly.io/docs/postgres/) is just insane we can quite literally get 6 databases of the same size as Heroku's or one massive 8-core 32GB machine for the same price.

The price is definitely nice but thats not the main reason we moved, the main one was to get away from the hidden limits and poor metrics. Thankfully Fly.io really excels in this regard providing not only a basic metrics dashboard but a MUCH more complete Grafana dashboard.

![](./basic.png)

![](./grafana.png)

Through that I can get a really clear view on how much memory, CPU, and importantly, IO the DB is using.

![](./io.png)

You can even setup alerts to let you know if things go out of whack.

What this information let me do was to see that we actually didn't need all the resources we were paying for on Heroku and so once the arbitrary IO limit was gone I reduced the size of the DB down from 8GB down to 4GB and the disk size down from 256GB to 40GB.

TBH I think we could even go smaller if we wanted to play it a bit more riskily.

# Cons

So its not all sunshine and unicorn farts of course.

Fly.io is NOT a managed service [as their website](https://fly.io/docs/postgres/getting-started/what-you-should-know/) takes great pains to highlight. This means upgrades and downtime are all down to you.

Its also quite a bit more work to get things setup just right. For example you have to set up your own external IP if you want to access your database from outside of your applications.

Fly is still pretty young and its platform is constantly moving. This can be both a good and bad thing. There are some rough edges in places but new features and upgrades are constantly being added and the devs are very responsive to customer feedback.

# Conclusion

So in the end if you start off paying $400 per month and reduce that down to $20 per month you can get a 20x saving on your DB costs AND get a peaceful rest at night knowing that you have lots more visibility into whats going on and alerts to let you know when things go pair shaped.

Hopefully this is the last database migration we ever have to make for the game!



Im super excited to announce that [Gangbusters](https://www.gangbusters.io/) (the company that I am a co-founder and CTO of) has been joined on its journey by the gaming powerhouse [Bandai Namco](https://021fund.bn-ent.net/en/news/)!

We are super excited by this as it not only allows us to grow our team and ambitions it also validates our strategy and enhances our brand by having the creator / publisher of [such large IPs](https://en.wikipedia.org/wiki/List_of_Bandai_Namco_video_game_franchises) behind us:

![](./bandi-ips.jpg)

Bandai are joining a number of other investors as part of our early-seed round to help us realize our Web3 ambitions and take BattleTabs to mobile.

![](./other-investors.png)

Its been a long roller-coaster of a journey to reach this point but after 3 years of working in a very small team im really happy that we are finally able to grow a little bit and hire some more gang-members.

We still have a tough path ahead of us with some ambitious plans but I think we are starting to assemble a fantastic team that will allow us to get there. I personally cant wait to show our incredible community what we can do now!

If you are interested in partnering up or what we are working on, reach out to: mike@gangbusters.io or brandon@gangbusters.io.



Its been a while since I have posted about BattleTabs but things most definitely haven't been standing still, so I think its time for an update!

# Background

Just quickly; [BattleTabs](https://battletabs.com/) is the premier game from [Gangbusters](https://www.gangbusters.io/), the company that im a co-founder of.

I have [posted quite a few times in the past](https://mikecann.blog/tags/battletabs) about BattleTabs so checkout those posts for more background on the game and how it works.

# Whats Changed

![](./update-notes.jpg)

Since November last year there have been over 40 updates to the game. Some of the major ones include:

- Rivalry Feature - challenge others to "just one more battle"
- Huge change to the battles architecture allowing for more complex ship-ship interactions
- New content - ships, emotes, etc
- Lots of changes to support Manifest v3 Chrome Extensions
- Added new sound and visual effects
- Added push notifications to the web versions of the game
- Added "Daily Deals" feature to the shop
- AI turn taking for players that go offline
- Board Ping feature
- TONs of bug fixes

In addition to the above we also started working on our long-term strategy for the company.

# Cards & Web3

We strongly believe in the power of community driven games. We think that a game can be good but the community is what makes it great. This is certainly the case for BattleTabs.

Over the almost 3 years of running the game we have amassed over a million downloads, millions of battles played and over 15,000 members on discord. 24 hours a day players on discord discuss the game, create content, share ideas or just generally hang out.

Given this, we started to think how we could empower our community to go further and at the same time develop a strategy that allows us to continue to develop the game and company.

This is where BattleTabs Cards comes in.

From our [community page](https://battletabs.com/battletabs-cards) on the topic:

> BattleTabs Cards are digital collectible cards that you can trade with other players. They will be released in Sets, where each set will feature a particular theme or art style. Every Set will be limited in quantity, and cards will have different rarities.

<iframe width="853" height="480" src="https://www.youtube.com/embed/LFV1ugBAb1M?playlist=LFV1ugBAb1M&loop=1;rel=0&autoplay=1&controls=0&showinfo=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

> Outside of their intrinsic value in looking amazing cards will also provide unique benefits in BattleTabs and other experiences we have in production. For example, you may craft a special cosmetic item or a ship skin in BattleTabs using a combination of cards, then display or trade the crafted treasure inside your own 3D sandbox! Watch as the newly crafted item comes alive with special animations and behaviours.

![](./storyboard_c.png)

> Additionally, BattleTabs Cards can be freely built-upon without any intervention or restrictions from us. Say you have a fantastic idea for a new game, you can team up with your friends, use the exact same "shared card database" that we use!

![](./storyboard_a.png)

> You could host tournaments with these cards, perhaps charging players a Common card to enter the tournament and setting Rare cards as a reward for the winners.

> Say you have a vast stack of cards saved up and want to give some to helpful members in the community, you can do that with cards!

We think it an exciting direction for the company and community. To kick things off we are running a tournament on [Fractal.is](https://fractal.is).

# Fractal Tournament

Our first card is going to be special. Rather than simply selling or giving them away we thought it would be more fun if we rewarded top players and community members by giving them preferential access.

So every day that the tournament runs (2 weeks), the top 20 players that participate will earn a silver token:

![](./silver-token.png)

The silver token along with the golden token which we awarded to a selection of of most loyal players will grant free minting rights to the Seed Card.

![](./gold-token.png)

The tournament starts tomorrow, monday the 15th of August. Head over to [https://www.fractal.is/battletabs/tournaments](https://www.fractal.is/battletabs/tournaments) for all the details!

# Fractal Interview

In preparation we sat down with Justin Kan who co-founded JustinTV (Twitch) and a number of other companies including [Fractal.is](https://Fractal.is) where the tournament is being held.

<iframe width="853" height="480" src="https://www.youtube.com/embed/nRbUc9N9hGY" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

We talk about the game, the tournament, web3 in general and some future projects. Check it out!

# Conclusion

The past 9 months have been a total rollercoaster of ups and downs (no one ever said running a startup would be easy huh?). The ever-changing funding landscape in particular has been a source of continual headaches.

It does however feel great to wake up each morning and work on something that brings obvious joy to thousands of people daily.

I hope we can continue to grow and improve our game, business and community for many years to come!

BTW: I generated the header image using [Dalle](https://openai.com/dall-e-2/).. so cool :)


Well its been years in the planning and months in the development but the "Big Summer Update" / "Seasons" / V1 update for our game [BattleTabs](https://battletabs.com) is finally out!

This is by far our biggest update yet. We generally don't like making large changes to the game like this, preferring small frequent incremental changes but it was necessary however as there are lots of exciting new overlapping systems and features in this update!



# Progression

For ages now there has been a large missing feature in the game one so obvious it stands as a testament to the [enjoyability and popularity](https://twitter.com/theBrandonWu/status/1458058505072652300) of the game that we were able to get this far without it. That missing feature was progression.

Sure we have had a rudimentary sort of progression via preset fleet unlocks but this was rather crude and players tend to whip through that part of the game:

![](./preset-fleet-unlocks.png)

The new update however brings a whole new dimension to progression in the game. Players now earn XP for various actions during a battle which contributes to their season level.

![](./game-over-modal.png)

# Season

Seasons are a feature common to Free-2-Play games and exist as a mechanism to keep the game feeling fresh as the developer continually works on new content. The season is periodically (every few months) refreshed and the player's level resets. This gives new players to the game a way to enter the game without feeling like they will never catch up with the players that joined before them.

The XP earned contributes to their season level:

![](./progress-scroller-1.png)

Each level the player can claim various rewards:

![](./reward1.png)

There a lots of different kinds of rewards: Avatar Cosmetics, Custom Emotes, Ship Blueprints, Gold, Gems etc

There are 60 levels and two "tiers" of rewards. Theres the free tier, that anyone can claim rewards from, then there is the premium "Battle Pass" tier:

![](./bp-modal.png)

The Battle Pass costs 2000 gold to unlock (1000 with Battle Club, our subscription product) and gives the player 2x XP rewards and substantially better rewards. Battle Pass also resets at the end of each season.

The player can grind enough gold during the free tier to unlock the Battle Pass for free eventually but they will loose all the double XP bonus they could have achieved along the way if they do that, so we are hoping this will be a strong incentive to visit our new Shop.

# Shop

![](./shop1.png)

Another common Free-2-Play mechanic that this update brings is In App Purchases (IAP) via the Shop. The player can spend fiat on in game currency in the form of Gold and Gems.

Gold can be used to unlock the Battle Pass but Gold and Gems can also be used to rent and buy ships.

# Ship Blueprints

Ships are the core game play feature for the game and can be thought of as cards in a traditional Collectable Card Game (CCG). You collect Ships (Cards) and combine them together in Fleets (Decks) and fight other players.

This update brings the concept of "Ship Blueprints" to game.

![](./blueprint-states.jpg)

The player can "discover" blueprints during the season then rent the ship for a few days for some gold. Once rented the ship can be used in a fleet. If it expires it can no longer be used and will have to be rented again. If the player get tired of renting a ship they can alternatively buy it outright with some gems. Once bought / owned they will never have to rent it again.

This Ship Rental concept gives the player something to spend their gold on and forces them into some interesting cost / benefit choices as they play.

# Content

The update is jam-packed with new content including:

- 6 New Ships
- 29 New Avatar Cosmetics
- 24 New Emotes
- 5 New Medals

The key is; once the season is over this content will no longer be unlockable. This incentives players to unlock everything before the season ends (FOMO). We may bring back some of the rewards in the future in the shop or through other rewarding mechanisms.

# Gangbusters

![](./logo_full.png)

This update brings a name-change for the company.

[3 years ago](https://mikecann.blog/posts/farewell-bamboo-hello-markd) Brandon and I started working on a SaaS product called "Markd.co" and so we named the company "Markd Ltd".

Some time later we decided to pivot the company and focus on BattleTabs. For for some time now we have felt that the name no longer suits what we are doing and have been searching for a new name.

We traded different ideas until we hit on "Gangbusters". We like it as it hints at excitement, growth and energy, all things we want to instil in the company.

"Gangbusters" as a word also has some utility as it can be split into "Gang" and "Busters" so employees or fans of the company could be called "Gang Members" or perhaps "Busters". Its also a word in common parlance that hasnt been taken, which helps :P

# Conclusion

I hope this post has given you a little taste of this update. We have worked really hard on it and I am personally quite proud of the results. We are due to be featured on the GooglePlay ChromeOS store soon and the hope is this will increase the number of users and excitement around the game significantly.

If you are in interested in checking out out head over to [https://battletabs.io/](https://battletabs.io/) and get started. Ill see you there!