
This is just going to be a quick one to say that I'm super excited to announce that I am joining [Convex](convex.dev) as a Developer Experience Engineer!

If you don't know what Convex is then do yourself a favor and [checkout their excellent site](https://www.convex.dev/) and [docs](https://docs.convex.dev/) but in short its “a sync platform that replaces your backend and client state management.” I half-jokingly describe it as “Firebase for adults”

I am super excited about this role as it means I get to do what I love most; tinkering with and producing content about cool technology.

# Why join Convex?

If you have been following along on this blog for the past year or so you will have noticed me be quite effusive about my love for the Convex platform:

- [Tinkering With Convex](https://mikecann.blog/posts/tinkering-with-convex)
- [Chatter Craft - collaborative art with just your voice](https://mikecann.blog/posts/chatter-craft-collaborative-art-with-just-your-voice)
- [Im now a Convex.dev Developer Champion!](https://mikecann.blog/posts/im-now-a-convex-developer-advocate)
- [Whisper Farm - Parallel Transcription](https://mikecann.blog/posts/whisper-farm-parallel-transcription)
- [Mikebot - The Virtual Me](https://mikecann.blog/posts/mikebot-the-virtual-me)

With 17+ years of professional development experience (yes, I'm getting old!) and a blog spanning over two decades, I've explored countless ways to build projects. After all this experience, I genuinely believe Convex is the best platform for building apps.

If you're curious about the specific technical features that won me over, check out my post: https://stack.convex.dev/convex-the-database-that-made-me-switch-careers

Beyond the technology, the company itself is exceptional. While it's no surprise that former Dropbox core-tech team members are brilliant, what truly stands out is their character and dedication to building [a fantastic developer experience](https://mikecann.blog/posts/tinkering-with-convex).

# What's happening to Gangbusters & BattleTabs?

I have a detailed explanation for leaving Gangbusters that I'll save for a longer retrospective post. The short version is that the company has run out of money. Unlike when we started 6+ years ago, I now have [a family](https://mikecann.blog/posts/joshua-christopher-cann) and [mortgage](https://mikecann.blog/posts/moving-home-and-startlink) to sustain. I chose not to seek additional funding because, honestly, I'm feeling burnt out from working on the same project for so long.

I want to emphasize my deep respect and admiration for my co-founder [Brandon](linkedin.com/in/thebrandonwu/), whose optimism throughout these 6+ years has been truly inspiring. I have so much to say about how great Brandon was to worth with but I think ill leave that for the retrospective.

We're currently figuring out the future of both the game and the company. We're open to all possibilities, so please reach out to [Brandon](https://www.notion.so/Gangbusters-a-Retrospective-150fd70ecfa0801ca372eeadcbf73658?pvs=21) or [me](https://www.notion.so/10ffd70ecfa0800eaf28d8aa2604be10?pvs=21) if you have any suggestions.

The game's community holds a special place in my heart. While players have come and gone over the years, those who've stayed have been one of my strongest motivations to keep developing this game, I ♥️ you guys!



You know what's better than one Mike Cann? That's right, infinity Mike Canns.

![](./infinity-monkeys.png)

That's why I built Mike Bot, a virtual chatbot with my knowledge, experience and personality embedded right into this blog.

![](./how-to-start-mikebot.png)

Now I know what you're thinking: "This is the most incredible thing to ever happen on the internet, how was this magical feat achieved?". Well, let me tell you...

# The Plan

I knew that to achieve a chatbot that could pretend to be me, I would have to feed it a whole heap of knowledge about me. Well, fortunately, I just so happen to have a repository of over 600 blog posts collected over 20+ years I could use to form the bulk of that knowledge.

Next was the meat of the problem: The Brain. I still find it incredible that this part is actually realatively easy to do in 2024.

Thanks [to these Convex posts](https://www.convex.dev/can-do/rag) on the subject, I had 3 different ways I could go about it:

1. [Using OpenAI's Assistants API](https://stack.convex.dev/ai-chat-using-openai-assistants-api)
2. [Using Langchain with our own storage](https://stack.convex.dev/ai-chat-using-langchain-and-convex)
3. [Custom message store and vector search retrieval](https://stack.convex.dev/ai-chat-with-convex-vector-search)

Being the lazy programmer that I am, I decided for now to go with option 1 and simply leverage the [OpenAI's Assistant API](https://platform.openai.com/docs/api-reference/assistants) for most of the hard work.

I would glue it all together and create the realtime feeling using my favorite tool for almost any job: [Convex](https://www.convex.dev/).

![](./convex-hammer.png)

# OpenAI Assistants

First step was to fire up the [OpenAI dashboard](https://platform.openai.com/assistants/) and create the Assistant.

![](./assistant-setup.png)

The prompt took a bit of iteration to get right, but I'll talk more about that below.

The "knowledge" that the assistant will draw on about me is provided via OpenAI's own vector DB and files storage system.

![](./vector-store.png)

Convex does have its own [VectorDB](https://docs.convex.dev/search/vector-search) and [File Storage](https://docs.convex.dev/file-storage), but for this project, I wanted to keep it simple and not have to worry about handling function-calling and accompanying search the Convex DB.

# React Component

Next up was how the user was going to interact with Mikebot. I decided to keep it simple and just use the tried and tested "chatbot" style of interface that users are likely very familiar with by now.

![](./mikebot-widget.png)

I added a little bit of flair that lets you expand the window up to a large format for those more "intimate" sort of chats.

![](./mikebot-expanded.png)

# Convex Glue

Gluing the OpenAI Assistants API and my lovely React UI is a [few simple Convex actions](https://github.com/mikecann/mikecann.blog/blob/main/convex/messages.ts#L22) [that call into the OpenAI API](https://github.com/mikecann/mikecann.blog/blob/main/convex/openai/assistants.ts#L15). The results are [streamed](https://github.com/mikecann/mikecann.blog/blob/main/convex/openai/assistants.ts#L100) into [the database](https://github.com/mikecann/mikecann.blog/blob/main/convex/schema.ts#L55).

Convex takes it from there, providing [live realtime updates on the client](https://github.com/mikecann/mikecann.blog/blob/main/components/mikebot/MessagesList.tsx#L48).

# Prompt Engineering

Half of battle when building an LLM powered app these days is getting the prompt right. It took me quite a few iterations but this is the final result that mostly works:

```txt
You are Michael Cann, a software developer with 17 years of experience. You write about AI, coding, and your projects on your blog. 

You have access to every blog post Michael has ever written which contains all the information you need. You may do the file search to retrieve this information. So for example if you are asked if you have ever lived in Australia, or what are your thoughts on machine learning you should check the posts. 

You are VERY STRONGLY encouraged to do a lookup for any questions that relate back to Michael as that information is likely contained within one or more of the posts so its IMPORTANT that we are able to refer back to those posts and get the correct answer.

Respond in a casual, humorous yet knowledgeable tone. Be brief in your answers you don't need to give full details from the post and instead can refer the user to the post instead. 

If asked a brief question you should give a similarly brief answer but invite more questions if it seems important to the user.

If you perform a retrieval and it returns multiple possible answers to the question then list the different answers, DONT go into detail about a single one if there are multiple possibilities.

If asked, the best way to contact mike is via email: mike.cann@gmail.com.
```

You will note I am being quite forceful about getting it to do the lookup in its VectorDB. For whatever reason, it was reluctant to do that at first. 

For example, if I asked it "Have you ever made a game?" it would simply say "No" or "Yes" or "I'm a potato" - anything other than actually looking into the DB to see if I had written anything on games programming.

Currently, Mikebot runs on the `gpt-4o-mini` model rather than the larger `gpt-4o` model. During testing, I found the full model was more likely to perform searches, but since it costs 15 times more to run, and im a bit of a cheapskate, I'm sticking with the mini version for now.

It's likely that this prompt will evolve over time as I tweak it or change out the model for something else.

# Uploading Posts

Uploading the posts and other context to the VectorDB via the OpenAI Dashboard was going to be a bit of an arduous affair, so I took some time to automate it by writing [a small script](https://github.com/mikecann/mikecann.blog/blob/main/scripts/openai/uploadPostsToStorage.ts) that is run on every build on the CI.

One thing to note here is that it's important [that I name](https://github.com/mikecann/mikecann.blog/blob/main/scripts/openai/uploadPostsToStorage.ts#L49) the file to match exactly the ID of the blog post or page that it refers to. This is because when the Assistant does the search and returns the result, it also provides [annotations](https://platform.openai.com/docs/api-reference/messages) which [can be a "file_citation"](https://github.com/mikecann/mikecann.blog/blob/main/convex/openai/assistants.ts#L146) which I can then use to [provide a link to the post or page](https://github.com/mikecann/mikecann.blog/blob/main/components/mikebot/MessageContent.tsx#L26).

![](./link-to-post.png)

# Conclusion

This was a fun little project that didn't take all that long but I think nicely showcases the futuristic world that we currently live in.

Got feedback or ideas for improvements? Feel free to leave a comment below... or better yet, save us both some time and just chat with Mikebot directly. He's probably more helpful than I am anyway!



So the big day is finally here! The moment we have been working on since we decided to bring our little [Chrome extension / Web game](https://battletabs.com/) to mobile some 2 years ago.

We have finally made the game globally available on both iOS and Android!

[![](./download-on-app-store.png)](https://apps.apple.com/us/app/battletabs/id6476166902)
[![](./get-on-google-play.png)](https://play.google.com/store/apps/details?id=io.battletabs)

Don't know what I'm talking about? Then check out our trailer:

<iframe width="853" height="480" src="https://www.youtube.com/embed/wGgJjEvNgt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

If you are interested in a little more detail about the history of the game and where that funny name comes from then you can check out [another video we put together here](https://mikecann.blog/posts/battletabs-in-7-minutes).

# Since Soft Launch

Over 6 months ago [we announced](https://mikecann.blog/posts/battletabs-enters-soft-launch) we were entering Soft Launch for mobile. Since then we have been iterating, adding features, tweaking features and fixing bugs. Things have taken a little more time than expected mainly because we decided to focus on our [Discord Activities launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities) which seemed like (and has been) a great opportunity for us.

With that behind us now we are back focused on mobile. The main priority has been performance and stability. One issue with building your mobile game using web technologies is that you unfortunately don't get as much control over resources such as GPU memory as you might do if you were going with a more native approach.

This means we have been at the whim of whatever browser technology is available on the platform. In the case of iOS this means WebKit. It has been known for a while now that WebKit on iOS isn't great, conspiracy theorists would say this is because Apple is intentionally trying to hobble web experiences on mobile in an effort to boost App Store sales. I don't know, all I know is that we have had a bit of a hard time with GPU memory crashes on iOS mobile.

These crashes can happen at just about any point in the game which makes it really tricky to replicate and thus debug the problem. The tooling that Apple provides (only available on Macs of course) isn't great and unfortunately crashes all the time when we try to use it.

We have narrowed down the bulk of the crashing is related to [browser layering](https://medium.com/masmovil-engineering/layers-layers-layers-be-careful-6838d59c07fa) particularly when combined with CSS animated elements. This seems to force a high GPU memory usage as WebKit attempts to hardware accelerate large layers.

Curiously Chrome (and thus Android) does not have this problem so I, as a Windows user, am not able to debug this problem from my PC.

We don't have a definitive solution for the problem, I suspect the issue is still there but we have tried our best to minimize it by removing most usages of CSS animations (on iOS) and changing the League Page over to a much simpler design as it was the culprit of many crashes.

![](./new-league-page.png)

# Apple Feature

A few weeks back we excitedly received an email from Apple informing us that we may be featured on the App Store.

![](./app-feature.png)

We submitted all the media information required so fingers crossed we do. Apple declined to comment on likelihood or timelines so we will see. If we do get a feature then it should hopefully be a big boost to game traffic. In preparation we have scaled up our servers just in case, I don't want another repeat of the [Discord Activities global launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities) ;)

*EDIT*: [We were indeed featured in the US](https://www.linkedin.com/feed/update/urn:li:activity:7258089646545334273/)!

![](./were-featured.jpeg)

# Conclusion

So it's been a long time in the making but we are super excited to have the game finally available to (just about) everyone's mobile device and tablet. I'm looking forward to battling with you guys from around the world soon!

[![](./download-on-app-store.png)](https://apps.apple.com/us/app/battletabs/id6476166902)
[![](./get-on-google-play.png)](https://play.google.com/store/apps/details?id=io.battletabs)



I recently took a much-needed vacation to an exotic location. While I was away, I kept a close eye on the [BattleTabs](https://battletabs.com/) servers. Given [our recent global launch](https://mikecann.blog/posts/battletabs-global-launch-on-discord-activities), things were still a bit unstable, and traffic was continuing to increase.

One issue was that, despite all my best efforts, our worker server was still regularly shutting down and not coming back online.

![](./downtime.png)

This generally happened at points of peak load and usually when I was in bed (typical). The result was that chunks of the game wouldn't work or would work in unexpected ways.

Technically, this should not have been possible as Fly.io's machines have [an auto-start/stop feature](https://fly.io/docs/launch/autostop-autostart/) that should ensure that at least one machine is running at all times.

Since I was on vacation and didn’t have time to debug the root cause of the issue, I decided the simplest thing to do was to write a script that would continually check if the worker server was alive, and if it wasn't, start it up.

I knew that this script would need to be online all the time, so I started thinking about how to host it. I could have easily used my favorite tool, [Convex](https://mikecann.blog/posts/im-now-a-convex-developer-advocate), but I had heard about the recently released [Replit Agents](https://docs.replit.com/replitai/agent), so I thought—why not give that a go?

# Replit Agents

If you're not familiar with Replit Agents, here's a quick primer:

<iframe width="853" height="480" src="https://www.youtube.com/embed/IYiVPrxY8-Y" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

The idea was to use their AI to generate my script and host it on Replit. It should be a fairly simple script and, therefore, hopefully a straightforward, isolated task for the AI.

![](./initial-impressions.png)

I hadn’t used Replit before, and my initial impressions were good. I was very impressed with the UI. It felt like a cut-down code editor, but with the agent chat on the side, taking up a large chunk of the UI to let you know that this is a sort of "pair programming" type of workflow.

I started out with what I felt was a pretty clear prompt (see the image above) and was happy to see that the agent was able to easily scaffold out the start of the project with a bunch of files.

After it had generated the files, it asked me if it worked. Unfortunately, it didn’t, so I had to dig through the code and do a bit of guesswork to try to work out what was going on.

![](./env.png)

It probably would have been better if it worked out this issue by itself, but I was impressed to see that they had a little inline UI for adding an environment variable. I thought that was a nice touch.

This is where things started to go off the rails a bit. The AI continued to make a lot of mistakes, both architecturally and logically, which required quite a few prompts on my part to sort out.

Eventually, I had what I thought should work, but I was still getting errors. I eventually tracked down the problem—it stemmed right back to the first prompt, where the AI was using a very outdated endpoint URL for the API.

![](./tracked-down-the-error.png)

After letting it know, it was able to fix the problem, and I finally had a working API call.

As a side note, I was impressed that Replit has a Git integration, and the AI made regular "checkpoint" commits as it went along.

With it working, I now wanted to have it run itself every 5 minutes. This cloud hosting and cron functionality is one of the core features of Replit, so I thought it should be pretty straightforward for the Agent.

![](./always-on.png)

I was super confused at first as it directed me towards an "Always On" feature that doesn’t seem to exist in Replit (anymore). When I pointed it out to the Agent, it even suggested using another service to run the cron!

I had to manually read through the Replit docs before I worked out how to do it myself in the UI.

![](./deployments.png)

With that, my script was up and working, checking the worker every 5 minutes.

# Conclusions

I really like the Replit Agents UI. I think they have done a great job with how it all looks and works. The functionality of Replit itself is also great, and I will definitely keep it in mind next time I need a quick and dirty project.

The AI, however, just makes way too many mistakes right now for me to recommend. Even with my simple script, I had to go in and manually correct it on multiple occasions.

I will, however, continue to monitor how this develops, as this way of developing software projects feels very much like the future direction for coding.



For the [longest time](https://mikecann.blog/posts/battletabs-6-months-later), Gangbusters has been a Discord-focused company. We use it as our primary method of interacting with our community, receiving bug reports, and gathering feature suggestions.

So when [we heard about](https://venturebeat.com/games/discord-launches-activities-with-light-games-and-a-new-app-store/) Discord's intention to make games playable within the app itself, we knew it was a strong opportunity for our game [BattleTabs](https://battletabs.com/).

Discord is a game-focused social platform where many players are hanging out with friends and looking for something to play together. Having our social multiplayer game embedded right there was a match made in heaven.

![](./battletabs-in-the-app-drawer.png)

In addition to reducing the discovery friction for BattleTabs, Discord also offers a number of other compelling advantages. One of these is the ability to directly message players (we had early access to this feature). We currently use it as another Push Notification channel, but it’s so much more powerful than that—we’ve barely scratched the surface of what’s possible.

![](./discord-dms.png)

Super-fast and friction-free payments are another strong benefit of using Discord, particularly over the web platform where the friction for paying for IAPs or subscriptions makes it super challenging to monetize games like ours. Discord even provides an app "store" where you can see all products that the game offers and purchase them directly without even opening the game.

![](./showing-the-store.png)

There are many other strong reasons to be excited about the Discord platform for games, especially when you read about the [incredible success](https://mojiworks.com/news/why-were-all-in-on-discord-activities) stories [from](https://a16z.com/discord-activities-social-gaming/) other [developers](https://discord.com/build-case-studies/frvr). You can see why we were keen to get on there.

Fast forward a year or two, and after much hard work by [Brandon](https://www.gangbusters.io/about), we found ourselves in the inner circle as one of Discord’s Developer Partners. The process of getting BattleTabs on Discord had begun.

The first step was a "geo test" or soft launch, focusing on Canada. This was an opportunity for Discord to see how the game performed before deciding whether to move forward with a global launch.

![](./happy-meme.png)

After two weeks of the game being available in Canada, we sat down with Discord again, and they gave us the very exciting news that the game had done exceedingly well. As a result, they wanted to take us to a global launch.

We scheduled the official global launch for 10 a.m. my time (Western Australia) on Tuesday, September 10th.

# Tuesday

After a brief call with Discord, we went live! We immediately started to see our usual traffic numbers get obliterated.

![](./initial-spike-graph.png)

And this is where the fun started. To explain the issues we were soon to face, I first have to explain the architecture for BattleTabs a bit.

![](./architecture1.png)

You can see from the above diagram that we have two different kinds of servers. Users connect to the "Web" servers via WebSockets, these scale out horizontally and generally aren’t a problem.

The other kind is the "worker" server. This is a singleton server and was serving many roles (foreshadowing). Some of its responsibilities include:

+ Listening for "app events" and performing various tasks like updating challenge progression, sending push notifications, awarding medals, updating league positions, etc.
+ Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.
+ Listening to database events and performing logic, such as alerting web servers to update subscribed users when their currency values change (due to rewards or other events).
+ Handling tasks like cheat detection, server metrics reporting, Discord server bot management, Discord live reporting, and more.

I was aware that the worker was a bottleneck in the architecture, but up to now, it hadn’t been an issue. The worker usually sat at a very low CPU and memory level.

Regardless, I beefed the worker up to the largest size that [our hosting supported](https://fly.io/docs/about/pricing/).

![](./worker-sitting-okay.png)

It was sitting at about 50% utilization for most of the day, which was a bit high but still within the bounds of comfort. So I went to bed at about 9 p.m., feeling good about the launch so far... little did I know what was coming.

![](./bed-time.png)

# Wednesday

I woke up to an absolute nightmare. The servers had clearly been having a bad time while I was asleep. I hopped on [our Discord Server](https://discord.gg/fTdxf4y9) and saw that many players were complaining about the game not working correctly.

I took a look at the app event queue and, to my horror, saw that there were over 3 million events in the queue waiting to be processed.

![](./3m-queued-events.png)

The number was decreasing, but even after much tweaking of the various settings, I calculated it would take 26 hours for my poor single worker server to clear them all down, on top of everything else it was dealing with.

Before I could make any architectural changes to the servers, I needed to clear this queue to avoid losing data or causing more problems by handling events out of order.

![](./slack-to-brandon.png)

So although the game was in a semi-broken state, I went to bed thinking that at least the queue would be mostly cleared by morning, allowing me to focus on fixing everything properly.

# Thursday

![](./4m.png)

... well sh*t ... this isn’t good...

It turns out that while I was sleeping, we had some "good" news from our friends at Discord...

![](./moving-you-up-to-rank-4.png)

Ah, crap... I knew at this point I had to take action. This queue wasn’t going to clear itself.

I asked Discord to drop us back down the list. As painful as it was to request LESS traffic, I needed to buy us some breathing room.

The next step was to parallelize that event queue processing.

So I created a new kind of server called an "eventsWorker." This server took much of the event processing work from the worker, and critically, it could be parallelized.

![](./architecture2.png)

There was a risk of out-of-order event processing, but I carefully examined each event and realized there wasn’t much event-order dependence, particularly between users. Anything that was dependent, I could likely fix later.

This realization gave me the confidence to scale up the number of eventsWorker instances. However, this put a huge strain on the database, so I scaled it up to the largest size that Fly would allow.

![](./16-performance-core-db-pricing.png)

This allowed the eventsWorkers to scale to 15 instances, ripping through the queue at over 1 million events per hour. The entire queue should be cleared by late afternoon.

Based on that assumption, I got to work preparing the update that would go out once the queue was resolved.

I decided to reduce as many places as possible that depended on app events, thus reducing pressure on the background workers.

After thinking it over, I realized that most events could be processed "locally" instead of being sent to the background.

To explain what I mean, let's take an example event, `matchFinished`. (By the way, if you're curious, [here are all the app events](https://gist.github.com/mikecann/c7ad434ffcf94eb4e5c0d0b30f7967e1) in the game.) Here's where this event is used:

+ BattleEventsService - to send push notifications to users about the battle outcome
+ BattleWebhooksService - to send API messages to our Tournaments bot for advancing tournaments
+ BattleTabsLiveService - to update the #battletabs-live channel in Discord
+ HighScoresService - to update highscore stats
+ MedalAwardingService - to potentially award a user a new medal
+ UserChatDetectionService - to check for abuse if a user plays the same person multiple times
+ UserStatsService - to update a user’s stats
+ ChallengeUpdatingService - to progress any challenges the user might be on
+ TVChannelsService - to update any playing BattleTabsTV instances

Each of these is a totally different feature in the game and independent of each other, which is why they are handled as events rather than inline function calls in the first place.

When a battle was finished, I would publish a `matchFinished` event to the queue ([managed by pgboss](https://github.com/timgit/pg-boss)). The event would then be picked up by a background worker to handle all that work mentioned above.

I realized, though, that instead of sending it to the background, I could simply emit a "server event" in addition to the "app event". These server events would only be listenable by services running on that one server.

This shifted the computational burden from the background workers to the web workers, which could easily scale horizontally.

It seems obvious now, but hindsight is 20/20. I had been so focused on client-side improvements that I missed these obvious server-side optimizations. To be fair, it was only the increased traffic that necessitated the change.

So, shortly after 4 p.m., the events were all cleared. I scaled the eventsWorkers down from 15 to 3, then pushed my "server events" change.

This immediately reduced worker load but increased web server load as expected. Web servers could [auto-scale](https://fly.io/docs/reference/autoscaling/) up however so I wasn't worried.

I went to bed feeling pretty good about things. I did tell Brandon (who is based in the UK) to call me if the servers went down.

I soon fell asleep, fairly confident I would get a good night’s rest...

# Friday

3:14 a.m. – I got a call from Brandon letting me know that the game was down.

I jumped out of bed and hopped on my PC. Sure enough, things were not looking good and hadn’t been for a couple of hours.

![](./friday-morning-graph.png)

This time, it wasn’t immediately obvious what was wrong. It appeared that the worker process was running out of memory. I attempted some desperate fixes but wasn’t sure about the underlying reason.

![](./desperate-commit-log.png)

By 4:30 a.m. my brain really wasn’t working properly. I felt like the game was stable, so I decided to get an hour of sleep.

### Later on Friday

Awake and rested, I realized what the problem might be.

As previously mentioned, one of the worker server’s responsibilities was:

> Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.

For example, matchmaking works by requesting a list of open battle proposals every few seconds, then looping over that list and creating battles as needed.

Turn timeouts, AI turn-taking, and a bunch of other systems worked similarly.

The issue was that this polling was effectively another kind of queue. If the number of rows returned from the database during each poll was too small, that queue would build up.

Unfortunately, I had no metrics around these polling "queues," so my first task was to get some visibility into them.

![](./poll-queue-growth-chart.png)

Sure enough, those queues had grown to huge levels.

Much like my app events problem earlier, I knew the solution would be parallelization.

So, I carefully moved the polling code over to the eventsWorker instances. I was very nervous about this, as I knew that allowing multiple servers to poll the database simultaneously risked race conditions and other problems due to polling overlaps.

My solution was to carefully place [distributed locks](https://github.com/mike-marcacci/node-redlock) to ensure that two different workers couldn’t work on the same item from a polling queue simultaneously.

This solution worked well, and I soon had the polling queues back under control.

![](./polling-under-control-graph.png)

That’s when disaster struck...

![](./slack-to-brandon-wild-db-growth.png)

I had been keeping a close eye on our database’s growth, knowing that if we ran out of space, it would put a hard stop to the game.

Well, it turns out that with all the chaos of the past 12 hours, I forgot to check the DB size, and it had grown wildly. It finally ran out of space and started refusing connections.

Fortunately, I was looking at the logs when I saw this message go by:

![](./jobs-cannot-archive-log-message.png)

It turns out that pgboss, our app events queuing and jobs system, was timing out when trying to archive completed jobs. I ran a quick query against the database and saw over 7 million event rows waiting to be archived, with another 3 million waiting to be deleted.

It seems like pgboss wasn’t designed to handle this scenario very well and was erroring out during its archival and deletion routines.

My solution was to manually delete all 10 million rows from the database immediately. This instantly freed up 70 GB of space. I then ran a [Postgres VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html) to clear up any deleted tuples.

Instantly, the database came alive again, and disaster was averted.

Phew! It had been a long day, but I was finally feeling good about things. I went to bed certain that everything was all good.

# Saturday

I woke up, hopped on my PC, and to my dismay, saw that my "worker stopped" alert had triggered.

![](./slack-message-alert-worker-stopped.png)

Fortunately, it didn’t take long to track down that this time the worker hadn’t actually gone down. The alert had triggered because [Axiom](https://axiom.co/) (our log aggregation tool) had stopped receiving log messages. This happened because our "[log shipper](https://github.com/superfly/fly-log-shipper)" instance had died.

I still don’t know why or how this happened, but I quickly gave it a poke, and things were up and running again.

From the Grafana logs on the servers, it appeared that everything had been fine overnight, much to my relief.

I think I’m going to leave the day-by-day commentary here, but I believe you’ve got a good sense of the firefight I was putting up at this point.

# Lessons

So what lessons did I learn from all this?

1. **Be wary of queues**. Anytime you’re building something that could get backed up if you 10x or 100x your traffic, it’s wise to deal with it now or, at the very least, get some metrics around it so you can keep an eye on it.

2. **Single points of failure are ticking time bombs**. For the same reasons as above, be cautious of any places that cannot scale horizontally.

3. **Use Axiom**. It’s fast, cheap, and powerful. Log everything, then build graphs.

![](./axiom-pricing.png)

500GB on the free tier? That’s just insane.

To put it into perspective: despite all the spam we produced over the past week, we’ve amassed 54 million log lines, amounting to just 17GB of space, which is only 1GB compressed!

![](./axiom-dataset-logs.png)

4. **Load test if you can** before you expect a large increase in traffic. Even if you can’t easily hit the expected numbers on your local machine, it should at least hint at where bottlenecks might occur.

# Conclusions & Thanks

It’s still a bit early to talk about the results of all this traffic on the business, but things are going well.

I would like to thank the special people at Discord who took a chance on us and were very patient while we fought through these problems. :)



I have a confession to make—I have a superpower: the ability to stay in the flow.

![Hero Image](./hero.webp)

Throughout the workday, I'm continually assaulted with potential "side quests."

A friend might send me a message with a link to an article they think I'd enjoy. Bam! New side quest: read the article, formulate an opinion, and reply.

Or I'm reading some technical document for work and suddenly come across a link to something tangential. Bam! New side quest: go down the rabbit hole following links.

Or I'm out chatting with someone about a fascinating topic, and they mention something I should check out. Bam! New side quest: read the article on my phone instead of paying attention to my friend.

Each of these side quests, although potentially fun, inevitably pulls me out of the "flow" of whatever I was doing, disrupting my most productive self.

So, what is my superpower? It's a practice or tool I've been a fan of for a long time...

## Save It for Later

[Pocket](https://getpocket.com/) (initially named "Read It Later") is a simple service that stores links you can browse later.

![Pocket Image](./pocket.webp)

It works via a mobile app and Chrome extension. You simply press a button to trigger the extension or use another mobile button to share to the app—very simple and quick.

This means you can stay in the flow but still track that side quest for later.

Pocket worked for me for many years, but then one day, they made an update that frustrated me, so I decided to switch to a similar feature built into my RSS reader of choice, [Inoreader](https://www.inoreader.com/).

But then Inoreader decided to jack up its prices beyond what I was willing to pay.

![Expired Image](./expired.webp)

So, I was kind of fed up at this point and thought what every self-respecting software engineer thinks to themselves: "Sod it, I'll just build it myself! How hard can it be?"

## StashIt

![StashIt Icon](./icon.webp)

Just like that, [StashIt](http://stashit.lol/) was born.

Quickly on the name and logo: I wanted to keep with the side quest/game-related theme with the "stash," but then I realized that combining "Stash" and "It" together created a rather amusing double/triple entendre, hence the domain choice https://stashit.lol/.

There isn't much to say about the functionality; if you know Pocket, then it's basically like that but with far fewer features.

![StashIt Screenshot](./stashit-ss.webp)

Right now, the main web app lets you add, archive, delete, and read items. This was all I really needed for the initial version, at least.

It was important that it worked well on mobile since I manage most of my side quests when I'm out and about or sitting on the sofa. So, the first thing I did was ensure that the web app was nice and responsive.

![Responsive Image](./responsive.webp)

I then wrapped it up into a [Capacitor](https://capacitorjs.com/) app.

![Mobile Image](./mobile.webp)

The final piece was putting together a quick Chrome extension that allowed me to quickly track that side quest when browsing the web.

![Extension Image](./extension.webp)

And that's about it, functionality-wise—nice and simple, but very useful.

Oh, one more thing: because I built it, I can tailor specific functionality to suit me.

For example, this blog now has an additional page: https://mikecann.blog/stash

![Blog Page Image](./blogpage.webp)

So, I now have a page I can point people to that shows a continually changing list of side quests I've been on lately. :)

## Technical Specifics

Another goal of this project was to explore some technical ideas and scratch some itches that my work at [Gangbusters](https://www.gangbusters.io/) can't currently reach.

So, in no particular order, here are some of the technologies used in this project:

- Convex
- React
- TypeScript
- Next.js
- Capacitor
- Vite
- Turbo-repo
- Chrome Extension API
- Capgo Live Updates
- Convex Auth
- Radix Themes
- Bun

This blog post would turn into a book if I talked about each of those, so instead, here are some notable thoughts on a few of them.

### Convex

![Convex Image](./convex.webp)

A fantastic experience, [as always](https://mikecann.blog/posts/tinkering-with-convex). It's always a pleasure to start a new project with Convex. The speed at which I can get going and how clear the data model is—it's just a joy.

However, I bumped into an issue I've mentioned before.

![Count Image](./count.webp)

Unfortunately, Convex doesn't have any "aggregation" query methods. This means that to show the number of items within the stash or archive, you have to use some sub-optimal solutions.

The standard recommendation is to keep track of this count yourself in the database and increment or decrement it as you add or remove items. This approach is potentially error-prone, as you must remember to update this cross-cutting concern in multiple mutations.

Another way is to return all the records in the table within a query, then use `.length` on the array. This is the method I ended up using for this solution. I capped the number of items returned to 500, so if the count exceeds 500, it will just read "500+."

This probably means that query is quite expensive to run, but for now, it's okay.

There is hope on the horizon. Convex is very aware of these "aggregation" issues and is [exploring ways to solve them](https://github.com/ldanilek/aggregate). Fingers crossed, as this is just about the only big thing I miss from Postgres.

### Next.js

![Next.js Image](./next.webp)

I decided to go with Next.js on this project. Although this blog is powered by Next.js, it's using the old pages way of doing things, and I haven't updated it in a while, so I thought I'd try the new hotness and see what everyone is talking about.

Well, I'm not impressed. Sure, it works and _may_ be flexible and fast (at render time), but at what price?

- I find the SSR confusion real. The React Server Components "use-client" or "use-server" is just a headache conceptually, and I end up sprinkling "use-client" everywhere to make the error messages go away.

[This post](https://www.mux.com/blog/what-are-react-server-components?utm_source=pocket_saves) makes me feel better that I'm not the only one.

- The dev experience is way slower than Vite because it doesn't pre-build the pages—only when you first visit them. You end up with a big pause when trying to navigate.

- Several times, my Next.js cache got corrupted. How? I have no idea. All I know is I suddenly started getting very strange errors that took me ages to work out. In the end, I just deleted the `.next` folder, built it again, and all was well. Still super frustrating.

So yeah, I don't think I'll go with Next.js next time.

### Capgo Live Updates

![Capgo Image](./capgo.webp)

[BattleTabs](https://mikecann.blog/posts/introducing-battletabs) is built on Capacitor, so I decided to go with it for the mobile app for this project too.

BattleTabs works by simply iframing https://battletabs.io. This means updates are a breeze, as we simply need to reload the iframe within the Capacitor wrapper to have it auto-update.

I probably should write a blog post on this at some point, as I think it's a super interesting way of designing a cross-platform app, but I'll leave that for another day.

The more traditional way of updating a web-based mobile app these days, however, is to use a service that can "live update" the code by having the device download the new code bundle, then unpack and replace the existing code. This means you're able to provide speedy updates without having to resubmit to the various app stores each time.

For a Capacitor app, there are two main options that I could see: either [Appflow's Live Updates Service](https://capacitorjs.com/docs/guides/deploying-updates) or [CapGo's Live Updates Service](https://capgo.app/).

Both have free and paid tiers, but Capgo also has an open-source version. I wanted to experiment with self-hosting my own updates using a [simple HTTP action](https://github.com/mikecann/stashit/blob/master/packages/convex/convex/updates.ts) I wrote using Convex, then storing the updates in Convex's storage system.

When the CI builds the game, it zips up the output and then [uploads it to the Convex backend, which creates a new version](https://github.com/mikecann/stashit/blob/master/apps/client/scripts/createAndUploadNewAppVersion.ts) ready for the mobile app to consume.

This all works really well and was surprisingly simple to do—I was pleasantly surprised.

### Radix Themes

![Radix Image](./radix.webp)

Radix Components are used

in many of the popular [ShadCN/ui](https://ui.shadcn.com/) components, so I thought I'd give their opinionated ["themes"](https://www.radix-ui.com/themes/docs/overview/getting-started) library a go.

I enjoyed the [very prescriptive](https://www.radix-ui.com/themes/docs/theme/overview) way of doing things—sometimes it's nice to just be told one way to do things.

However, I still found myself needing to use the `style` prop fairly often. I also found myself wishing for a bit more control around styling. I started to suspect that I would have to pull in Tailwind. I did resist, but I think I'll have to bow to the inevitable eventually.

I also found it kind of odd how only some CSS properties were listed as attributes. I suspect this is intentional, as with everything in Radix, but coming from the pure flexibility of the [emotion](https://emotion.sh/docs/introduction)-based [ChakraUI](https://v2.chakra-ui.com/), it just felt a little frustrating at times. Again, I suspect this is intentional.

So, all in all, I'm not exactly sure what to think about Radix Themes. I think I'm still looking for the perfect UI library. ChakraUI gets close—if only it wasn't such a massive drain on TypeScript compilation performance.

### Convex Auth

![Auth Image](./auth.webp)

All my Convex projects up to this point have been using [Clerk](https://clerk.com/) for their authentication. However, I was getting issues when trying to implement Clerk into the Capacitor app, so I decided, why not give the [newly released Convex Auth](https://www.convex.dev/auth) a try instead?

It's all very new stuff, but it seemed to work quite well. I think I have a bit more tinkering to do with it, but I was pleasantly surprised by how smooth everything worked.

## Project Status & Conclusions

For now, this is still very much a personal project that I'm going to keep tinkering with and adding to as I need it.

I was thinking of releasing the source, but I decided not to just yet, as everything is a bit all over the place, and I'm likely to change stuff over the next several months. If you're super interested in having a look, let me know, and I can sort something out. :)

I may revisit this project in a future post, so stay tuned!



I enjoy reading [Ian Macartney's](https://stack.convex.dev/author/ian-macartney) posts on the Convex blog. When I saw his piece on [Work Stealing](https://stack.convex.dev/work-stealing), a strategy for distributing compute-heavy workloads, I decided to try it myself.

# TL;DR

Check out the demo video below to see what I built:

<iframe width="100%" height="400" src="https://www.youtube.com/embed/sovYE3sWszY" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Code: [https://github.com/mikecann/whisper-farm](https://github.com/mikecann/whisper-farm)

# Whisper Farm

I aimed to speed up podcast transcription by parallelizing the task into separate "jobs" and performing each in parallel on [Fly.io's GPU machines](https://fly.io/gpu).

Each Fly.io machine runs a single Docker container with a Whisper binary from [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win), an [FFmpeg](https://github.com/kribblo/node-ffmpeg-installer) binary, and a Node.js "worker" for "chunking" or "transcription" jobs.

To orchestrate this, I used [Convex](https://convex.dev), an excellent backend as a service that I’ve discussed [many times before](https://mikecann.blog/tags/convex).

# Challenges

This project was more challenging than I initially thought.

Most difficulties were related to Docker and Fly.io, while the Convex part was problem-free.

## Docker

Not being a Docker expert caused several headaches when packaging the Whisper model and code.

I started off in high hopes after reading [this blog post](https://fly.io/blog/transcribing-on-fly-gpu-machines/) from Fly. However, I soon realized I didn't want to run it as a standalone container. Instead, I wanted my worker code to run alongside it, simplifying scaling and cost management. I wanted a self-contained worker.

Merging two Dockerfiles turned out to be complex. Eventually, I abandoned that approach and used a [Whisper binary](https://github.com/Purfview/whisper-standalone-win), dynamically downloading and unzipping it when the container was constructed.

Another issue was the large Docker image size, around 15 GB, causing problems on Fly.io. Machines wouldn't start or took a long time, likely due to the image size. Fly.io's error messages weren't helpful here.

To reduce the Docker image size, I learned (thanks, ChatGPT!) that each "RUN" command creates a new layer. By rearranging and combining "RUN" commands, I reduced the image size from 15 GB to 5 GB.

![](./dockerimagesize.webp)

This experience taught me that I probably need to learn a bit more about Docker fundamentals.

# UI - Shadcn & Tailwind

Another side quest was experimenting with "the new hotness" in web UI design.

![](./shadcnui.webp)

[Shadcn/ui](https://ui.shadcn.com/) is popular right now. The idea is to copy and paste component code directly into your project and modify it, creating your own component library.

Shadcn uses Tailwind for styling. Although I’ve avoided Tailwind due to its "[stringly typed](https://wiki.c2.com/?StringlyTyped)" nature, I decided to try it since it’s widely used.

My conclusions?

I’m just not sold on Tailwind. I dislike its stringly typed nature and the need for additional tools like `prettier-plugin-tailwindcss` and extra linting rules. Managing className attributes requires more libraries like `tailwind-merge` and `class-variance-authority` which irks me. Tailwind also needs its own compiler/toolchain which is yet another headache to deal with.

Shadcn/ui was also disappointing. While the copy-paste nature is quick, the components lacked features I expected from a UI library. Modifying components that often looked like this is just simply confusing:

![](./tailwind.png)

# Issues and Future Work

Back to the main point of this post.

I think I proved the concept of work-stealing and learned a lot along the way. There is however more work that would be needed to take this further.

Some key areas include:

- Transcription results might exceed the 1 MB row limit in the database (https://docs.convex.dev/production/state/limits), so storing it as a file might be necessary.
- Error handling for Fly.io machine startup needs improvement.
- More testing is required to ensure stability.
- The code is messy and needs tidying up.
- There’s no authorization or authentication.
- Worker count limits are needed to prevent potential issues.
- The UI is basic and was more of an experiment with Shadcn and Tailwind.



I've been meaning to tinker with Temporal for quite some time now, as it intersects some of my key interests: [Event Sourcing](https://mikecann.blog/posts/serverless-databaseless-event-sourcing), [durable execution](https://mikecann.blog/posts/tech-to-tinker-with-in-2022), and [cool application architectures](https://mikecann.blog/posts/tinkering-with-convex).

# What is Temporal?

From their [website](https://temporal.io/):

> Durable Execution is a development abstraction that preserves complete application state so that upon host or software failure, it can seamlessly migrate execution to another machine.

But what does that mean exactly?

From a coder's perspective, it means you can define "Workflows" in TypeScript such as:

```ts
export async function SubscriptionWorkflow(email: string, trialPeriod: string | number) {
  await sendWelcomeEmailToUser(email);
  await sleep(trialPeriod);
  await sendSubscriptionEndedEmailToUser(email);
}
```

Temporal ensures this function continues to exist even if the server that runs it shuts down. When the server starts up again, it can resume from the correct location, making it "durable." Temporal also handles errors and retries the steps in the Workflow if they fail.

If this doesn't immediately seem impressive, consider how difficult it is to resume a function's execution at any arbitrary point after the fact, especially when dealing with non-deterministic states like DateTimes and random numbers.

# Exploring Temporal

My goal with any "tinkering" I do is to understand the technology enough to form an opinion about it.

I started with [the docs](https://docs.temporal.io/), which are quite thorough. Setting up the environment included installing the Temporal CLI and running their local service, which even comes with a nice WebUI to display your Workflows:

![](./webui.png)

After reading through the initial pages of the docs, I summarized the key structure and terms with this diagram:

![](./diagram.png)

Clients start Workflows, which can contain several Activities. Workflows enter a Task Queue from which Workers pull and execute tasks. If activities fail, they can be retried an infinite (configurable) number of times.

Every "[Action](https://docs.temporal.io/cloud/pricing#action)" the system takes (starting a workflow, running an activity, signals, queries, etc.) is an event recorded within the system, which is then recorded against your workflow so you can view exactly what happened and when:

![](./events.png)

## Event Sourcing

It's worth taking a tangent to discuss another application architecture dear to my heart: [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html).

I won't explain all of Event Sourcing here, as I've covered it before on this blog and there are many good deep-dives available online. The basic idea is that your entire application should be described with immutable events. To get the current state of your application, you fold/reduce your events down to a state. For example, you might have a "UserCreated" event with the user's name "Dave." Later, if Dave changes his name to "Davina," the system triggers a "UserChangedName" event.

Because we record all the events in order and each event is immutable, we can always reconstruct the current state of the user at a later date and see exactly what led to that state. This state reconstruction is called "Projection" in Event Sourcing terminology and is fundamental to Event Sourcing.

Now, suppose you want to model a "Process" like the Temporal Workflow example at the start of this post.

1. A user clicking a button creates a "UserTrialSubscriptionStartRequested" event.
2. A "Process Manager" picks up that event, sends the user an email, and emits a "UserTrialSubscriptionStarted" event.
3. It could also set up a timer so that after the trial period ends, a "UserTrialPeriodEnded" event is fired.
4. The Process Manager detects that event, sends the user another email to let them know their trial has ended, and emits a "UserTrialSubscriptionEnded" event.

What I've outlined with "Process Managers" is exactly what a Temporal "Workflow" does but with a nicer developer experience. Temporal uses clever build-time tooling to ensure that Activities within a Workflow are the entry points when an event occurs. Because side effects are constrained to Activities, we can replay a Workflow to resume a paused state.

# Why?

So why would you want to do this? Many applications (finance, health, military, industry, etc.) need to ensure something happens and have a clear record of exactly what happened to potentially reconstruct events.

Event-based architectures like Event Sourcing and Temporal capture the system as events, then reconstruct the world from those events, rather than the other way around (a typical CRUD system). Temporal.io cleverly combines the best parts of Event Sourcing (the Process Managers) into an entirely standalone product and names it Durable Execution.

# Unanswered Questions

The Temporal docs are extensive, and there's a lot I haven't explored yet.

I've run out of time for this tinkering session, but next time I'd like to build a simple application to see what works and what doesn't. Here are some notes for next time:

- How does input validation on Workflows work? Is there input validation at all?
- Signals and Queries look interesting but a bit convoluted—need to explore those more.
- Should you use Temporal for your entire application state or just Workflow-based state? If not, how do the two combine?

# Pricing

Temporal uses a per-Action-based pricing model:

![](./actionpricing.png)

One million "actions" costs $25. This seems quite expensive since everything results in an action. For comparison, [Convex](https://www.convex.dev/pricing) charges $25 per month for 25 million function calls, and [Cloudflare](https://developers.cloudflare.com/workers/platform/pricing/) charges $0.3 per 1 million function calls.

However, I'd need to make a real app to get a proper sense of how expensive Temporal would be in practice.

# Thinking in Convex

Speaking of Convex, I can't help but think about how I might implement a "Workflow-like" structure on Convex.

I've been considering building an Event-Sourced application on Convex for a while, but I think a simpler structure via State Machines might achieve most of what you want... 🤔

Stay tuned—I may well tinker with this in the future!



As is [rapidly becoming a trend](https://mikecann.blog/posts/tinkering-with-spacetime-db) in my tinkering adventures, today's post chronicles a bit of a failed adventure. But that's okay, it's always good to explore and learn, particularly in the rapidly evolving field of AI.

# LazyAI

There are a lot of these text-to-app startups around at the moment. One of the most famous is [Devin from Cognition.ai](https://www.cognition.ai/introducing-devin). That one is not available yet, but I wanted to explore another one that has been getting some buzz lately: [Lazy.ai](https://www.getlazy.ai/).

![](./tagline.png)

Its tagline is "Idea to software 10X faster." Let's see how it does.

# The Project

As I am [a massive Convex.dev fan](https://mikecann.blog/posts/im-now-a-convex-developer-advocate), I knew it had to be something Convex related. I felt like this could be tough for the AI, as Convex is not as well-known yet, and so the LLM may not have as much pre-trained knowledge.

Something I had wanted to experiment with for a while was some sort of visualization for Convex database schemas so you can see the relationship between tables and types. I was imagining some sort of class diagram:

![](./classdiagram.png)

Or even better, a program that can generate the diagram dynamically such as [TSDiagram](https://tsdiagram.com/).

![](./tsdiagram.png)

# LazyAI Conversation

So, with that, I entered the following prompt into LazyAI:

> I would like to build an app that takes in a convex database schema and generates a Directed Acyclic Graph of the relationships between the tables and types.

It came back with a helpful plan of attack.

![](./lazy1.png)

I felt like maybe I was being a bit tough on it, so I gave it a bit more info:

> My project is TypeScript based. I want to be able to run this tool as a TypeScript script and have it analyze the convex.dev database schema that I give it. For example:

```ts
const schema = defineSchema({
  messages: defineTable({
    authorId: v.id("users"),
    body: v.string(),
    likes: v.int64(),
    replies: v.int64(),
    isReplyToMessageId: v.optional(v.id("messages")),
    imageId: v.optional(v.string()),
  }).searchIndex("search_by_body", {
    searchField: "body",
    filterFields: ["authorId"],
  }),

  users: defineTable({
    handle: v.string(),
    name: v.string(),
    pictureUrl: v.union(v.string(), v.null()),
    tokenIdentifier: v.string(),
  })
    .index("by_token", ["tokenIdentifier"])
    .index("by_handle", ["handle"]),

  lists: defineTable(
    v.union(
      v.object({
        kind: v.literal("all_messages"),
        ownerId: v.id("users"),
        name: v.string(),
      }),
      v.object({
        kind: v.literal("search"),
        ownerId: v.id("users"),
        name: v.string(),
        query: v.string(),
        includeReplies: v.boolean(),
      })
    )
  ).index("by_ownerId", ["ownerId"]),

  likes: defineTable({
    likerId: v.id("users"),
    messageId: v.id("messages"),
  })
    .index("by_likerId", ["likerId"])
    .index("by_messageId", ["messageId"])
    .index("by_messageId_likerId", ["messageId", "likerId"]),
});
```

The response it came back with was:

![](./lazy2.png)

I felt like that was actually a reasonable plan and told it to go ahead.

Now, the way the app works is that it's split into three parts: Test, Code, and Env Secrets.

![](./lazyparts.png)

So in theory, I should be able to test the output it generates. Well, this is where things unfortunately start to go off the rails a bit because to test (even your first project), you need to sign up for a pro membership.

![](./lazypro.png)

Which I would have been happy to do if I felt like the tool was going to give me some value, but the pricing scared me off.

![](./lazypricing.png)

$39.89 per month just to see if the code works and then usage costs on top of that?! No chance I was going to pay that just to check it out.

So after a little bit more back and forth with the AI, I decided to take a look at the code and copy it into VSCode to see if I could get it to run myself.

Well, unfortunately, I was to be disappointed again.

![](./lazyerrors.png)

There was no chance this code would compile or even run. There are TypeScript type errors all over the place and the imports wouldn't work from the wrong packages, etc.

I tried to give it the benefit of the doubt though and fixed the compile errors and import errors and tried to run it. Sure enough, no dice.

![](./lazyrunerror.png)

# LazyAI Conclusion

It's not ready yet. I think the task I gave it was quite tricky and maybe I was asking too much, but I think I was willing to give it the benefit of the doubt until I saw the pricing for it. Then my sympathy evaporated.

Fortunately, there are a lot of these "AI Project Generation" projects out there and as I still had a bit of time left to tinker, I thought I would give another one a go.

# Pythagora.ai

https://www.pythagora.ai/

![](./pythagora.png)

This one touts itself as a "dev tool that builds apps from scratch by talking to you." Sounds great, let's give it a shot.

This one works a little differently. It is a VSCode extension that you need to install [from the store](https://marketplace.visualstudio.com/items?itemName=PythagoraTechnologies.gpt-pilot-vs-code&ssr=false#review-details). So I headed over there and immediately was discouraged to see that it was only 3 stars out of 5. I was not off to a good start.

![](./pyreviews.png)

After a bit of a confusing setup (you have to first install another tool, gpt-pilot apparently) I had it working.

![](./pyworking.png)

(just a side note, having an AI running on my local system from an unknown developer with full access makes me kind of nervous.)

This time I was a little more generous with the initial prompt:

> I would like to build a Node.js TypeScript app that takes in a convex.dev database schema and generates a graph of the relationships between the tables and types.
>
> I want to be able to run this tool as a TypeScript script inside of my convex project and have it analyze the convex.dev database schema that I give it.
>
> I want the output to be an image that looks like something like a class diagram from classic UML.
>
> This is an example convex schema:

```ts
const schema = defineSchema({
  messages: defineTable({
    authorId: v.id("users"),
    body: v.string(),
    likes: v.int64(),
    replies: v.int64(),
    isReplyToMessageId: v.optional(v.id("messages")),
    imageId: v.optional(v.string()),
  }).searchIndex("search_by_body", {
    searchField: "body",
    filterFields: ["authorId"],
  }),

  users: defineTable({
    handle: v.string(),
    name: v.string(),
    pictureUrl: v.union(v.string(), v.null()),
    tokenIdentifier: v.string(),
  })
    .index("by_token", ["tokenIdentifier"])
    .index("by_handle", ["handle"]),

  lists: defineTable(
    v.union(
      v.object({
        kind: v.literal("all_messages"),
        ownerId: v.id("users"),
        name: v.string(),
      }),
      v.object({
        kind: v.literal("search"),
        ownerId: v.id("users"),
        name: v.string(),
        query: v.string(),
        includeReplies: v.boolean(),
      })
    )
  ).index("by_ownerId", ["ownerId"]),

  likes: defineTable({
    likerId: v.id("users"),
    messageId: v.id("messages"),
  })
    .index("by_likerId", ["likerId"])
    .index("by_messageId", ["messageId"])
    .index("by_messageId_likerId", ["messageId", "likerId"]),
});
```

Then followed a plan that the AI came up with and interestingly, it asks me before it runs commands:

![](./pycommands.png)

Ughh, it seems like it gets stuck quite easily though when something isn't installed that it needs:

[

![](./pydot.png)](./pydot.png)

I didn't want to install anything else right now, so I just clicked past it.

It seemed to continue just fine, then began installing a bunch of Node packages and setting up a TypeScript project for me, which was kind of nice.

It stops every now and then to check with you to make sure everything is looking good, which is nice.

![](./pyreview.png)

It started to generate some TypeScript. It looked like there were some errors, but I was starting to feel confident that it might be able to debug its way out of it. Then I hit:

![](./pyfreetrialover.png)

Fortunately, their pricing is much more reasonable than LazyAI and even lets you use your own OpenAI keys!

![](./pyapikey.png)

I was keen to see where this went, so I decided to spend $10 (and tip the developer) to find out.

I put together this little video to give you a sense of what the process feels like:

<iframe width="853" height="480" src="https://www.youtube.com/embed/YmhPVFl-dVE" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Unfortunately, as can be seen from the video, the code is very broken and would not run even if I followed the instructions they gave me.

# Conclusion

Unfortunately, I had now run out of time to tinker with this any more, but I felt like it was possible that I could have gotten to something usable with Pythagora, which I didn't think was likely with LazyAI.

Could I build a graph visualization faster myself? Probably, but that's not the point. The point is that I can potentially hand this task off to my "junior developer" and have them work on it for me while I do something else, like touching grass for example (ewww).

So my takeaway is that it's not ready yet, at least these two aren't ready.

LazyAI has a long way to go, but I think I do prefer the running code in an online sandbox process over Pythagora's run the code on your local system approach, which quite frankly makes me nervous.

Running in a sandbox also means that the AI doesn't have to continually pause to ask me if it's okay to run a command (like Pythagora does), as it can simply run it without fear of damaging the underlying operating system.

I am looking forward to future versions of App Generators. In particular, [Devin](https://www.cognition.ai/introducing-devin) looks very promising and also runs in a sandbox-like environment. I added myself to their waitlist, so stay tuned. I may well revisit this in a future blog post!



Phew its been a long long journey for our little game [BattleTabs](https://battletabs.com/) ([watch a summary video here](https://mikecann.blog/posts/battletabs-in-7-minutes)) but finally we are entering the phase that is a culmination of all the hard work, creativity, and perseverance: [the soft launch!](https://play.google.com/store/apps/details?id=io.battletabs)

# The Trailer

Before I start woffling on and loose you, heres some eye-candy:

<iframe width="853" height="480" src="https://www.youtube.com/embed/wGgJjEvNgt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Its almost not cringe! :D

# What Is a Soft Launch?

For those who might be wondering, a "soft launch" is a strategy used in the gaming industry where a new game is released to a limited audience or in specific regions before its official worldwide release.

In our case we are launching today [just on Android and just in the Philippines](https://play.google.com/store/apps/details?id=io.battletabs).

This phase allows us to gather valuable feedback, fine-tune gameplay, and ensure that everything is as perfect as it can be.

It’s a critical step in ensuring that BattleTabs delivers the best possible experience to its players.

# The Last 18 months

For the past year and half we have been beavering away at breakneck pace converting our web and chrome extension game over to work on mobile. Along the way we have added many new features and refined many of the existing.

Many parts of the game were changed in particular the UI has undergone a massive upgrade to make sure that the game works on small handheld devices in portrait mode.

That last part is worth underscoring. BattleTabs works in both Landscape (on the web or tablets) and Portrait (on mobile). This was no small task but I think we have pulled it off and im pretty proud of it.

![](./layout_1.png)

![](./layout_2.png)

Its also worth noting that, as often get asked this, the game is entirely Typescript using React for the UI and BabylonJS for the 3D rendering. I don't think there are too many games out there that have taken this path not sure if that is a good thing or not but it certainly sets us apart.

# What’s Next?

During the soft launch, we’ll be closely monitoring every aspect of the game, from user experience to the technical performance, particularly server performance.

We still have some important changes we want to make before final release such as a new First Time User Experience (tutorial) as that is super critical for players who are quick to dismiss something if they are confused or frustrated.

Post launch we have a number of exciting improvements and features planned but I might have to leave that for a future post :)

# How You Can Get Involved

Right now the game is only available on Android and in the Philippines but we are going to slowly be expanding the number of countries and platforms we are on.

We invite you to join us in this exciting phase. The best place to stay up to date is to [join the 16,000 other people on our Discord Server](https://discord.gg/battletabs). We post to the [#announcements](https://discord.com/channels/770955477215084554/770963532455477310) channel regularly about everything thats going on with the game.

Your feedback will be invaluable in shaping the final version of BattleTabs. Whether it’s a bug report, a suggestion, or even just sharing your favorite moments, every bit of communication helps us make BattleTabs better for everyone!

# Wrapping Up

The soft launch is just the beginning of the next chapter in the BattleTabs saga.

We’re incredibly proud of what we’ve achieved so far and even more excited about what the future holds.

Stay tuned, and [let the battles begin](https://play.google.com/store/apps/details?id=io.battletabs)!



Wow can you believe it, exactly 10 years to the day since I took the almighty leap into the unknown and traveled half way around the work from the UK to Western Australia to start my new life with a girl I had met just a few short months ago.

I thought I would take this opportunity to look back and reflect on the last 10 years of my life in Australia.

So much has happened in the intervening years that its hard to know where to start so I think ill do something a bit different and do one of those "toplist" kind of posts..

# 10 Major Events in 10 Years

## 1. Arrived in Australia

This [very blog documented](https://mikecann.blog/posts/mike-goes-down-under) the day that I arrived in Australia 3652.5 days ago.

I remember being in a sort of shock for several days when I first arrived. I had heard so much about Australia from Kelsie on our travels so when I finally arrived it was hard to wrap my head around.

I remember in the first week seeing a scorpion scuttle across the living room floor and being assured that I had nothing to worry about.

![](./scorpion.jpg)

I remember being stunned that there were beautiful parrots and cockatoos just flying around all the time and no one batting an eye lid at it.

I remember everything smelling so different, a rich dry-eucalyptus smell that I now associate with home.

I remember many long days working by myself on [Mr Nibbles](https://mikecann.blog/posts/indie-game-project-management-with-trello) and other projects, often not speaking to anyone all day then being super excited when Kelsie returned home that "I just wouldn't shut up".

I remember stressing about visa's and how I was going to stay in Australia, so thats why after a year I..

## 2. Moved to Perth

I joined just about the only gaming company that was hiring in Perth at the time called [TapSlots](https://www.linkedin.com/company/tap-slots/). They were kind enough to sponsor my visa through the early stages of 456 all the way up to the Permenant Residency (one step before Citizenship).

I was super happy there, I found the work rewarding and the people excellent. I made some life long friends there.

A year or so after I moved, Kelsie joined me and we began renting a place in the beautiful South Perth.

I was living the dream, my daily commute involved a quick stroll past the zoo and the sounds of the howler monkeys, through a park down to the ferry. A short ferry across the water where dolphins often could be seen in the water, then another walk through a park before I arrived at the office. It really was a great life.

![](./sp.jpg)

But before long the old traveling itch started again and I wanted to explore a bit more before I became too comfortable, so thats why in 2017 Kelsie and I..

## 3. Traveled Australia

Kelsie and I loved the idea of exploring more of the vast country so we went out and bought a car and a camper and [set off for 5 months](https://mikecann.blog/posts/and-off-we-go).

![](./camper.jpg)

We went to so many places, had so many adventures it would be impossible to recount them here. My laptop still bears the scars from my cleptomania obsession of collecting stickers from each major place we visited.

![](./laptop.jpg)

Before we left I had purchased a brand new (at the time) consumer drone called the DJI Mavic Pro. This was so early that most places didnt have any rules or policies around drones so I took full advantage of that capturing thousands of incredible photos and videos which I attempted to sell over at: https://aboveunder.com.au/

![](./aboveunder.jpg)

Before long tho it was time to return back to Perth and real life but that was okay because by this point I had..

## 4. Made new Friends

It seems like the older you get the harder it is to make friends. I find making friends really important to general happiness and really helps to make a place feel more like home.

I have joined numerous clubs, groups and meetups in my years here to both try to grow my network and make new friends. It takes real effort but its most certainly worth doing.

One of the friends I made along the way caused me to..

## 5. Learnt a new Sport

I technically had actually tried Kiteboarding already while I as on my South American trip but it wasnt until my (at the time) neighbor Stan convinced me to pick it up again in Australia.

![](./kiting.jpg)

Im so glad he did as I have had an awesome time kiting around, making the most of the vast west australian coastline and prodigious and consistent wind.

Other than the wind WA also has a lot of coastline so I have over the years gotten more and more into Freediving. If you are interested I have made a number of videos about my dives here: https://www.youtube.com/watch?v=J9IhSAuuFDY&list=PLdFGn0KeUCYer1aL7cEzpcDIg51vycrio

With myself settling into Aussie life it was just about time that I..

## 6. Became a Citizen

In January of 2020 [I officially became](https://mikecann.blog/posts/a-personal-update) a citizen of Australia!

![](./citizenship.jpg)

It was a long windy 6-year road of different visas and constant worry but I finally made it.

Im told tho that im not truly an Aussie until I have caught a Barra.. so I have that to aim for still.

Around the same time I also..

## 7. Started a Company

I have always been entrepreneurial so when my co-founder Brandon raised the possibility of securing some funding to work on a new SaaS business I [lept at the opportunity](https://mikecann.blog/posts/farewell-bamboo-hello-markd).

That has been an incredibly difficult but rewarding adventure that is still ongoing with [Gangbusters](https://mikecann.blog/posts/bandai-namco-invests-in-gangbusters) as we work on brining our premier title BattleTabs to mobile.

![](./battletabs.png)

If running your own business was not hard enough I also recently...

## 8. Became a Father

![](./joshie.png)

Among all the milestones I've reached, becoming a father was by far the most transformational. The birth of my son, Joshie, introduced a profound shift in my world view, priorities, and daily routines.

Suddenly, the adventures had a new co-star, and my projects had a deeper purpose. Watching him grow, explore, and start to form his own relationship with Australia has been an incredible journey.

Parenthood is a mix of joy, worry, sleepless nights, and unparalleled love. It has grounded me in ways I never anticipated, anchoring my adventurous spirit with a new sense of responsibility and wonder.

It often goes hand in hand but it was around this time that I also..

## 9. Got Married

![](./wedding.jpg)

Kelsie and I had shared countless adventures, faced life's ups and downs, and grown together in ways neither of us could have imagined when we first met.

It was only fitting that we decided to solidify our journey together by tying the knot. Our wedding was exactly as I wanted it, small and intimate with just family nearby.

My sister even flew all the way out from the UK as a suprise to join us for the big day :D

Our honeymoon was exactly as I liked it, functional, as you see we had just..

## 10. Bought a house

After years of moving, traveling, and exploring, Kelsie and I decided it was time to lay down more permanent roots. [Buying a house in Busselton](https://mikecann.blog/posts/moving-home-and-startlink) was a milestone that symbolized not just a physical home, but the establishment of our family's base camp.

<iframe width="853" height="480" src="https://www.youtube.com/embed/T4kaZX1ZzzM" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

We only moved in here less than 6 months ago so it still a bit early to call it home but there are definitely lots of signs that im going to enjoy it here for hopefully another 10 years or more!



Due to my current focus on transitioning [BattleTabs](https://mikecann.blog/posts/battletabs-in-7-minutes) to mobile, my time for other activities is quite limited. However, I feel compelled to discuss a concerning practice I recently encountered on Meetup.com, known as a dark pattern.

As a longtime user, I've tolerated Meetup.com's mediocre app experience because it was a key resource for discovering tech-related meetups in Perth. When I relocated to Busselton in September, I decided to initiate my own tech meetup, [South West Devs](https://www.meetup.com/south-west-devs/). Unfortunately, Meetup.com does not offer a free tier for group hosts, forcing me to commit to a six-month subscription. My hope was that this period would be sufficient to gauge interest.

Regrettably, interest was lacking, partly because many local developers were unfamiliar with Meetup.com, not even considering it as a resource.

This brings me to the crux of my post. Attempting to cancel my subscription on Meetup.com I was presented with this screen:

![](./dark-meetup.png)

So you can see the problem, I am forced to talk to one of their sales team before I am able to cancel the subscription.

From a business standpoint, the rationale behind such a practice is understandable. It serves to re-engage users or propose alternative plans. Nonetheless, this approach feels inherently manipulative and, frankly, kinda scummy.

The frustration this experience induces is, to my knowledge, universally disliked.

A more user-friendly alternative would involve a simple cancellation process, followed by a post-cancellation email or an occasional reminder within the app itself, offering re-engagement opportunities in a less intrusive manner.

Regrettably, Meetup.com is not alone in employing this tactic. Upon sharing my experience within various groups, I discovered that other companies, including Xara and Adobe, engage in similar practices.

For now, I must redirect my attention to mobile game development, but I felt this issue warranted a brief discussion.



As we close in on our year-long goal of bringing [BattleTabs](https://mikecann.blog/posts/battletabs-in-7-minutes) to mobile the time has come for some long-overdue performance improvements.

I have been putting it off as performance improvements are always a bit of a project-black-hole sucking all time, space and energy. Fortunately I was able to escape this relativistic singularly and come out of the other side with some sizeable performance gains and interesting insights in tuning a BabylonJS game that I thought I would share here.

# Before Getting Started

First thing's first. When attempting any sort of performance optimization pass its critical to lay a good foundation down first. We need to first get some good diagnostics info from the game. This allows us to see which parts of our game are causing the performance drops.

Fortunately BabylonJS has some pretty good metrics built into the engine that you can checkout in the inspector:

![](./babylon-inspector-stats.png)

It unfortunately doesnt show me those numbers as a percentage of frametime and it also takes up quite a bit of the screen. So I decided to make a little stats counter in the top left that shows me the critical numbers and their percentage of the total frame.

![](./my-stats-counter.png)

With this in place I was able to start to narrow down what I think I can spend some time looking to a little more deeply.

One other quick tool I like to use is Chrome's CPU throttling feature:

![](./chrome-cpu-throttle.png)

I develop the game on a powerful desktop PC which is likely more powerful than many of our users' devices. So using the CPU throttle it lets me simulate what its like for them a little more closely.

Arguably this slants things more towards a CPU bottleneck rather than a GPU bottleneck but lets just ignore that for now, ill discuss it more a bit later.

# Problem Areas Identified

I quickly noticed that there were three large contributors towards the total frame time: "Mesh Selection", "Animations" and "Render".

## Render

Render is a pretty obvious one, it's the time for the GPU to get the pixels on the screen. There was likely some stuff I could do here to speed things up but I decided not to tackle this one for now as it was likely going to involve some quality tradeoffs or digging through shader code which I would rather not do just yet.

## Mesh Selection

I wasn't too sure what this meant but after reading [Babylon's performance documentation](https://doc.babylonjs.com/features/featuresDeepDive/scene/optimize_your_scene#freezing-the-active-meshes) I had a better idea. As far as I understood it, it is the time taken for Babylon to choose which meshes it should send to the GPU for rendering. This is dependant mainly whether the mesh is within the camera view area (Frustrum).

BattleTabs doesn't really have many meshes and most of them are on screen most of the time so I wasnt really sure why exactly this number was so high.

I decided to [raise this issue on the BabylonJS forums](https://forum.babylonjs.com/t/why-is-mesh-selection-time-so-high-and-how-to-reduce-it/47069).

![](./forum-post.png)

After a bit more experimentation I added a "FREEZE" button to my stats which called the `scene.freezeActiveMeshes` which was also mentioned in the Babylon performance docs. This had a dramatic effect on the Mesh Selection time, effectively removing it.

<iframe width="100%" height="480" src="https://www.youtube.com/embed/HtVbtbm8I5w" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

As far as I understand it, freezing active meshes means that whatever is currently "active" (viewable within the Camera Frustrum) will ALWAYS be incuded in the list of meshes to be rendered.

This is great as it removes all that frustum culling time but it has a nasty side effect. If anything is added or removed from the scene, or its visibility changes, then it wont be reflected on the screen. This is because the mesh selection is "frozen". Animations also froze (im not exactly sure why this is).

All of these issues were possible to work around but it was going to be a bit of a pain going through the code with a fine-toothed-comb looking for places that might cause problems. Not only that it was likely to cause future bugs as its a nasty cross-cutting concern you can often forget.

I decided to sit on this one for now and look into the final large chunk of frame time I had yet to explore.

## Animations

Animations were taking over a third of frame time, this was a surprise to me. We only have 4-8 ships animating at any one time and [I was under the impression that babylon's skeleton animations were computed on the GPU](https://doc.babylonjs.com/features/featuresDeepDive/mesh/bonesSkeletons#picking-a-mesh-attached-to-a-skeleton) so there was no reason why they should be so slow.

I had read others having issues with this in Babylon and they suggested [that one could bake their animations](https://doc.babylonjs.com/features/featuresDeepDive/animation/baked_texture_animations) into whats know as "Vertex Animation Textures (VAT)".

These are basically pre-computed positions for each (bone? vertex?) each frame. These positions are stored in a giant array then loaded a runtime, passed to the GPU then used in a shader to update the positions of each bone / vertex each frame.

Doing this was going to be a bit of a pain as it meant we would have to add an additional design-time step of producing these VATs for each ship (and skin) and keeping them updated any time a change is made. It also precluded us from using blended animations, but we werent using those so I wasnt too bothered about that.

As with all performance work I did the bare minium of absolutely the nastiest-hackyiest code I could to test the theory and was surprised that VATs actually worked:

<iframe width="100%" height="480" src="https://www.youtube.com/embed/gIfKTqxTS4o" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

VATs meant slower game loads as each animation for each ship now had to load in another texture. It likely also meant more VRAM usage as these new textures had to be sent to the GPU but I suspected this tradeoff was going to be an acceptable one for BattleTabs so I ploughed ahead.

# Putting it all together

With the theories in place and experiments conducted I reset my git branch and started again, this time properly coding it up. This was an arduous process as it meant handling lots of edge-cases that the experiments didn't cater.

Eventually I thought I had it all looking pretty good. To make sure I had the problem solved I put together a torture test scene:

<iframe width="100%" height="480" src="https://www.youtube.com/embed/xJ8DGv_NzAE" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

The above shows that even with 100 ships animating I wasnt spending any time in Mesh Selection or Animations, huzzah!

# Conclusion

So in total I saw about a 70-100% performance improvement once I set my CPU throttle to 6x slowdown. This is quite a nice gain and makes me question why BabylonJS as an engine doesnt put more effort into reducing CPU load as JS is inherantly single-threaded and GPUs are so powerful these days it seems to me that the CPU is almost always going to be the bottleneck.

When I quizzed Evengi_Popov one of Babylons core-contributors he agreed:

![](./jedi.png)

So who knows, perhaps we will see some of this stuff built into the engine in the future.

For now im happy with the results, theres always more that could be done but I think its taken a little bit of pressure off of many of our users' poor devices and given them a better gaming experience.

Now onto the last few things before the mobile release!


2024

Archive

2024

Archive

2024

Archive