
For the [longest time](https://mikecann.blog/posts/battletabs-6-months-later), Gangbusters has been a Discord-focused company. We use it as our primary method of interacting with our community, receiving bug reports, and gathering feature suggestions.

So when [we heard about](https://venturebeat.com/games/discord-launches-activities-with-light-games-and-a-new-app-store/) Discord's intention to make games playable within the app itself, we knew it was a strong opportunity for our game [BattleTabs](https://battletabs.com/).

Discord is a game-focused social platform where many players are hanging out with friends and looking for something to play together. Having our social multiplayer game embedded right there was a match made in heaven.

![](./battletabs-in-the-app-drawer.png)

In addition to reducing the discovery friction for BattleTabs, Discord also offers a number of other compelling advantages. One of these is the ability to directly message players (we had early access to this feature). We currently use it as another Push Notification channel, but it’s so much more powerful than that—we’ve barely scratched the surface of what’s possible.

![](./discord-dms.png)

Super-fast and friction-free payments are another strong benefit of using Discord, particularly over the web platform where the friction for paying for IAPs or subscriptions makes it super challenging to monetize games like ours. Discord even provides an app "store" where you can see all products that the game offers and purchase them directly without even opening the game.

![](./showing-the-store.png)

There are many other strong reasons to be excited about the Discord platform for games, especially when you read about the [incredible success](https://mojiworks.com/news/why-were-all-in-on-discord-activities) stories [from](https://a16z.com/discord-activities-social-gaming/) other [developers](https://discord.com/build-case-studies/frvr). You can see why we were keen to get on there.

Fast forward a year or two, and after much hard work by [Brandon](https://www.gangbusters.io/about), we found ourselves in the inner circle as one of Discord’s Developer Partners. The process of getting BattleTabs on Discord had begun.

The first step was a "geo test" or soft launch, focusing on Canada. This was an opportunity for Discord to see how the game performed before deciding whether to move forward with a global launch.

![](./happy-meme.png)

After two weeks of the game being available in Canada, we sat down with Discord again, and they gave us the very exciting news that the game had done exceedingly well. As a result, they wanted to take us to a global launch.

We scheduled the official global launch for 10 a.m. my time (Western Australia) on Tuesday, September 10th.

# Tuesday

After a brief call with Discord, we went live! We immediately started to see our usual traffic numbers get obliterated.

![](./initial-spike-graph.png)

And this is where the fun started. To explain the issues we were soon to face, I first have to explain the architecture for BattleTabs a bit.

![](./architecture1.png)

You can see from the above diagram that we have two different kinds of servers. Users connect to the "Web" servers via WebSockets, these scale out horizontally and generally aren’t a problem.

The other kind is the "worker" server. This is a singleton server and was serving many roles (foreshadowing). Some of its responsibilities include:

+ Listening for "app events" and performing various tasks like updating challenge progression, sending push notifications, awarding medals, updating league positions, etc.
+ Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.
+ Listening to database events and performing logic, such as alerting web servers to update subscribed users when their currency values change (due to rewards or other events).
+ Handling tasks like cheat detection, server metrics reporting, Discord server bot management, Discord live reporting, and more.

I was aware that the worker was a bottleneck in the architecture, but up to now, it hadn’t been an issue. The worker usually sat at a very low CPU and memory level.

Regardless, I beefed the worker up to the largest size that [our hosting supported](https://fly.io/docs/about/pricing/).

![](./worker-sitting-okay.png)

It was sitting at about 50% utilization for most of the day, which was a bit high but still within the bounds of comfort. So I went to bed at about 9 p.m., feeling good about the launch so far... little did I know what was coming.

![](./bed-time.png)

# Wednesday

I woke up to an absolute nightmare. The servers had clearly been having a bad time while I was asleep. I hopped on [our Discord Server](https://discord.gg/fTdxf4y9) and saw that many players were complaining about the game not working correctly.

I took a look at the app event queue and, to my horror, saw that there were over 3 million events in the queue waiting to be processed.

![](./3m-queued-events.png)

The number was decreasing, but even after much tweaking of the various settings, I calculated it would take 26 hours for my poor single worker server to clear them all down, on top of everything else it was dealing with.

Before I could make any architectural changes to the servers, I needed to clear this queue to avoid losing data or causing more problems by handling events out of order.

![](./slack-to-brandon.png)

So although the game was in a semi-broken state, I went to bed thinking that at least the queue would be mostly cleared by morning, allowing me to focus on fixing everything properly.

# Thursday

![](./4m.png)

... well sh*t ... this isn’t good...

It turns out that while I was sleeping, we had some "good" news from our friends at Discord...

![](./moving-you-up-to-rank-4.png)

Ah, crap... I knew at this point I had to take action. This queue wasn’t going to clear itself.

I asked Discord to drop us back down the list. As painful as it was to request LESS traffic, I needed to buy us some breathing room.

The next step was to parallelize that event queue processing.

So I created a new kind of server called an "eventsWorker." This server took much of the event processing work from the worker, and critically, it could be parallelized.

![](./architecture2.png)

There was a risk of out-of-order event processing, but I carefully examined each event and realized there wasn’t much event-order dependence, particularly between users. Anything that was dependent, I could likely fix later.

This realization gave me the confidence to scale up the number of eventsWorker instances. However, this put a huge strain on the database, so I scaled it up to the largest size that Fly would allow.

![](./16-performance-core-db-pricing.png)

This allowed the eventsWorkers to scale to 15 instances, ripping through the queue at over 1 million events per hour. The entire queue should be cleared by late afternoon.

Based on that assumption, I got to work preparing the update that would go out once the queue was resolved.

I decided to reduce as many places as possible that depended on app events, thus reducing pressure on the background workers.

After thinking it over, I realized that most events could be processed "locally" instead of being sent to the background.

To explain what I mean, let's take an example event, `matchFinished`. (By the way, if you're curious, [here are all the app events](https://gist.github.com/mikecann/c7ad434ffcf94eb4e5c0d0b30f7967e1) in the game.) Here's where this event is used:

+ BattleEventsService - to send push notifications to users about the battle outcome
+ BattleWebhooksService - to send API messages to our Tournaments bot for advancing tournaments
+ BattleTabsLiveService - to update the #battletabs-live channel in Discord
+ HighScoresService - to update highscore stats
+ MedalAwardingService - to potentially award a user a new medal
+ UserChatDetectionService - to check for abuse if a user plays the same person multiple times
+ UserStatsService - to update a user’s stats
+ ChallengeUpdatingService - to progress any challenges the user might be on
+ TVChannelsService - to update any playing BattleTabsTV instances

Each of these is a totally different feature in the game and independent of each other, which is why they are handled as events rather than inline function calls in the first place.

When a battle was finished, I would publish a `matchFinished` event to the queue ([managed by pgboss](https://github.com/timgit/pg-boss)). The event would then be picked up by a background worker to handle all that work mentioned above.

I realized, though, that instead of sending it to the background, I could simply emit a "server event" in addition to the "app event". These server events would only be listenable by services running on that one server.

This shifted the computational burden from the background workers to the web workers, which could easily scale horizontally.

It seems obvious now, but hindsight is 20/20. I had been so focused on client-side improvements that I missed these obvious server-side optimizations. To be fair, it was only the increased traffic that necessitated the change.

So, shortly after 4 p.m., the events were all cleared. I scaled the eventsWorkers down from 15 to 3, then pushed my "server events" change.

This immediately reduced worker load but increased web server load as expected. Web servers could [auto-scale](https://fly.io/docs/reference/autoscaling/) up however so I wasn't worried.

I went to bed feeling pretty good about things. I did tell Brandon (who is based in the UK) to call me if the servers went down.

I soon fell asleep, fairly confident I would get a good night’s rest...

# Friday

3:14 a.m. – I got a call from Brandon letting me know that the game was down.

I jumped out of bed and hopped on my PC. Sure enough, things were not looking good and hadn’t been for a couple of hours.

![](./friday-morning-graph.png)

This time, it wasn’t immediately obvious what was wrong. It appeared that the worker process was running out of memory. I attempted some desperate fixes but wasn’t sure about the underlying reason.

![](./desperate-commit-log.png)

By 4:30 a.m. my brain really wasn’t working properly. I felt like the game was stable, so I decided to get an hour of sleep.

### Later on Friday

Awake and rested, I realized what the problem might be.

As previously mentioned, one of the worker server’s responsibilities was:

> Polling the database for matchmaking, AI turn-taking, daily shop reward rotations, etc.

For example, matchmaking works by requesting a list of open battle proposals every few seconds, then looping over that list and creating battles as needed.

Turn timeouts, AI turn-taking, and a bunch of other systems worked similarly.

The issue was that this polling was effectively another kind of queue. If the number of rows returned from the database during each poll was too small, that queue would build up.

Unfortunately, I had no metrics around these polling "queues," so my first task was to get some visibility into them.

![](./poll-queue-growth-chart.png)

Sure enough, those queues had grown to huge levels.

Much like my app events problem earlier, I knew the solution would be parallelization.

So, I carefully moved the polling code over to the eventsWorker instances. I was very nervous about this, as I knew that allowing multiple servers to poll the database simultaneously risked race conditions and other problems due to polling overlaps.

My solution was to carefully place [distributed locks](https://github.com/mike-marcacci/node-redlock) to ensure that two different workers couldn’t work on the same item from a polling queue simultaneously.

This solution worked well, and I soon had the polling queues back under control.

![](./polling-under-control-graph.png)

That’s when disaster struck...

![](./slack-to-brandon-wild-db-growth.png)

I had been keeping a close eye on our database’s growth, knowing that if we ran out of space, it would put a hard stop to the game.

Well, it turns out that with all the chaos of the past 12 hours, I forgot to check the DB size, and it had grown wildly. It finally ran out of space and started refusing connections.

Fortunately, I was looking at the logs when I saw this message go by:

![](./jobs-cannot-archive-log-message.png)

It turns out that pgboss, our app events queuing and jobs system, was timing out when trying to archive completed jobs. I ran a quick query against the database and saw over 7 million event rows waiting to be archived, with another 3 million waiting to be deleted.

It seems like pgboss wasn’t designed to handle this scenario very well and was erroring out during its archival and deletion routines.

My solution was to manually delete all 10 million rows from the database immediately. This instantly freed up 70 GB of space. I then ran a [Postgres VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html) to clear up any deleted tuples.

Instantly, the database came alive again, and disaster was averted.

Phew! It had been a long day, but I was finally feeling good about things. I went to bed certain that everything was all good.

# Saturday

I woke up, hopped on my PC, and to my dismay, saw that my "worker stopped" alert had triggered.

![](./slack-message-alert-worker-stopped.png)

Fortunately, it didn’t take long to track down that this time the worker hadn’t actually gone down. The alert had triggered because [Axiom](https://axiom.co/) (our log aggregation tool) had stopped receiving log messages. This happened because our "[log shipper](https://github.com/superfly/fly-log-shipper)" instance had died.

I still don’t know why or how this happened, but I quickly gave it a poke, and things were up and running again.

From the Grafana logs on the servers, it appeared that everything had been fine overnight, much to my relief.

I think I’m going to leave the day-by-day commentary here, but I believe you’ve got a good sense of the firefight I was putting up at this point.

# Lessons

So what lessons did I learn from all this?

1. **Be wary of queues**. Anytime you’re building something that could get backed up if you 10x or 100x your traffic, it’s wise to deal with it now or, at the very least, get some metrics around it so you can keep an eye on it.

2. **Single points of failure are ticking time bombs**. For the same reasons as above, be cautious of any places that cannot scale horizontally.

3. **Use Axiom**. It’s fast, cheap, and powerful. Log everything, then build graphs.

![](./axiom-pricing.png)

500GB on the free tier? That’s just insane.

To put it into perspective: despite all the spam we produced over the past week, we’ve amassed 54 million log lines, amounting to just 17GB of space, which is only 1GB compressed!

![](./axiom-dataset-logs.png)

4. **Load test if you can** before you expect a large increase in traffic. Even if you can’t easily hit the expected numbers on your local machine, it should at least hint at where bottlenecks might occur.

# Conclusions & Thanks

It’s still a bit early to talk about the results of all this traffic on the business, but things are going well.

I would like to thank the special people at Discord who took a chance on us and were very patient while we fought through these problems. :)



Im super excited to announce that [Gangbusters](https://www.gangbusters.io/) (the company that I am a co-founder and CTO of) has been joined on its journey by the gaming powerhouse [Bandai Namco](https://021fund.bn-ent.net/en/news/)!

We are super excited by this as it not only allows us to grow our team and ambitions it also validates our strategy and enhances our brand by having the creator / publisher of [such large IPs](https://en.wikipedia.org/wiki/List_of_Bandai_Namco_video_game_franchises) behind us:

![](./bandi-ips.jpg)

Bandai are joining a number of other investors as part of our early-seed round to help us realize our Web3 ambitions and take BattleTabs to mobile.

![](./other-investors.png)

Its been a long roller-coaster of a journey to reach this point but after 3 years of working in a very small team im really happy that we are finally able to grow a little bit and hire some more gang-members.

We still have a tough path ahead of us with some ambitious plans but I think we are starting to assemble a fantastic team that will allow us to get there. I personally cant wait to show our incredible community what we can do now!

If you are interested in partnering up or what we are working on, reach out to: mike@gangbusters.io or brandon@gangbusters.io.


For the past couple of weeks I have run a Sunday market stall for my side-project-business Above Under. Its my first time ever running a stall and its been a really interesting experience thus far.



# Location Location Location

My daily commute to work is one of the reasons I love living where I do. I walk past the zoo, through a park, down a beautiful tree-lined street then catch the ferry, often watching the dolphins feeding in the river, then through another park and im at work.

Each time I walk down that beautiful tree-lined street I though to myself; this would be a great place to have a market, and lo and behold I was walking down here the other week and saw that indeed they had just started to run markets here!

![](./mends-street.jpg)

Its a beautiful area, theres lots of foot traffic from the shops, cafes and the zoo so I thought to myself, why dont I have a go at running a market stall?

# Why?

Although Above Under has a website, Facebook, Instagram and all the other usual gamet of social media destinations I think that my best chance of showing people how spectacular some of the imagery is by actually putting it infront of them.

A market stall also lets me demonstrate the various material types that I can print on from various photo paper posts, to acrylic to solid metal Alumalux.

Although I had seen photography stalls at markets before (although I am the only photography stall at this market) I had never seen a stall specialize in drone photography.

# The Stall

So I spent a busy week rushing around buying a table, gazebo and a few picture stands and easals, not to mention ordering \$500+ worth of prints in various sizes and types as examples. There was some doubt if I would manage to get it all together in time but I just about managed it.

![](./the-stall.jpg)

# Lessons

The first Sunday running the stall, it was really very windy and despite bringing some weight and buckets full of sand, our gazebo very almost took off in the heavy winds. So for the second week I went out and invested in 4 heavy weights for each of the 4 legs of the gazebo.

Another thing I learnt is more about the business itself. I didnt really know what to expect sales wise on the day so I spent a bit of time setting up a square.com account and bought one of their contactless readers for \$50. In my mind I was expecting that people might want to buy xmas gifts and some people might want to order prints for later delivery so I would be prepared.

What actually seems to have been the case is that people walk over, have a look and a chat then take a business card. The majority of the keen people are looking to find an image that matches their decor, either for themselves or for a client if they are an interior decorator.

So my pitch has changed from focusing on the products available to promoting which images would look good with which color combinations.

One other thing I learnt was that some people didnt understand how I had made the images. Infact several people asked if they were paintings. So I decided to put my drone out on the table infront so people can see how I actually took the images.

![](./stall-with-drone.jpg)

However this has now led to people asking if my drone was for sale. Sigh!

# Experiment

Its all just an experiment at this stage. Im enjoying learning retail and sales lessons the hard way, by actually doing it. Its good experience which should follow through into selling things online too.

So if you are in the market for some high quality Australian drone photography come down to the Mends Street Markets in South Perth, ill be here each Sunday for the next few weeks at least!

![](./dancin.jpg)

![](./cookie-shots.jpg)

Its been a while since I have done a hackathon. I spent [half of the year](/posts/and-off-we-go/) camping my way around Australia so I guess thats to be expected. So when I saw that startup weekend was running again at the weekend I thought sure, why not.



# TLDR;

I had an idea, pitched it, gathered a team and validated an idea over the course of a weekend. I overcame some personal hurdles and came out the end with a potentially exciting business idea.

# What is Startup Weekend?

During Startup Weekend the objective is to go from idea to business over the course of just a couple of days. You form a team with total strangers and work together to validate your business and if possible build a prototype of your product.

The last time I did Startup Weekend was [back in 2015](/posts/startup-weekend-perth-2015-tuckr/) when I worked on a business called Tuckr that aimed to help people locate where those awesome food trucks are near you.

I remember ending the weekend that time happy but thinking that I probably wouldnt go again as it didn't really involve much technical work. It was more of an exercise in product validation than actually technical work and as I consider myself a highly technical person I didn't feel that useful.

# So why sign up again?

Often the only way to grow as a person is to get out of your comfort zone and do things that you arent good at or scare you.

I have always been scared of is standing up infront of people and presenting. Im much more comfortable sat infront of a computer in my own little word working on my own little problems.

So this time I decided I wanted to pitch an idea, form a team if people were interested and take it all the way through to the end and stand up infront of the judges and give a final pitch.

# The warm up

So friday rolls around, I rock up about 6pm, theres quite a few people there already. I have done this a few times before, I know what the deal is. I mingle around, grab a beer, chat to a few people.

Before I know it, its 7:30 and time for pitches. Theres about 90 people there and about 20 of those line up to take their turn to pitch their idea to those remaining in their chairs.

Im near the back of the queue because I want to see what other people are pitching just incase I want to chicken out and join one of them instead.

Theres some really interesting pitches from all kinds of different industries but none of them has grabbed my attention enough to stop me pitching, I guess the beer must have helped with the courage.

# The pitch

So I take the stage, I have 1 minute to explain my idea, I have practiced it a couple of times so I know that I should be able to do it in the time.

> Hi im Mike Cann, im a professional games developer but im also a blogger that has kept a blog going for over 10 years now.
>
> One of the constant problems with blogging is how do I get more people to read my posts?
>
> One solution to the problem can be seen on websites like Medium.com. At the bottom of each post is a "Reccomended for you" section that lists blog posts from other Medium.com authors.
>
> This circulates traffic between the authors on Medium.com and thus is good for everyone.
>
> But what if your blog isnt on Medium.com? Like mikecann.blog for example?
>
> I dont want to have to move all my posts over to Medium to take advantage of that post syndication feature.
>
> Fortunately there are alternatives. If you google this problem you will encounter Outbrain and Taboola.
>
> These multi-billion-dollar companies however are not targeted towards casual bloggers like me. You probably have seen them, they can be found at the bottom of websites like cnn.com and thewest.com.au. The recommended articles often have click-baity titles like "Details From The Set Of 'Bewitched' That'll Stun Fans" or "Mums, these workout hacks are just for you".
>
> This is not something I want next to my content.
>
> I want to build a click-sharing service for bloggers that shows relevant recommendations for my content without the clickbait.
>
> To validate this idea I need a designer and people with blogging or advertising experience.
>
> Thankyou.

![](./whatIwant.png)

# Team formation

With the pitch over I couldnt relax just yet. Now I had write a poster to attract others to join my team.

![](./theposter.jpg)

I tried to convey the basic business model in as simple a way as possible.

Well it seemed to work as I managed to gain 3 other members of the team. Ro, a fellow blogger and business woman, Alexis a writer with a bunch of blogger contacts and Andrew a designer.

So we had one developer (me), one designer to make pretty pictures and two business people with domain experience. IMO this is the perfect team!

![](./ro.jpg)
![](./andrew.jpg)
![](./alexis.jpg)

# Planning

The remainder of the evening was dedicated to introducing ourselfs and working out how we were going to work together, what our roles were and how we were going to communicate.

One thing that surprised me during this phase was that I had to re-pitch the idea to my team again before they fully understood what it was I wanted to build. Perhaps the idea was too complex? Perhaps I didnt explain it well enough first time? Either way I must have left some sort of impression as they joined me regardless.

# Trello

I like to use Trello for organizing a project during hackathons. Well I actually like using Trello for organizing just about any project but I particularly like it for hackathons because its really easy to get started, everyone can collaborate images, files, videos, whatever at the same time and most importantly its free!

![](./trelloscreeny.png)

# Saturday

The following day was all about validation. I had a few things I wanted to cover.

1. Has this been done before?
2. How do Outbrain and Taboola work, what are their positives and negatives?
3. Do bloggers even want this?
4. What are we going to call our business?

Note, none of those points actually involve any development work. From previous experience I knew how time constrained we were going to be over the weekend so I decided early on all we were going to do is validate the idea, not actually try to build anything.

With me acting as leader / coordinator my time was going to be very limited for development work.

So I spent the day giving Andrew our designer direction as he began the work of building the website we would send people to try to get feedback and thus judge whether it was something that bloggers wanted or not.

I also assisted Ro and Alexis with tasks that addressed the other points. One interesting thing that came out of Ro's research into Outbrain was the fact that you needed a credit card just to sign up and get started. This is definately something I want to avoid with our business.

# Naming

We spent a little bit of time on the Saturday morning coming up for a name with ourself. We had a little bit of back and forth, we wanted something that gave a hint that we were a blog syndication service but without too many syllable.

In the end settled on "Syndi.me", its not perfect as it could be spelt in a few different ways but it conveys the information we want without being too long, and critically the domain was available!

# Syndi.me website

![](./syndiwebsitescreeny.png)

By the end of Saturday evening we had finished the website: https://syndi.me and posted it to a number of blogger facebook groups and sub-reddits so hopefully over the course of the next day or so we will get a little bit of feedback to see if people are interested.

We used Squarespace to build the site as its quick and relatively low cost. Andrew was very familiar with Squarespace too so that cut down the learning time.

I found it tricky to work out what copy to put on the website. How to convey the idea without being too complicated right away. Fortunately Alexis was there to help me. In the end what we went for was a simple summary, followed by more in-depth information in an FAQ page.

# Sunday

By the time sunday rolled around it was all about preparation for the pitch. Building slides and practice practice practice.

We had a number of helpful presentations from the organisers during the weekend which really honed our pitch by encouraging us to present the idea as a story.

After the 10th time practising the pitch I was feeling pretty confident that I could do it. I knew things were going to be different when I actually get up there though.

# The final pitch

So by late afternoon it was the big moment. We all took our seats as the pitches began.

I was up 5th of 11 so I had a few to watch first which was both a blessing and a curse as there were some really great pitches. Infact as a whole the pitches were much, much better than any other Startup Weekend I had been to.

So finally it was my turn. I took the microphone and got up and stood infront of the projector and 100 other people and began my rehearsed pitch..

<iframe width="853" height="480" src="https://www.youtube.com/embed/jPYgKR1hGlg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Oh dear! Not good, I almost choked on the second slide. I dont know what happened exactly I think I was concentrating on the slide before which pushed out of my head what I was going to say next, then I started to panic and that then spiralled down into more panic.

I stopped, took a breath and decided to just move on and try not to think about it.

The rest of the pitch went well so fortunately it want a total disaster.

# Judging

Once I had finished my 4 minute pitch the whole team came up on stage while the judges asked us some questions and gave us feedback.

One of the first points mentioned by one of the judges, Charlie Gunningham, was that my back was to them the whole time, whoops!

The rest of the feedback was very positive, they all thought the idea was great and couldnt believe that it hasnt been done before and we should continue it on!

# The presentations

With that nerve racking experience out of the way we took our seats once more and watched the other pitches.

Once they were all over, the judges went into the back room and conferred for a few minutes before coming out and declaring the winner.

As I mentioned above I was really very impressed by the pitches this time. There was some really great ideas and some very well presented information. Some of the groups really did an excellent job portraying their information.

I wasnt expecting to win anything considering the mess I had almost made of the pitch but to my great surprise when Chloie stood up and announced that the runner up prize went to Syndi.me I was totally taken aback. Wow we had come second! (well technically third as there was a join first prize this year)

![](./tshirt.jpg)

We went up on stage to applause, was given a back of goodies, shook hands with the judges then took our seats with massive smiles on our faces.

# Conclusion

I am so glad at how the weekend turned out. I achieved what I aimed for. Pitch and idea, form a team and try to validate an idea.

I was super lucky with the team that joined me over the weekend, they were just the right mix of people with no egos. We worked together so well, and they were incredibly supportive of me even though I freely admitted I wasnt the best presenter but wanted to give it a go regardless.

So now I have to decide if I want to take the judges advice and continue our work on Syndi.me.

What do you think dear reader? Leave a comment below if you think I should continue on with Syndi.me and actually build a working solution..

So this weekend I took part in an event I signed up for some time ago: Startup Weekend, Perth. Startup Weekend is an event where teams all over the world come together to build a startup over the course of a weekend.



[![2015-11-16_08-28-59](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-16_08-28-59-1024x615.png)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-16_08-28-59.png)

Being very interested in startups as I am and having enjoyed [GovHack earlier this year](https://www.mikecann.blog/portfolio/projects/govhack-2015-should-i-drive/) I thought it would be a perfect event for me.

It started on Friday after work. We piled into the startup-hub Spacecubed on St Georges Terrace. After a few looseners and semi-nervous chitchat we were soon taking our seats for the introduction presentation by the excellent [Marcus Holmes](https://www.startupnews.com.au/about/).

[![2015-11-13 19.49.25](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-13-19.49.25-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-13-19.49.25.jpg)

After the short talk split into informal groups to play "Startups against humanity" (I think). Each group was given a "Problem" and a number of "Solution" cards. In typical [Cards Against Humanity](https://cardsagainsthumanity.com/) style, the problems and solutions were suitably 'humerous' in nature.

As I understood it, the purpose of this game was to get the creative juices flowing. For me tho it just seemed to devolve into a shouting competition with the loudest people making the decisions. Being small and on the outside of our group I couldn't really see what was going on or contribute. Instead I just decided to take a bathroom break and wait for it to finish.

Thankfully the game was only played for a brief time before the real action started, pitching time. The pitches were held in the same fashion as Govhack. Pitchers would line up and take turn for one minute each to pitch their idea to the group.

This whole process went rather smoothly, all credit to Marcus and his team there for keeping the pitches moving rapidly. Infact the entire event was run very well, it felt much better organised than Govhack, but then again it was the 7th time it had been run.

[![2015-11-13 19.49.27](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-13-19.49.271-1024x765.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-13-19.49.271.jpg)

After presenting their concept the pitchers knocked together quick posters and affixed them to the walls. We, the non-pitchers then went around the various posters and deliberated over which one we wanted to join.

At Govhack we were given little stickers to stick on each poster to show our interest. I didn't see the point in that at Govhack so thankfully that process wasn't followed through at Startup Weekend.

The [mistake I made at Govhack](https://www.mikecann.blog/portfolio/projects/govhack-2015-should-i-drive/) was to deliberate for too long. I couldn't make my mind up at that event and very quickly the teams had formed and I was left at the end wondering around like a lost lamb wondering where to go.

I was determined not to make that mistake again so after one quick lap of the available options I decided upon Mary's "Stop the hangry". I chose it because the idea was simple: "Develop an app that maps food trucks in Perth".

The idea resonated with me as I could quite easily see a situation: it's late, you want to eat, all the good restaurants are booked and you don't want take away or Maccas yet again, what you really crave is some wood-smoked-ribs or perhaps spicy mexican tacos but you have no clue where that food-truck you saw that one time would be.

When Mary pitched the idea I was stood at the back of the room and noticed quite a few people nodding their heads around the room. For me this was also a good sign that the idea was good, it was an instant crowd validation.

So I signed up to "Stop the hangry" which eventually turned into "Tuckr". The irony is that Mary already knew of me via my blog post of Govhack, she had even downloaded Mr Nibbles for her children to play, we were going to get on just fine :)

Surprisingly (to me at least) we struggled at first to get the required 5 team members. I thought the idea was solid but it seemed like others weren't so sure. It also looked like I needn't have rushed into my team choice as the group forming stage took much longer than the Govhack event.

I had quite a few offers by other teams to join them. It seemed like developers were very much in demand at the event and hot property. I was sold on the food truck idea however and instead tried to convince them to join us instead.

Just as the situation was starting to look dire, Mary and I we finally secured our third Member Adam then soon after Trent, Daniel and finally the golden catch, Sydel the designer, an even rarer breed than developers at Startup Weekend.

[![2015-11-14 12.53.23](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-14-12.53.23-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-14-12.53.23.jpg)

So we had 6, two developers, one designer and 3 non-tecchies, an ideal composition, things were looking good so we set to work.

It had been heavily suggested to us, then over the course of the weekend hammered into us, that if you wanted to win it wasn't about building the thing, it was about validating that people wanted your thing.

Now this is the critical rub for the entire event for me. The more and more I go to events like this and other startup related meetups I find out about the type of person I am and what drives me. I am discovering that I driven by the technology of the product rather than the process of business creation itself. I love to make the thing not necessarily make the business that makes the thing.

I totally understand that the business is completely necessary. You must have a vehicle to drive your product and more importantly you need to make sure you are creating the correct product in the first place. This process is what Startup Weekend is all about and unfortunately for me, not about actually creating the thing.

In hindsight this is kind of obvious, the event is called Startup Weekend, its about building a startup in a weekend. It's not a hackathon like Govhack which is all about hacking together various technologies to solve a well defined existing problem.

So although I really wanted to make the actual app, we decided it would have been better if we just concentrated on making a landing page and designing how the app would look and function. These things are what the judges are actually looking for, not an actual working product.

[![2015-11-14 16.47.21](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-14-16.47.21-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-14-16.47.21.jpg)

So until 8pm on Friday, 10pm on Saturday and 4pm on Sunday I hacked together a landing page: [https://tuckr.co](https://tuckr.co) with Adam whilst Sydel produced some awesome mockups for the app. Daniel, Mary and Trent spent the time trying to validate the idea and develop the business model.

During the entire weekend we had regular "check-ins" where all the teams would come together in the main Spacecubed area and let others know what they had been working on and what difficulties there were having. It quickly became obvious that many of the teams were far ahead of the curve. By Saturday morning some of them already had a landing page or product and some had even already generated revenue.

I found check-ins both inspiring and depressing. The were inspiring because could see that other groups were suffering some of the same problems to you but it was also sometimes disheartening to hear how far ahead some of the teams were.

[![2015-11-15 11.36.01](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-15-11.36.01-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-15-11.36.01.jpg)

Soon enough Sunday afternoon rolled around and it was time to present our startup to the the rest of the teams and hopefully win over the four judges.

The presentation process was much better organised than Govhack. Instead of each team going up and individually connecting and disconnecting their laptop from the projector (which took ages and untold number of problems) every team was required to submit their slides 1 hour before we were due to present. That way a single laptop was used to run the slides. It also meant that there was no way you could have demoed a product even if you wanted to.

Our presentation went well, Mary did a great job of presenting our business concept to the room while I cowered backstage clicking next on the slides.

Sadly tho we didnt win this year, there was some truly excellent competition in the form of [Veri Vote](https://veri.vote/), [Codename Canary](https://codenamecanary.github.io/) and [YoPro](https://twitter.com/yoproapp). Hats off to those guys, they did an excellent job and Veri Vote and Canary had especially great solutions.

Special mention should go to [Mulch Club](https://www.mulchclub.co/) too who were a twice disqualified team (no fault of their own) who managed to generate close to \$1000 over the course of the weeknd which is just phenominal.

So after the awards ceremony the event was wrapped up at 8pm. I enjoyed my Startup Weekend, I learnt a whole bunch about business and startup creation and about myself. Would I do it again? Im not sure, I learnt a lot but I think I had more fun at Govhack where we hacking tech together to solve existing problems.

Massive thanks to my awesome team, you guys did a great job and I hope we can stay in touch and perhaps team up again next time!

[![2015-11-15 19.53.33-2](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-15-19.53.33-2-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/11/2015-11-15-19.53.33-2.jpg)

Well I took a short break this weekend from Mr Nibbles Forever development to head down the street to [SpaceCubed](https://spacecubed.com/) and compete in the [2015 GovHack competition](https://www.govhack.org/perth2015/).



I had heard of GovHack from one of the [Morning Startup](https://www.meetup.com/Morning-Startup-Perth/) meetings I like to attend every Wednesday here in Perth.

The West Australian government has been running this competition for several years now to try to promote their new open data policy. The idea is simple, get a bunch of nerds together, give them the data and offer some prizes and see what happens.

[![CI-222zUcAAvd7s.jpg-large](https://www.mikecann.blog/wp-content/uploads/2015/07/CI-222zUcAAvd7s.jpg-large-768x1024.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/07/CI-222zUcAAvd7s.jpg-large.jpg)

Well I turned up on Friday evening not knowing anyone and having no clue what to expect. So I grabbed a beer and started chatting.

Some people were in teams already, some people worked together for a day job but it seemed like there were quite a few other people in the same boat as me, new to it, without a team or project or idea. That made me feel a little better, but I still had no clue what to expect.

[![CI_vYWaUEAA1-_P.jpg-large](https://www.mikecann.blog/wp-content/uploads/2015/07/CI_vYWaUEAA1-_P.jpg-large-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/07/CI_vYWaUEAA1-_P.jpg-large.jpg)

After a few short speeches the participants were invited to stand up and pitch their idea. All in all there were about 20 pitches. Some sounded interesting, some sounded really wacky such as one team wanted to use the data to become evil-masterminds. One team even suggested that they were going to correlate the prizes against the data and see which prizes were the best to go for then work out and idea from there.

![CI_Fh1FUAAAv7yd.jpg-large](https://www.mikecann.blog/wp-content/uploads/2015/07/CI_Fh1FUAAAv7yd.jpg-large.jpg)

After the pitches were over, we given 5 stickers to stick posters that had been hastily assembled by the pitchers. I think that the idea of this was to gauge the popularity of the project but im not sure exactly how that affected things in the end.

Also this was apparent the point that you were supposed to be looking to join one of the teams too. I didnt realize this and I spent ages walking around and deliberating while groups of people started separating and disappearing.

The problem was, none of the projects really jumped out at me. The only pitch that grabbed my attention was by a guy that wanted to predict if you are going to win a court case by looking at old case documents, I thought that one sounded pretty interesting but unfortunately I couldn't find him anywhere, and none of the organizers could find him either.

In the end I ended up wandering around for about 30 minutes like a lost lamb as everyone started to form teams. In sheer desperation I started butting into the teams and asking what they were doing and if they needed a developer. After a couple times trying this I was about to give up and go home when I decided to try one more team. These guys were planning on building a app that asked the simple question, should I drive or take public transport to my destination?

I thought this idea seemed simple enough, plus it sounded like something I could help out on, I knew front-end and back-end, had built web and mobile apps before so I signend up.

[![CI_Y6mkUEAAG2fh.jpg-large](https://www.mikecann.blog/wp-content/uploads/2015/07/CI_Y6mkUEAAG2fh.jpg-large-1024x652.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/07/CI_Y6mkUEAAG2fh.jpg-large.jpg)

I think I got really lucky. They turned out to be a truly awesome bunch of people. There were 7 of us in total: Jake, the team leader and front end developer. Cam, the back-end dude with lots of experience with mapping and spacial data. Kim, the data miner and wrangler. Felix, a young but incredibly bright lad who did a bit of everything (including winning the young-and-up-coming guy to watch out for award). James, and his friend Apara who did the super important job of organizing the project and video. And of-course me, who took the role of tooling / data converter / uploader / production / whatever else needed doing.

It was the right mix of skills, ages and experience and over the next 3 days we had a ball. We were asked at one point by one of the VIPs / guests that was walking around how everything was working, who was deciding what needed doing etc.

Well that was a good question. It all just seemed to come together. One of us would say "oh, wouldnt it be cool if it could do X?" then we would all agree and then one person would go off and get the data while others did others set about converting it and uploading it.

We were told to put a lot of effort into the video as that would be the thing the national judges would use to choose who their winners were (national prizes are separate from the local WA prizes and were worth \$80k). So right from the start we were scripting and planning the video.

It all kind of came together on the Sunday morning when we filmed it all and produced the following:

<iframe width="700" height="500" src="https://www.youtube.com/embed/hgfu31GtvQk" frameborder="0" allowfullscreen></iframe>

I am pretty proud that we were able to not only build a working app that uses the various data sets but were also able to produce the video in the very limited time. Hopefully it conveys the purpose of the app well and maybe also shows you a little bit about the team and how much fun we all had.

We weren't quite done tho as we still had to do a live presentation to a panel of judges. We had 3 minutes to show what we had, then a little more time for questions from the judges.

[![2015-07-05 16.52.41](https://www.mikecann.blog/wp-content/uploads/2015/07/2015-07-05-16.52.41-1024x1024.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/07/2015-07-05-16.52.41.jpg)

Personally I felt that this one probably the worst part of the event. There were 20 teams presenting, we were number 14\.

We had been told that it was supposed to be 3 minutes long however it seemed like the first 3 or 4 teams were going way way over that, 10-15 minutes each.

For some reason they changed their policy half way through so now you had exactly 3 minutes and were clapped off when the buzzer sounded. It seemed a little unfair to us who werent included in the earlier set of presentations.

Also the presentations were plagued with audio / visual difficulties thanks to the fact that we had to continually plug and unplug laptops. Presentation and demos that worked fine on a 15" laptop looked terrible or didnt work when switched to a projector resolution.

So most teams seemed rushed, unorganized and generally let all their hard work down by failing to properly present what they had done. The judges also probably didnt get a good feel for what the projects were about or what the problem was they were trying to solve.

Some teams simply played their 3 minute video then asked for questions at the end. Personally I think this worked much better as there was much less to go wrong, plus you had already spent a whole bunch of time on the video, you may as well let it do the talking instead of you. I wish we had done this.

Nevertheless our presentation went okay, nothing too major went wrong (apart from from projector issues) but we just didn't have enough time to do the project justice.

After all the presentations, the judges disappeared for 45 minutes while they deliberated, enough time to grab a beer, then the awards started.

![CJJboKPUkAE8Wnm.jpg-large](https://www.mikecann.blog/wp-content/uploads/2015/07/CJJboKPUkAE8Wnm.jpg-large.jpg)

There were \$35,000 up for grabs in prizes over several different categories. You were supposed to elect which prizes you were going for on your project page, some teams chose to select most if not all of the prizes. [We went for 16 of the prizes](https://hackerspace.govhack.org/content/should-i-drive).

In the end we ended up winning 2 of them including the single largest prize: [Main Roads Keep WA Moving Prize](https://hackerspace.govhack.org/prize_entries/main-roads-keep-wa-moving-prize) of $2,500\. In total we won $3,000 between us which was great, not the best but not the worst.

![2015-07-06_19-07-28](https://www.mikecann.blog/wp-content/uploads/2015/07/2015-07-06_19-07-28.jpg)

For me tho, it wasn't about the prizes (in fact I didn't even know there was any cash prizes when I entered), it was more about getting out there and meeting other like-minded people who wanted to work together and build something new and cool and perhaps learn a thing or two about the WA data along the way.

I think I succeeded in that modest goal 10 fold and am incredibly grateful to GovHack and my team for letting me be a part of it.

[![Team shot](https://www.mikecann.blog/wp-content/uploads/2015/07/Team-shot-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2015/07/Team-shot.jpg)

I talked about this in my [last blog post](https://www.mikecann.blog/myprojects/mr-nibbles-alpha-8/) but Mr Nibbles Forever is starting to take shape now. The features have been ready for some time now and the the last update saw 95% of the game's artwork go in. Basically the game itself its pretty much there apart from bug fixes and polish.



Anyone who has worked with indie games will tell you that making the game is the easy bit, its all the rest that is hard. Its that phase that I am now entering into. There is much that I need to do before I can release the game so I thought I would try to get some goals for the next few weeks so I can plan out the next series of steps and hopefully it will help others who are in a similar situation.

## 22nd-28th June

1. Design landing page for the game. This is important as I will need a place to put the trailer and to send people who are interested in the game. I have read several good resources on how this page should look. Some examples of other landing pages are: https://www.pixelprospector.com/list-of-websites-and-blogs-from-game-developers/. Once landing page is designed, run it pass some designer friends for help improving it.
2. Pester people for feedback on Alpha 8\. I need to get some real-world feedback for the game so I can fix bugs and improve the polish for release 9.
3. Contact some marketing people for help and advice.

## 29th June - 5th July

1. Do bug-fixing and polish pass of the game. Push release 9 of the game to people for testing and feedback.

## 6th - 12th July

1. Implement landing page design.
2. Gather feedback from release 9 of the game

## 13th - 19th July

1. Do bug-fixing and polish pass of the game. Push release 10 of the game to people for testing and feedback.
2. Research and plan the teaser trailer.
3. Construct Press Kit

## 20th - 26th July

1. Create game trailer and publish to landing page.
2. Contact media and other social networks, point to landing page.

## 27th July - 2nd August

1. Final bug fix and polish pass.
2. Contact more media and social networks.

## 3rd - 9th August

1. Release Android version of the game.
2. Contact media and social networks.

## 10th - 16th August

1. Release iOS version of the game.
2. Contact media and social networks

Well that's the plan anyways. We will see if its fee sable over the coming weeks. Ill try to stick to it where I can. Given that I can stick to it then hopefully the game should be released in 6-weeks!

I'm extremely proud to announce the launch of a project I have been working on for quite a while now.

[Printomi](https://www.printomi.com/) is a service for social games that allows players to take snapshots of their worlds and upload them to their own personal gallery. Players can like, comment share and order poster prints of their virtual creations.



The way it works is we provide the game owner with a tiny API that is used to interface with Printomi. The bulk of the Printomi client is downloaded at runtime from the Printomi servers. This is good for two reasons. Firstly by keeping down the size of the code that must be embedded into the game we can keep down the initial load time the player must sit through before they can play the game. Secondly we can upload changes and fixes to Printomi without requiring the game owner to recompile their code.

Once downloaded the printomi client is ready to be be used. In the case of [Gourmet Ranch](https://apps.facebook.com/gourmetranch) a themed button is shown to the player inviting them to click:

[![](/wp-content/uploads/2012/04/btn-to-click.png "btn-to-click")](/wp-content/uploads/2012/04/btn-to-click.png)

Once clicked (or activated by the JavaScript API) the main Printomi window opens:

[![](/wp-content/uploads/2012/04/01.jpg "01")](/wp-content/uploads/2012/04/01.jpg)

The window has controls to zoom in and move the print about allowing the player to get the perfect angle of their world:

[![](/wp-content/uploads/2012/04/02.jpg "02")](/wp-content/uploads/2012/04/02.jpg)

Once happy the player can then saves the print to the printomi servers. While the image is uploading the user can continue to play the game. Once the upload is complete the window is opened once more offering to show the player their print.

[![](/wp-content/uploads/2012/04/03.jpg "03")](/wp-content/uploads/2012/04/03.jpg)

Printomi is tightly integrated with Facebook so that it's as easy as possible for new users to get started:

[![](/wp-content/uploads/2012/04/04.jpg "04")](/wp-content/uploads/2012/04/04.jpg)

Once connected the new print is then available for viewing, sharing, liking, commenting, etc:

[![](/wp-content/uploads/2012/04/05.jpg "05")](/wp-content/uploads/2012/04/05.jpg)

In addition the prints are organised into galleries so that users can browse the uploads of others:

[![](/wp-content/uploads/2012/04/06.jpg "06")](/wp-content/uploads/2012/04/06.jpg)

If a user particularly likes one print then they can order it as a poster:

[![](/wp-content/uploads/2012/04/07.jpg "07")](/wp-content/uploads/2012/04/07.jpg)

Currently we only offer posters to people living in the US however we plan on opening it up to other countries when we can.

To print to such a large poster (24x18" @ 150DPI) and maintain the quality of the final product we must capture a large number of pixels. For example here is a comparison between what a user would see on printomi.com and what we store behind the scenes:

[![](/wp-content/uploads/2012/04/08.jpg "08")](/wp-content/uploads/2012/04/08.jpg)

The technology that lets us capture these large images and store them in a compressed way (to conserve disk space and therefore cost) is quite complex and will have to wait for a later post.

Printomi is currently only available to a small subset of the [Gourmet Ranch](https://apps.facebook.com/gourmetranch) user base but we are performing a steady-rollout to everyone, so if you don't have access to the button in Gourmet Ranch just yet then don't worry it should be available for you soon!

Printomi is a culmination of many many hours of hard work, late nights and stress. But its been worth it! I've had a blast throughout the whole process and am proud of the result. I have also learnt a whole bunch in the process. I have had to work with so many technologies to pull this off, to name a few: AS3, C++, C#, ASP.Net, MySQL, IIS, AWS. I have particularly enjoyed the C# work having largely abandoned the language since my [early work](/posts/windows-7-taskbar-performance-monitor-v0-2/) [with it.](/posts/killer-space-penguins/)

[![](/wp-content/uploads/2012/04/fb1.jpg "fb")](https://www.facebook.com/printomi)[![](/wp-content/uploads/2012/04/tw.jpg "tw")](https://twitter.com/#!/printomi)

If you would like to support me or if you would like to keep up to date with the goings on with [Printomi](https://www.printomi.com/) then you can like us on [Facebook](https://www.facebook.com/printomi) or follow us on [Twitter](https://twitter.com/#!/printomi) :)

## More

If you would like to read more on Printomi then checkout these posts:
[catlist id=27 numberposts=100]

![sorrygj](/wp-content/uploads/2009/06/sorrygj.gif "sorrygj")

As it has now [been posted](https://support.gamejacket.com/forums/19376/entries/40976) on the GameJacket Support site I guess its okay for me to announce too that GameJacket finished trading on the 8th while I was [away in Canada](https://www.mikecann.blog/?p=554).



I have had an excellent time at the little start-up and have learnt alot about both technical aspects of the web but also about how a small business works and functions. Its sad to see GJ go as it had so much promise in the early days. Without the funding for expansion however we were never going to be able to compete with the multi-million pound Mochi Ads.

On the last day before I took off on my trip we all had a little farewell party in the office. A little wine tasting and some cheese and crackers. The pictures and a video are [on my Picassa](https://picasaweb.google.com/mike.cann/FarewellGJ09#), but here are a couple:

[![](https://lh3.ggpht.com/_vZ6zE_QJfu0/SjpSlANQ3pI/AAAAAAAAZ7k/gUNnRBb1LaI/s288/DSC03738.JPG)](https://picasaweb.google.com/lh/photo/kt-QnclqVpz1Ai5HdrE55A?feat=embedwebsite)[![](https://lh4.ggpht.com/_vZ6zE_QJfu0/SjpSiYzg1WI/AAAAAAAAZ7Q/3YmKiuP37YY/s288/DSC03736.JPG)](https://picasaweb.google.com/lh/photo/vwnFuHInOwfR0nRBkqzlNA?feat=embedwebsite)