Its been a little while since I have done a pure tech focused blog post but I discovered an issue with React Hooks and Chrome Extensions for which I could find no mention of anywhere on the internet so I thought I would share the issue and my solution.



# The Issue

So the problem is; if you build a chrome extension with a background page and try to use a `useEffect` hook on that page, you will notice that the handler of the effect doesn't trigger.

```typescript
import * as React from "react"
import { render } from "react-dom"
import { useEffect } from "react"

function App() {
  console.log("Execution reaches here..")

  useEffect(() => {
    console.log("..but this never gets triggered.")
  }, [])

  return <div />
}

render(<App />, document.getElementById("root"))
```

This is obviously not quite right.

I put together a demo app for this here: https://github.com/mikecann/bgpage-hooks-issue

# The Workaround

It appears as if this issue started happening somewhere between 16.8 and 16.9 so you could just downgrade your react version.

I also discovered that you can workaround the issue with this little hack:

```typescript
function useBackgroundKeepaliveHack() {
  const [tick, setTick] = useState(0)
  useLayoutEffect(() => {
    setTick(1)
    setInterval(() => setTick(prev => prev + 1), 1000)
  }, [])
}
```

If you add that into your component tree then hte effect now triggers as expected, very curious. Perhaps it has something to do with the fact that background pages aren't actually rendered to the screen thus perhaps React can detect that and prevent effect updates?

I have reported the issue to the React github page so hopefully it will be looked into and fixed at some point: https://github.com/facebook/react/issues/16629

I spent a few hours tinkering with the new (alpha release) version of React which includes the new React Hooks feature. I wanted to experiment with it in a semi-realistic environment (not CodeSandbox). I have built a few chrome extensions in the past and I knew the challenges well so I thought it would be a good scenario to challenge React Hooks and see what it can do.



# TLDR:

The below video is a quick demo of what the chrome extension does.

<iframe width="853" height="480" src="https://www.youtube.com/embed/iydk07TCOOs" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

The video demonstrates how the state is able to be shared between the background page, child pages and content scripts simultaneously.

The code is available for your perusal here: https://github.com/mikecann/react-hooks-chrome-extension

If you want to hear about some of the issues and their solutions I faced read on..

# The Chrome Extension State Sharing Problem

Over the years I have authored a number of Chrome Extensions. They have been written in all manner of different ways using all manner of languages and libraries; React, Aurelia, Angular, Mobx, Mobx State Tree, Typescript, Redux and plain old JS just to name a few.

One of the problems that I always face with many of the extensions is how to share the state between the various pages in a chrome extension. Chrome Extension's have multiple "entry points"; a content script (injected), a settings page or a browser action page (the popup that you get when you click the button in chrome). These multiple entries means that you need a way of sharing some common state between all the pages.

Fortunately chrome has provided a number of different ways of doing this. Below are my two favorite.

## Storage Sync

One way of sharing state is via the [chrome sync storage](https://developer.chrome.com/extensions/storage) mechanism. Whenever the state changes on one of your pages you write those changes to `chrome.storage.sync`. Other pages listen for changes to storage and subsequently update their own internal state.

This method is okay but it has its limitations which include storage size limits (5mb unless you give unlimited storage permission), rate limiting and other potential permission issues.

## Message Passing

A second method for syncing state is involves each page passing messages to each other. Using `chrome.runtime.connect` a page can connect to the background page and inform each other of when the state changes.

This method however works well but only if you have a central storage of state and are able to detect changes to that state in a unified way.. enter react-hooks...

# React Hooks

[React Hooks](https://reactjs.org/docs/hooks-intro.html) is a brand new feature (currently in alpha) for [React](https://reactjs.org/). When I heard about it I got excited because it looked like the React guys were once again going to blow my mind by exploring a wholely new way of building apps.

The full details on how hooks work are in [the excellent docs](https://reactjs.org/docs/hooks-intro.html) but the idea is is fairly simple: allow users to write functional style components while allowing state to be managed in a composable, reusable way.

I knew that there was only so much I was going to be able to understand about the technology from reading about it. I was going to have to dig in there and tinker with it a little bit.

## The Event Listener Array Issue

One issue I faced early one with React Hooks was a fundamental "stale state" issue when dealing with event listeners. To demonstrate the issue take the following example:

```typescript
export function useRuntimePorts() {
  const [ports, updatePorts] = useState<chrome.runtime.Port[]>([])

  function onPortConnected(port: chrome.runtime.Port) {
    console.log("port connected", ports.length)
    updatePorts([...ports, port])
  }

  useEffect(() => {
    chrome.runtime.onConnect.addListener(onPortConnected)
    return () => {
      chrome.runtime.onConnect.removeListener(onPortConnected)
    }
  })

  return ports
}
```

This is a custom react hook that's job is to manage the state of connected "ports" from other pages in the extension. It listens to the chrome runtime api for connections from other pages and updates its state (array of ports) which it then exposes to consuming code by returning the ports.

When a new connection is made, we get a console message that tells us a port was connected. We are also given the number of currently connected ports. As expected the number of ports increases as we open more connections from other pages.

The keen eye will notice that the `effect` is re-executed every time that the hook is re-executed on each re-render. This means the event listener is continually being added and removed. This seems wasteful and luckily the React API has a solution in the form of a ["third" parameter](https://reactjs.org/docs/hooks-reference.html#conditionally-firing-an-effect) that you can pass to `useEffect`, an array that indicates only run the effect on the update if one of the the elements in the array has changed.

So if I simply supply an empty array it means that the effect will only be run once:

```typescript
  ...

  useEffect(() => {
    ...
  }, []);

  ...
```

This is great, it now means we dont wastefully attach and detach the listener every time the component is re-rendered.

When we try this out in out extension however a problem has arisen. The console message that tells us that a port has connected now no longer counts up the number of ports, it stays at 0. Why is this?

Well if you think about it, it makes sense, we arent re-attaching the event listener on each time the hook is executed. That means that the `ports` variable inside of the `onPortConnected` handler is always going to be the original array the first time that this hook was run. So doing an immutable append `[...ports, port]` is always just going to append the latest port to an empty array.

I struggled with this issue for quite some time. I tried quite a few different things including trying out `useRef` to hold a mutable array, but that seemed hacky. There must be a better way I thought to myself.

Eventually I decided to pose the question on the [Reactiflux discord chat](https://discordapp.com/channels/102860784329052160/103696749012467712). A fine fellow by the name of Kingdaro responded with:

> Try using the callback in the update function
> `updatePorts(ports => [...ports, port])`. In theory that'll make it so that the port will reliably always be added to the end of whatever's there, as opposed to using what's in scope

Bingo! That was the missing link. I hadn't realized you could pass a function to the state update function instead of the next state ([despite it clearly mentioned in the docs](https://reactjs.org/docs/hooks-reference.html#functional-updates)). With that subtle change and my hook now looked like:

```typescript
export function useRuntimePorts() {
  const [ports, updatePorts] = useState<chrome.runtime.Port[]>([])

  function onPortConnected(port: chrome.runtime.Port) {
    updatePorts(prev => {
      console.log("port connected", prev.length)
      return [...prev, port]
    })
  }

  useEffect(() => {
    chrome.runtime.onConnect.addListener(onPortConnected)
    return () => {
      chrome.runtime.onConnect.removeListener(onPortConnected)
    }
  }, [])

  return ports
}
```

And huzah! The console message logged out incrementing port count as expected.

## Testing

One other major question I had for React Hooks was how does one go about testing them?

Fortunately wiser folks (in the form of the prolific Kent Dodds) had already thought long an hard on this and created this excellent video: https://egghead.io/lessons/react-test-react-components-that-use-react-hooks and wrote this blog post: https://blog.kentcdodds.com/react-hooks-whats-going-to-happen-to-my-tests-df4c2b4d67b7

So armed with these I began testing.

```typescript
let state: ReturnType<typeof usePorts> | undefined

function SomeComponent() {
  state = usePorts(jest.fn())
  return <div>hello</div>
}

it("should add a port when one connects", () => {
  const { rerender } = render(<SomeComponent />)
  rerender(<SomeComponent />)
  const port = portFactory()
  chrome.runtime.onConnect.addListener.mock.calls[0][0](port)
  expect(state).toEqual({
    ports: [port],
    lastConnected: port,
    lastDisconnected: undefined,
  })
  expect(state!.ports[0].id).toEqual(0)
})
```

The above is a sample of my test for the `usePorts` hook. You will note that I include the hook inside a component `SomeComponent` with which I use to interrogate the state returned from the hook each render. I do this rather than calling the function directly because although hooks look pure they aren't actually pure. They require that they are run within the context of a rendering Component to work.

The `rerender()` call is necessary because hooks that use `useEffect` by their nature are asynchronous, thus to force React to apply the effect of the hook before we run out testing logic, we must re-render the component.

I am really quite happy with how easy they are to test and how little mocking there is. The only mocks I am using are the chrome API calls because they dont exist inside the testing environment. This means I am not testing the implementation and just the result which makes the tests far less brittle to refactors.

## The Performance Question

I only have one outstanding concern about Hooks, does re-defining then re-executing the scoped functions each render create extra memory on the heap?

What I mean is given my `useRuntimePorts`example from above, each time this hook is run is a new scoped `onPortConnected` function placed on the heap? If so, wouldnt this create a lot of extra memory and thus pressure on the garbage collector?

Basically, I would love to know what are the performance implications, specifically memory / garbage collection, of hooks?

# Conclusion

All in all im very impressed with React Hooks. The code that it produces is clean, easy (ish) to understand and is very testable. No proxys or magic is needed (sorry mobx, I love you but I hate the proxy magic), just plain old Javascript objects.

I cant wait to use hooks in a production app.

If you want to checkout the code then its all up on my repo: https://github.com/mikecann/react-hooks-chrome-extension

Building the frame and assembling [the hardware](/projects/mikes-mirror-hardware/) for my Smart Mirror was a tough but fun journey. But now its time for what im good at, the software.



# Parts

This is a 3 part series, find the other parts here:

1. [Mikes Mirror - Overview](/projects/mikes-mirror-overview/)
2. [Mikes Mirror - Hardware](/projects/mikes-mirror-hardware/)
3. Mikes Mirror - Software

# Research

I started off by doing a bunch of reading and research on what software to use on my mirror.

I discovered a popular project called [MagicMirror<sup>2</sup>](https://magicmirror.builders/) which is an electron based smart mirror project. Magic Mirror comes with a huge number of [Third Party Modules](https://github.com/MichMich/MagicMirror/wiki/3rd-Party-Modules) that you can use to add all sorts of functionality to your mirror in a modular way.

I initially experimented with MagicMirror<sup>2</sup> but quickly found myself getting frustrated at the way the code was structured. I felt that developing my own modules as going to be a pain so I decided I could probably save some time in the long run by rolling my own solution.

# Mikes Mirror

![Mikes Mirror on GH](./githubss.png)

My from-scratch solution uses React, Typescript and Electron for the front end with Python and Dlib for the Facial Recognition part and Sonus (snowboy, Google Cloud) for the speech recognition part and a number of other libraries.

# Profiles

One of the key requirements for me (that MagicMirror was missing) was that the mirror should support multiple different people using the mirror. I wanted this ability so that when someone stands in front of it they get a mirror that is tailored to them.

I call this user-specific content a "Profile" and is simply another react component that contains a collection of "Plugins" that are laid out however you want. So for example here is my Profile:

```typescript
import * as React from "react"
import Particles from "../plugins/particles/Particles"
import Clock from "../plugins/clock/Clock"
import WeatherChart from "../plugins/weather-chart/WeatherChart"
import ProfilePicture from "../plugins/profile-picture/ProfilePicture"
import SystemInfo from "../plugins/system-info/SystemInfo"
import css from "./styles"

export default class MikesProfile extends React.Component<any, any> {
  componentDidMount() {
    // Play a sound when showing the profile
    new Howl({ src: ["./bell.wav"] }).play()
  }

  render() {
    return (
      <div className={css.profile}>
        <Particles />

        <div className={css.rootContainer}>
          <div className={css.hozContainer}>
            <Clock />
            <div className={css.flex} />
            <ProfilePicture url="../facial_recognition/faces/mike.jpg" />
          </div>

          <div className={css.flex} />

          <div className={css.hozContainer}>
            <WeatherChart />
            <div className={css.flex} />
            <div className={css.vertContainer}>
              <div className={css.flex} />
              <SystemInfo />
            </div>
          </div>
        </div>
      </div>
    )
  }
}
```

# Plugins

Most of the logic is contained within composable "Plugins".

![Plugins](./pluginsss.png)

For now I have only written a few plugins. I have included them in the main repo but they could quite easily be extracted out into separate repos much like MagicMirror<sup>2</sup> does it.

Unlike MagicMirror<sup>2</sup> my widgets are much much simpler to develop as they are just React components. IMO this is a perfect use-case for the component based, only-update-when-needed nature of React.

[Checkout some of the widgets in the repo](https://github.com/mikecann/mikes-mirror/tree/master/src/widgets) for examples of how simple and intuitive they are to develop.

[![](./general-ss.jpg)](./general-ss.jpg)

A couple of examples of plugins are:

- clock - displays a clock and date
- display-deactivator - when the widget is mounted it waits for a set amount of time before turning off the display, when it is unmounted it turns the display back on again. This is great for saving power when no one is stood infront of the mirror.
- face-profile-switcher - will change profile when it detects that person infront of it
- system-info - reports some system stats like CPU and MEM load
- weather-chart - shows a chart of the Perth weather

# MobX

I use MobX to manage the state on some of the more complex and core functionality such as the facial profile switching and the voice commands. Having a centralized state also means it can more easily be shared between components and reduce coupling.

# Facial Recognition

I had spent quite a bit of time trying to get Facial Recognition working how I wanted. In the end I ended up using the awesome [facial_recognition](https://github.com/ageitgey/face_recognition) library on github.

The facial_recognition library in turn relies upon Dlib which means that its a bit of a pain to install (LOTS of dependencies) but I did eventually get it working with reasonable performance on my target hardware.

For now im using a modified version of [facerec_from_webcam_faster.py](https://github.com/ageitgey/face_recognition/blob/master/examples/facerec_from_webcam_faster.py) example that comes with the library. I modified it to read from a directory of faces rather than hardcoding them. I also modified it to report results as JSON to stdout which then is read by the main process in the Electron app.

[![](./webcam-active.png)](./webcam-active.png)

Currently the facial recognition only trains on a single example of peoples faces. This means that its not as accurate as it could be particularly under different lighting conditions.

The library does contain another example [face_recognition_knn.py](https://github.com/ageitgey/face_recognition/blob/master/examples/face_recognition_knn.py) which is a little more complex but supports training a face with multiple examples using the k-nearest-neighbors algorithm. I hope to integrate that with the webcam sometime in the future.

# Speech Recognition

Speech recognition was actually harder that I had imagined it would be. I bounced around quite a few different libraries trying to get them to work on my hardware.

In the end I settled on a javascript project called [Sonus](https://github.com/evancohen/sonus). Its a pretty simple project that combines the hotword detecting power of [Snowboy](https://github.com/Kitt-AI/snowboy) with the more complex sentence comprehension power of [Google's Speech-to-Text API](https://cloud.google.com/speech-to-text/).

Like the Facial Recognition, Sonus runs in a separate process that is spawned by the Electron app and then communicates via a JSON protocol over sdtout.

Snowboy continually listens in a background for a given hotword (mine is "smart mirror"). If it detects the hotword then Sonus starts streaming the recording to the Google Speech-toText API which analyses it and returns partial a partial match as you speak and then the final match once you have finished speaking.

[![](./voice-commands.png)](./voice-commands.png)

This combined local-only Snowboy plus cloud based recognition power of Google means that I am able to keep costs and processing requirements down for the mirror.

There are definitely some issues with Snowboy however which I hope to solve in the future.

1. It only works on Linux. This means that should I ever want the mirror to be powered by windows in the future then I cant. It also means I have to develop on my ubuntu laptop rather than my desktop. This isnt a total show stopper however as I really like linux and am hoping to move my desktop machine over to the OS soon.

2. Snowboy comes with a bunch of different pre-trained hotwords and you can even supposedly train your own using [their API](https://snowboy.kitt.ai/). My experience trying to train my own model using my own samples however didnt work. Even the popular pre-trained models dont seem to work particularly well with female voices. Kelsie for example doesnt seem to reliably trigger the hotword for example.

# Voice Commands

Just being able to turn speech into text is pretty useless unless you can then do something with it. So I built a system thats capable of recognising the text and matching it to a command to be triggered.

At first I was using a simple regex matching strategy but quickly found that was limited. A simple example is the regex `lock profile` would be triggered by a user saying "lock profile" and "unlock profile". Now obviously there are ways of working around this but there were a number of other more complex edge-cases too, so I decided to go looking for a better solution.

Natural language detection is a well studied area and there were a bunch of different options on NPM. After playing with a few I settled on [natural-language-commander](https://github.com/will-wow/natural-language-commander). It doesnt have many starts but definitely deserves more (I do love libraries written in Typescript).

So now I can structure my commands like this for example:

```typescript
nlc.registerIntent({
  intent: "lock profile",
  utterances: ["lock profile", "lock my profile"],
  callback: () => profiles.lockProfile(),
})

nlc.registerIntent({
  intent: "lock profile to..",
  slots: [{ name: "Profile", type: "STRING" }],
  utterances: ["lock profile to {Profile}", "fix profile {Profile}"],
  callback: (profile: string) => {
    profiles.unlockProfile()
    profiles.changeProfile(profile)
    profiles.lockProfile()
  },
})

nlc.registerIntent({
  intent: "unlock profile",
  utterances: ["unlock profile", "unlock my profile"],
  callback: () => profiles.unlockProfile(),
})
```

Much better! For a full list of commands in the mirror you can see this file: [commands.ts](https://github.com/mikecann/mikes-mirror/blob/master/src/commands.ts)

I will definitely be adding more as I go.

# Text to Speech

One key feature of any "smart assistant" type project is the ability for the assistant to speak back to you.

For now I opted for a simple solution provided by the [GNUstep speech engine](http://manpages.ubuntu.com/manpages/trusty/man1/say.1.html). Once installed on the OS all I need to do from the app is spawn a command-line instance with the args `say "hello world"`.

The voice quality is pretty basic is you are used to more advanced assistants from the likes of Google or Apple but will do for now.

I did consider just plugging into Google's Text-to-Speech API but decided that I didnt really want the roundtrip time associated with sending the text to the cloud then downloading the resulting WAV file and playing it even though the quality of the returned speech would be far superior to GNUstep.

Perhaps in the future ill research a custom DNN-based text-to-speech solution.

# Future Plugins

Theres so much more I want to add to the mirror but I felt that in its current state its "good enough" for demonstration at least. I suspect that this will be a project that I will keep coming back to and tweaking as I go over a number of months and years.

Some random things I would love to add:

- Live Fitbit stats from my watch
- Voice controls for our Chromecast for our TV
- Speech for when people join and leave the wifi network
- Breaking news reports when something major happens in the world
- .. anything else I can dream up!

[![](./cute-cat-ss.jpg)](./cute-cat-ss.jpg)

# Conclusion

Well I hoped you enjoyed reading about my Smart Mirror project. It took me quite a bit longer than I had hoped to get it to a presentable state. There are just so many options and things to play with. Every time I thought it was "good enough" I would go off on another tangent learning something else which would take me weeks.

Despite the time and cost overrun however I found it a fun and worthwhile project and if this is your sort of thing I definitely recommend giving it a go. I just hope this series of posts helps so you can learn from my mistakes. Please do let me know via email or in the comments below if you give it a go.

Thanks again for sticking with me through these lengthy posts. Let me know in the comments if you enjoyed this, perhaps ill do an update to the project sometime in the future.

# More Reading

This is a 3 part series, find the other parts here:

1. [Mikes Mirror - Overview](/projects/mikes-mirror-overview/)
2. [Mikes Mirror - Hardware](/projects/mikes-mirror-hardware/)
3. [Mikes Mirror - Software](/projects/mikes-mirror-software/)

So I decided to build a Smart Mirror as a hobby project. I knew that I could handle the software side of thing but building a frame and wiring the electronics up was going to be a whole new adventure.



# Parts

This is a 3 part series, find the other parts here:

1. [Mikes Mirror - Overview](/projects/mikes-mirror-overview/)
2. Mikes Mirror - Hardware
3. [Mikes Mirror - Software](/projects/mikes-mirror-software/)

# Research

So first thing I did when trying to work out how to build this thing was to look at other people's efforts. I watched hours of Youtube videos and read just about every post on the [Magic Mirror forum](https://forum.magicmirror.builders/category/12/show-your-mirror).

I recorded some of the best ones I liked in a [Trello Board](https://trello.com/b/dYbx0Zc0/smart-mirror):

[![](./trello-board.png)](./trello-board.png)

# Requirements

After doing the research I realised there were way too many different options so in an attempt to narrow my possibilities I would enumerating my requirements:

1. It must be able to fit in the little alcove near my front door:

[![](./alcove.jpg)](./alcove.jpg)

2. Ideally it will blend in with the other wooden furniture in the apartment.
3. Its a rental apartment so we cant put holes in the wall, so it must be light enough to be able to be supported by stick-on hooks.
4. The processor must have enough power to run Facial Recognition, Speech Recognition and a display at a decent framerate all at the same time.
5. It shouldnt cost too much

# The Computer

To perform the various functions of the mirror I needed a computer of some sort. It needed to be small and light enough that I could attach it behind the mirror.

One of the most obvious and well known options for this is the Raspberry Pi. Through my research I had discovered that there were a great many other types of devices with significantly higher specs than the Pi. This great article lists 20 of them for example: https://www.techrepublic.com/pictures/want-a-more-powerful-raspberry-pi-choose-from-these-20-alternatives/

In the end I decided to buy not one but two computers.

[![](./both-computers.jpg)](./both-computers.jpg)

Firstly the Pine Rock64:

[![](./rock64.jpg)](./rock69.jpg)

This is a cheap ARM powered board the size of a credit card that according to reviews is significantly more powerful and more memory for the price than the Raspberry Pi 3.

I also decided to buy an Up Core:

[![](./up-core.png)](./up-core.png)

The Up Core is about 4 times as expensive at the Rock64 but is even more powerful and significantly, it runs on an Intel x86 CPU instead of an ARM CPU. This is important because I knew from past experience that there may be issues trying to get things to work on ARM powered devices.

Projects on Github tend to be written with x86 in mind and drivers tend to expect common operating systems, so with the Up Core being x86 means that I can run Ubuntu or even Windows if I want.

When they arrived I tinkered with both but ended up using the Up Core in the mirror due to the extra performance and the compatibility of the operating system with all the software.

# Camera

I looked at quite a few different cameras on Aliexpress.com. I had a lot of unknowns.

- I wasn't sure if there were going to be drivers for them
- I wasn't sure if the quality was good enough for facial recognition
- I wasn't sure what lens-angle I needed
- I wasn't sure how I was going to build it into the frame.

In the end, due to my uncertainties and the fact that they were all relatively cheap, I decided to pick up three different cameras:

[![](./cam1.jpg)](./cam1.jpg)
([Zbravo 720p endoscope](https://www.aliexpress.com/item/HD-720P-6LEDs-5-5MM-USB-Endoscope-Borescope-Snake-Inspection-Pipe-Tube-Video-Mini-Camera-IP67/32747992958.html?spm=a2g0s.9042311.0.0.1ee84c4dtkgQAk))

[![](./cam2.jpg)](./cam2.jpg)
([Diske 720p pinhole camera](https://www.aliexpress.com/item/HD-Micro-USB-Camera-1-0MP-USB-CCTV-Camera-720P-USB-Module-With-Cables-USB-Board/32825411576.html?spm=a2g0s.9042311.0.0.1ee84c4dtkgQAk))

[![](./cam3.jpg)](./cam3.jpg)
([Kayeton 1080p camera](https://www.aliexpress.com/item/2MP-Full-HD-1080P-Webcam-OV3660-UVC-Plug-Play-Android-Linux-Windows-Mac-USB-Camera-Module/32815156322.html?spm=a2g0s.9042311.0.0.1ee84c4dtkgQAk))

I ended up using the third camera in the end mainly due to the superior picture quality over the others.

One thing to note for next time is that all 3 of the cameras had no trouble running on both Linux and Windows. I didnt have to go searching around for drivers, they just worked :)

# Microphone

As with the camera I wasnt sure about microphones. I ended up picking up this cheap one from ebay:

[![](./mic1.jpg)](./mic1.jpg)
([usb mic](hhttps://www.ebay.com.au/itm/Mini-Wired-USB-3-5-Capacitance-Studio-Microphone-SF-555B-Black-for-Computer-TO/262945101741?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m2749.l2649))

It does the job but I think ill have to invest in something a little more sensitive (and expensive) soon so I can use the voice controls from a further distance.

# The Display

I decided early on that I wanted the display to reach all the way to the edge of the mirror. During my research I noticed that some people mounted a smaller display behind a large mirror thus giving the illusion of a large screen:

[![](./submount.jpg)](./submount.jpg)
(img showing monitor mounted in only a small part of the frame)

I was concerned that this might be noticeable under some lighting conditions. I also wanted to be able to show things in each corner of the mirror. So for me I needed to have a single large display that covered the entire mirror.

At first I considered using a touch sensitive display of some-sort. After some more thought I realized that it would probably be dangerous having people poking the screen on the wall when its only attached by sick-on hooks. Instead of touch I would use voice and camera as input.

I had a choice to make around whether I wanted to use a PC computer monitor or a TV. LED TVs usually start at about 32" and go all the way up to a bajillion inches. I had about 40" of space to work with in the little alcove but as I was weight constrained and TVs tend to be heavy I was starting to think that probably a PC monitor would be best.

Another factor that made me decide on a PC monitor instead of a TV was the resolution. I was concerned that because you would stand relatively close to the mirror you might notice a lower resolution (720-1080p) TV.

[![](./tvpixel.jpg)](./tvpixel.jpg)

So for those reasons I finally decided on a PC monitor. After some searching on Gumtree for 32" inch (about as big as they affordable get) displays I decided that it didnt cost that much more to buy one new. That way I could get exactly what I wanted whereas I was constantly compromising on Gumtree.

So the one I picked up in the end was an Acer 31.5" Monitor ER320HQ from Office Works:

[![](./thedisplay.jpg)](./thedisplay.jpg)

Its got everything I need, a bright 100,000,000:1 contrast display with Full HD (1920 x 1080) resolution and critically only weighed 4kg once I had taken it apart.

[![](./thedisplaytakenapart.jpg)](./thedisplaytakenapart.jpg)

# The Glass

Another of the key components of a Smart Mirror is the two-way glass.

It functions in much the same way a pair of mirrored sunglasses work. Its reflects all light from one direction but allows light from the other direction to pass through.

This means that when a pixel in the display is white then it will shine through and be visible, otherwise you eye will just assume there is nothing there and see the reflection instead.

[![](./two-way-diagram.jpg)](./two-way-diagram.jpg)
(source [twowaymirrors.com](http://twowaymirrors.com/))

Just like the display I had a lot of different potential options for the glass.

1. A pre-cut two way mirrored glass
2. A two-way laminate plastic which you can stick onto a regular piece of cit glass.
3. An acrylic two-way layer.

I decided against number 3) because I had seen examples of it where it had warped a little bit which detracted from the effect somewhat. I thought about 2) but in the end I decided it would probably be safer just to get it pre-made and thus save me the potential of making a mistake applying the laminate.

So option 1) it was. I ended up phoning around a whole bunch of glass suppliers and recording their prices and availability on Trello:

[![](./trello-glass-card.png)](./trello-glass-card.png)

In the end I chose the cheapest supplier... I should have known that something was up when they quoted me half the price of all the others...

About a week later I got a call saying that the glass was ready for me to come pick up, so I excitedly hopped in the car and drove the 30 minutes to pick it up. It was wrapped up so I didnt get to check it out until I got home:

[![](./first-glass.jpg)](./first-glass.jpg)

At first I was impressed, then I turned it over and realised that it was mirrored on BOTH SIDES. Doh! I gave them a call and explained to them that when I asked for a "two way mirrored glass" I didnt mean it was mirrored on both sides... They tried to argue with me for a while, I even quoted the definition of "two way mirror" to them:

> A one-way mirror, also called two-way mirror[1] (or two-way glass) and semi-transparent mirror, is a reciprocal mirror that is partially reflective and partially transparent. The perception of one-way transmission is achieved when one side of the mirror is brightly lit and the other side is dark. This allows viewing from the darkened side but not vice versa.
> (source [wikipedia](https://en.wikipedia.org/wiki/One-way_mirror))

In the end they relented and decided to give me a refund. Sigh.

So now I was onto the next cheapest supplier. I placed an order and a week later I went to collect it. I got it home and unpacked it to discover it looked great:

[![](./second-glass.jpg)](./second-glass.jpg)

Except for that engraved logo at the bottom of the mirror.. FFS! I cant have that on a mirror that is supposed to have a display shining through!

I called them up and they explained that it was common practice for them to engrave on all their "safety glass". Well as I wasnt aware of that fact and it wasnt mentioned to me on their website so they agreed to cut me another piece and _not_ engrave it this time.

So another week goes by and I finally pick up a piece of glass that is actually "two-way" and doesnt have an engraved logo on it, huzzah!

# Building the Frame

Now I had the display and the glass I can finally start the part that was concerning me, the wooden frame.

I was concerned because I had very little experience with woodworking and had virtually no tools. So after watching a few more videos of people making wooden mirror frames I thought I had a basic idea of how I was going to put it together.

[![](./frame-in-two-parts.jpg)](./frame-in-two-parts.jpg)
(yes I diagrammed it out in Unity)

It was basically going to be constructed in two parts which were then to be glued together. According to the videos I had watched, wood glue was apparently really strong and up to the task of keeping it all together.

So off to Bunnings I went and picked up bunch of tools and wood and got to work on the frame.

First I cut 4 bits of wood for the box part of the frame:

[![](./cutting-wood-to-size.jpg)](./cutting-wood-to-size.jpg)

I joined them together with some nails and glue.

Then I cut some more pieces for the front part of the frame and started to glue them to the box:

[![](./gluing-wood.jpg)](./gluing-wood.jpg)

Things were going well, at this point I was feeling like a bit of a Swanson.

[![](./swanson.jpg)](./swanson.jpg)

.. I knew things were going too well... I just noticed this:

[![](./big-overlap-problem.jpg)](./big-overlap-problem.jpg)

That's quite an overlap. Unfortunately I didnt have any more spare wood so I decided that it was something that I could fix with sandpaper... lots of sandpaper:

[![](./rounded-corner-no-overlap.jpg)](./rounded-corner-no-overlap.jpg)

Hey presto! Its fixed :) Its a little thinner on one side but you cant really notice it unless you are looking for it.

One other problem was my poor job at sawing the 45 degree angles for the front of the frame. This was definitely a problem as that was a part you were going to be looking at all the time so that gap really detracted from the quality of the frame.

It was at that point that Kelsie's dad introduced me to the wonders of wood putty:

[![](./wood-putty-crack.jpg)](./wood-putty-crack.jpg)

Huzzah! Problem sovled, just a little more sandpaper and you couldnt tell my rookie mistake was made.

Next up I needed to cut out holes for the speakers for the frame.

[![](./drill-speakers.jpg)](./drill-speakers.jpg)

To be honest, the speakers are on of the main reasons for the mirrors thickness. I think if I was to do it again I might rethink where I place the speakers so that I can try to make the mirror a little thinner.

[![](./speakers-inserted2.jpg)](./speakers-inserted2.jpg)

So now the frame is done its on to staining it roughly the same colour as the rest of the furniture in the apartment.

[![](./about-to-stain.jpg)](./about-to-stain.jpg)

[![](./fully-stained.jpg)](./fully-stained.jpg)

Hey its starting to look quite good, now I just have to worry about assembling it all and attaching it to the wall.

# Assembly

Now I had all the parts I started to assemble them together.

First I laid down the glass mirror side down, then placed the display ontop of it:

[![](./glass-and-display-in-frame.jpg)](./glass-and-display-in-frame.jpg)

I then attached the speakers and the display control board using double sided padded tape. Then wired it all up.

[![](./display-controller-attached.jpg)](./display-controller-attached.jpg)

If you look closely in the above picture you can see that I use angled brackets to hold the display in place and stop it from moving around when I put it on the wall. These are screwed into the wood in a way which meant it would be tricky to remove.

I also screwed in two hooks which I then fastened a piece of yellow string between, this part was surprisingly tough to do. Because I used nylon string it kept stretching and going slack. I wish I knew more about knots, perhaps some sort of self-tightening knot would have done it?

I then grabbed a HDMI cable and attached the Up Core to it and powered it on..

[![](./upside-down-display.jpg)](./upside-down-display.jpg)

It works! But its upsidown.. Whoops! Now that I had screwed everything in, it wasnt really going to be feasible to unscrew it and turn it around. I didnt want to risk splitting the wood.

Fortunately I was able to change it in software so all is not lost, onwards!

# Attaching to the Wall

Now this was the bit that had me the most worried in the entire project.

Because its a rental we werent allowed to put nails or screws in the wall so I had to do the whole thing with adhesive wall hooks.

I whipped out the scales. The whole thing weighed 10.5 kgs which meant I would need 4 of the strongest adhesive hooks I could find at Bunnings:

[![](./wall-hooks.jpg)](./wall-hooks.jpg)

After a bit of help from Kelsie we finally managed to get it up on the wall and it held!

[![](./it-on-the-wall.jpg)](./it-on-the-wall.jpg)

We decided it was probably a good idea to leave it there over night to make sure that it was going to hold before doing anything else.

[![](./not-confident.jpg)](./not-confident.jpg)

Needless to say we werent sure it was going to work.

# Final Touches

Phew! It survived the night and that means we are on the home stretch, just a couple more things.

I attached the Up Core board to the back using some more padded double sided tape to prevent shorts.

[![](./assembly.jpg)](./assembly.jpg)

I then put it back on the wall and noticed that the wires looked kind of messy:

[![](./messy-wires.jpg)](./messy-wires.jpg)

So I picked up some cable tidies from ebay

[![](./organised-cables.jpg)](./organised-cables.jpg)

Then I added the camera and microphone to the top of it (I wish I had come up with a better place for these):

[![](./microphone-camera.jpg)](./microphone-camera.jpg)

Ahh much better, looking good! Now I just have to make the software work. As a professional software developer that should be easy.. right?

# Conclusions

Phew! That was a long process, thanks for sticking with me. It turns out, woodworking is actually much harder than I had expected. Apparently being good at making software doesnt automatically translate to being good at hardware, who would have thought it!

I made plenty of mistakes but im actually really happy with the result. I dont expect it to last a million years like a Ron Swanson crafted masterpiece but it does the job and doesnt look too out of place with the rest of the furniture :)

# Moar!

This is a 3 part series, find the other parts here:

1. [Mikes Mirror - Overview](/projects/mikes-mirror-overview/)
2. Mikes Mirror - Hardware
3. [Mikes Mirror - Software](/projects/mikes-mirror-software/)

I have been wanting to do this for a while and as I recently found myself with a little spare time I decided to finally do it... build a Smart Mirror.



# TL;DR

Checkout the [video](#video).

# What Is a Smart Mirror?

Put simply it is just a computer display behind a two-way mirror so you can both see your reflection and the monitor behind.

![Mirror explode](./mirrorexplode.jpg)
_(img src https://magicmirror.builders/)_

The key concept is that much like a pair of sunglasses or a police interrogation room, you can only see whats behind the mirrored surface if there is light shining through. This means that when the screen is dark you will get a reflection but when the monitor is displaying something you will see that through the mirror instead.

Connected to the display is a small lower-power computer such as a Raspberry Pi, this lets you show whatever you want on the mirror.

# Why?

Good Question. The short answer is: why not? The long answer is: I think its just a cool fun project plus there are a number of things this could be useful for. Its basically your own personal dashboard so on mine I have it showing the weather, time, and hopefully in the future my Fitbit stats from my watch.

![Mirror explode](./healthmock.jpg)
_(img src http://daniel-erhart.com/smartmirror.html)_

When you pair the mirror with some Facial Recognition or Audio Commands then it could become the smart-hub for your home except unlike pure audio devices such as Alexa or Google Home you can have a visual representation too.

# Building

The building of the mirror was quite an involved process so im going to dedicate posts to both the [software](/projects/mikes-mirror-software/) and [hardware](/projects/mikes-mirror-hardware/) side of things.

# Cost

So how much did the whole thing cost to build?

- Monitor - Office Works - \$257
- UpCore - \$181
- Two Way Glass - \$150
- Random Cables - \$25
- Woodworking Tools - \$112
- Wood - \$25

**Total: 750 AUD**

Which is quite a bit more that is strictly necessary and would probably be quite a bit cheaper if I was to do it again.

For example, the I ended up buying the display brand new from Office Works instead of picking one up cheaper on Gumtree. I had very specific requirements for the screen (read more in the [hardware](/projects/mikes-mirror-hardware/) post) and although I could have found a cheaper one second hand there would have been compromises.

The Up Core x86 processor was definitely more expensive than some alternatives. I did buy a cheaper processor ARM processor too but I was concerned about compatibility and performance so I decided to grab the more expensive UpCore x86 CPU as a backup. Im glad I did as its a great little machine and really powerful.

As I had no woodworking tools either, I had to spend about \$100 to buy some. This cost would obviously not be needed next time, tho next time I think I would probably go for one that didn't need a frame, read more in the [hardware](/projects/mikes-mirror-hardware/) post.

# Time

To be honest im not sure exactly sure how long it took. I really depends on how to measure the time, some things were done in parallel, but here is a rough breakdown:

- The thinking planning and research was over a week or two
- Time taken waiting for the CPUs to arrive was a couple of weeks to here in Australia
- Getting the glass cut (three times) was a few weeks
- Building the frame and assembling was done over about 3 days
- Writing the software has been off and on again for a few months (probably could be analysed from the [GitHub repo](https://github.com/mikecann/mikes-mirror))

So probably off and on again I have been working on for the last 3 months or so.

# <a name="video"></a> Video Demo

<iframe width="853" height="480" src="https://www.youtube.com/embed/FHjZnk6JYVQ" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# What went well

- Im impressed with the performance of the Up Core. For the price and size this little thing performs really well and is x86 so runs just about anything you want.
- Im alo impressed with the monitor, it was really light and when I took it all apart the display controls still worked and fit nicely underneath.

[![](./bottom-controls.jpg)](./bottom-controls.jpg)

- Building everything on React and Electron was a good choice. It was quick to iterate and there are tons of prebuilt libraries out there thanks to NPM.

# When went not so well

- I had to order the two way glass three times (more on this in the [hardware](/projects/mikes-mirror-hardware/) post)
- Light from the window behind the mirror causes issues with the face detection sometimes. As a attempt at a solution I bought a desk lamp to try to illuminate your face a little more:

[![](./mirror-with-desklight.jpg)](./mirror-with-desklight.jpg)

- I put the screen in upside down so that on-screen display settings for the monitor are upside down, whoops!

[![](./upsidown-ocd.jpg)](./upsidown-ocd.jpg)

- Im really not a woodworker and so didnt do as good as job as I would have liked there (more on this in the hardware post).
- Racked up an expensive Google Bill because I was recording voice for 500 minutes when I wasn't aware it was running, whoops!

# Conclusions

It was a fun and challenging project. I would definitely recommend it if you are looking for a hardware / software project to work on. There's lots of room to customize it to your needs, just take a look at the [Magic Mirror forum](https://forum.magicmirror.builders/category/12/show-your-mirror) for other peoples projects to see how varied mirror designs can be.

If I was to do it again I think I might so for something that didn't require a bunch of woodwork like this frameless design from Max Braun:

[![](./frameless-mirror.png)](./frameless-mirror.png)

There's plenty more things I want to add and improve to the mirror. I suspect this will be a project ill keep adding to over time and maybe some time in the future I will do a follow-up post, so subscribe to keep up to date :)

# More Reading

This is a 3 part series, find the other parts here:

1. Mikes Mirror - Overview
2. [Mikes Mirror - Hardware](/projects/mikes-mirror-hardware/)
3. [Mikes Mirror - Software](/projects/mikes-mirror-software/)

[Post To Tumblr](https://chrome.google.com/webstore/detail/post-to-tumblr/dbpicbbcpanckagpdjflgojlknomoiah) my popular Chrome extension will be entering its 6th year of continual development this year so I thought it fitting that I give it a much needed overhaul.

It had been long coming, the last major update I gave it was about 1.5 years ago. In that version I moved the backend from Google App Engine over to Parse and I rewrote the frontend to use requireJS and knockoutJS. Although they have served me well, a lot has happened in the past 18 months and maintaining the client code was starting to become painful.



So I decided to checkout what all the cool kids were talking about these days: React.

[React](https://facebook.github.io/react/) is a really cool Javascript framework from Facebook which uses some interesting technologies to render the UI of your application using a component-based structure. It does this by inventing its own language / markup format called JSX which is a hybrid of Javascript and HTML much like MXML was a hybrid of XML and AS3\.

Although awesome to use, and I managed to get it working pretty quick with Typescript too I found that the build chain needed was really a headache. I went through so many different tools and variants. I found that people no longer use Grunt, its now Gulp then I heard some people just use NPM scripts then I read that you want to be using Webpack or Browserify with React. So many tools and options really hampered my progress and ended up giving me a headache.

I eventually fought through the tooling quagmire and ended up with a Gulp / Webpack / Typescript hybrid that sort of work but it takes a long time to compile which i'm not happy about so im definitely on the lookout for a better solution there.

Anyways, now that I have a shiney new client codebase I decided to go through some of the features that people have suggested and implement them so since version 6 came out I have done 6 more updates:

> - v6.7 - 25/01/2016
>
> * Added a new option to general that lets you set what state the format post window should be opened in:
>
> * Added a privacy policy to the account page
>
> * Require privacy policy to be accepted before signing up
>
> * Added a "clear formatting" button to the format page.
>
> * Users that arent logged in must not provide an email when reporting an error
>
> * Fixed a spelling mistake (Thanks Tom)
>
> * Logs older than 2 days are now removed
>
> - v6.6 - 19/01/2016
>
> * The "Remind Me in A Month" and Two Weeks are now working as expected, sorry about that!
>
> * One-click now also gets a reminder.
>
> - v6.5 - 18/01/2016
>
> * The donate button now correctly links to the donation page.
>
> * Added another donation popup, only showing it every 6 months.
>
> * Removed some unnecessary logging to preserve privacy.
>
> - v6.4 - 14/01/2016
>
> * Increased the size of the format post page a little, its 2016 after all ;)
>
> * HTML editing text areas are now resizeable.
>
> * Previous tag sets are now listed in reverse date order that they were used. It displays a maximum of 6 tags. If you have more tags sets you can optionally select to view them.
>
> * Fixed an issue with context menu items not refreshing after adding an account
>
> - v6.3 - 14/01/2016
>
> * Post scheduling is back in addition to queuing. Queuing a post just adds it to the queue, scheduling lets you pick a specific date. (thanks joe, vitor, blinkingline, and others!)
>
> * Fixed an error in the oneclick options which was causing a few errors including not able to rearrange sibling menu items. (thanks adena)
>
> * Reduced the number of plugins TinyMCE is using in the format post window which I hope will increase performance of the text areas, let me know if you are still experiencing issues. (thanks shiinto)
>
> - v6.2 - 12/01/2016
>
> * Added saved tags back to the format post page
>
> - v6.1 - 12/01/2016
>
> * Added HTML editing back to the various text areas in the extension, sorry about that!
>
> * Currently you cannot click a text area in the format post window to have it auto-fill with a value, im working on it :)
>
> - v6.0 - 11/01/2016
>
> * Huge new internal update, should make future changes much easier but may have caused a few bugs, please do report them if you see them!

So im keen to do new features and upgrades so email me if you have suggestions!

As always you can grab Post To Tumblr over at the Chrome Web Store: [https://chrome.google.com/webstore/detail/post-to-tumblr/dbpicbbcpanckagpdjflgojlknomoiah](https://chrome.google.com/webstore/detail/post-to-tumblr/dbpicbbcpanckagpdjflgojlknomoiah)