Project GamOS Postmortem

Project GamOS was the most ambitious project I have embarked upon as a solo creator. A tight marriage of hardware and software, it exercised every technical skill I possess from soldering and circuit design, to efficient embedded programming, to CAD and 3D printing.

The project was crafted as a bespoke item, intended as a gift for a singular individual. Every aspect of it was designed around that intention. Sadly, it never will be given to that person, and so there is no reason for continued development on the project, or continued secrecy. It will forever be unfinished, and its components will likely be broken up and recycled into future projects instead.

As much as I can, I will also avoid discussing the emotional reasoning behind design decisions of the project. Rather, this is a place to celebrate the project itself. Although it will never exist in the manner I envisioned, I hope this document gives some insight into the ideas, development, and love that I poured in. In at least this manner, it will live on.

NB: A knowledge of digital circuitry and coding is expected for readers of this document. I’m going to throw around some abbreviations and terms of art without really defining them.

Table of Contents

1. Introduction and Project Overview

Project GamOS was an electronic puzzle box, where each puzzle was geofenced to a particular location in Milwaukee, Wisconsin. These locations represented significant locations to me and the intended recipient. Each location had a puzzle with a completion condition, and upon solving them, the box would open.

The box itself would have a map of Milwaukee on one side, and each of the other three sides would contain all the components needed for one location’s puzzle. The top of the box would open, the mechanism of which is discussed further in section 8.

Design began on the project by brainstorming locations, which resulted in the following list:

Due to the geography of Milwaukee, only some of these locations could cohesively work together. I needed to select locations that would be nicely distributed within a square space to represent as a map of the city. This immediately disqualified the Seven Bridges Trail, which lay too far south to comfortably include in a square segment of the city.

The other key selection aspect was determining a good puzzle to fit with the location. The puzzle needed some purpose for existing in that particular location, depending on environment and contextual clues to suggest the answer. Through this selection process, the original design settled on three location/puzzle pairs listed below. These are as described in the original project spec, and during development some of these specs began to alter for feasibility.

Puzzle One: Fuel Cafe

Fuel Cafe Center Street is a delightful grunge coffee shop in the Riverwest neighborhood of Milwaukee. The neon signs advertising “Killer Coffee Lousy Service” do not lie. They’ve got a great food menu as well.

One wall of the cafe is a giant mural of a line of motorbike racers. Each of their bikes has a number on it. The puzzle here is that the box itself had a keypad, and the sequence of numbers from the mural would have to be entered in.

Puzzle Two: Art Museum

The art museum puzzle would be a mini scavenger hunt for specific paintings. Once at the art museum, a low-resolution 8x8 representation of the painting would be shown on the NeoPixel matrix. This would be the hint for which painting to find. To verify discovery of the painting, the onboard camera would be pointed at the painting and a photo taken. Through some simple, lazy image recognition scheme, it would be determined that the right painting was discovered. After finding maybe three paintings, the puzzle would complete.

Puzzle Three: Fish Mural

This is an animated light-up mural on the side of the Riverside Theater in Milwaukee. The best place to view it is from the East bank of the Milwaukee River along Water St, which is the geofenced area.

The puzzle on the box would depict an engraved solar arc, with an RGB LED placed along the arc at the horizon for sunset. As long as the box was powered on, the color of the LED would update hourly, from a dark navy blue overnight, moving through purples and reds in the morning, white during the day, and then back again in the evening. Together, this would suggest the solution: the box needed to be brought to that location at sundown, which would solve the puzzle. It had no further interaction, only the time- and location-based aspects.

2. Hardware Bill of Materials

Where possible the links lead to where I sourced the component. The price represents the total cost for the quantity of an item used in the project, not a single unit cost. This list also does not include the solder, wire, 3D printed components, singular components such as resistors, or other tools necessary for the build.

Item Qty Price
Arduino Uno 1 $35
Adafruit Ultimate GPS Breakout 1 $39.95
6600MAh Li-Ion Battery 1 $29.50
PowerBoost 1000 Charger 1 $19.95
8x8 NeoPixel Grid 1 $34.95
Tactile Pushbutton 9 $1.25
DC Motor 130 size 1 $1.95
ArduCam OV2640 2MP Camera 1 $25.99
Red 5mm LED 3 $0.48
RGB 5mm LED 1 $0.50
MCP23008 i2c Port Expander 1 $1.95
SPDT Switch 1 $0.95
3.5mm Mono Headphone Jack 1 $1.25
Dual H-Bridge Motor Driver 1 $2.95

3. Wiring and Schematics

The following is the pinout for the Arduino Uno. The project as fully described would have exceeded the pins of the Arduino Uno itself, and so an 8-pin I2C GPIO expander was also included.

Arduino Pinout

Pin Mode Description
0 TX TX GPS
1 RX RX GPS
2 INPUT MCP23008 Interrupt
~3 OUTPUT RGB R
4 DOUT NeoPixel DOUT
~5 OUTPUT RGB G
~6 OUTPUT RGB B
7 INPUT Phone Jack connect
8 OUTPUT Map LED Museum
~9 INPUT Low Battery
~10 Camera SS
~11/MOSI MOSI Camera MOSI
12/MISO MISO Camera MISO
13/SCK SCK Camera SCK
14/A0 ANALOG Phone Jack Mono
15/A1 OUTPUT Map LED Fuel
16/A2 OUTPUT Map LED Kawa
17/A3 OUTPUT Map LED Fish
18/A4/SDA SDA MCP23008 SDA, Camera SDA
19/A5/SCL SCL MCP23008 SCL, Camera SCL

Pin Expander Pinout

Pin Mode Description
0 OUTPUT KEYPAD A
1 OUTPUT KEYPAD B
2 OUTPUT KEYPAD C
3 INPUT_PULLUP KEYPAD 1
4 INPUT_PULLUP KEYPAD 2
5 INPUT_PULLUP KEYPAD 3
6 OUTPUT DC H-Bridge Forward
7 OUTPUT DC H-Bridge Reverse

4: Firmware Architecture

5: Keypad Prototype

An n-by-m keypad is easy to construct with n * m switches, and requires n + m pins to control. A matrix of wires runs horizontally and vertically, and each switch rests at the meeting point of a horizontal and a vertical wire. When the switch is pushed the two wires are bridged. This was easily prototyped with some tactile buttons soldered to perfboard. Since the Fuel Cafe mural did not include any 0’s, I was able to construct the keypad as a 3x3 matrix, requiring 6 pins to drive.

picture of keypad

In my exploration of the most efficient way to run a keypad, I found some description of a design using an array of different values of resistor such that each individual button press would provide a distinct resistance to the current flowing through. This could then be reversed based on the known 5v logic level to determine which key was pressed. Were I to build the project again I likely would use such a design to reduce the keypad to a single analog-in wire. Such a design results in a mild increase in component use and firmware complexity but saves the use of 5 pins, which would have brought me very close to not requiring the 8-pin expansion.

An ongoing mantra of mine in the project was to “keep it simple”. However definitions of simple can vary. The simple button matrix felt “simple” at the time, in the sense it was easy to reason about and was a purely digital solution. Using an analog pin to measure varying voltage felt, at the time, more complex. In retrospect, releasing the 5 additional pins would have been a greater save in complexity.

The gist of the software is to drive one axis of wires high one at a time, and then scan across the inputs of the other axis of wires. When a switch is pressed, it will short a pair of these together which will indicate which key was pressed. Once implemented in such a manner however, I found that Arduino Uno INPUT pins will tend to pick up random noise from the environment and thus not consistently read 0 without the use of a pulldown resistor to tie the pin to ground. Adding pulldown resistors seemed like too much work, so I just flipped the logic instead: The outputs would be driven high by default, and then one by one be switched to low. The input pins were configured to INPUT_PULLUP so the closing of the switch would pull the pin low and trigger a button press.

Even once this change was made in software, I was still getting random presses of certain buttons during testing. It turns out, while soldering the keypad together, the insulation of some of the wires melted, and was causing random shorts even when buttons weren’t pressed. At this time, I didn’t have a multimeter, so this was found by prying the wires apart as much as possible until the input stopped. A bit of Kapton tape between the wires prevented further shorts, and the keypad prototype was fully functional.

Here is the source code for the keypad driver prototype. I was still toying with different possible modular styles and had not settled on a firmware architecture at this point. Since the keypad was originally intended just for solving the Fuel Cafe puzzle, the controlling code is wrapped in a class called Fuel.

// Fuel.h

#define KEYPAD_A 10
#define KEYPAD_B 11
#define KEYPAD_C 12

#define KEYPAD_1 2
#define KEYPAD_2 3
#define KEYPAD_3 4

#include <stdint.h>

class Fuel {
  public:
    Fuel();

    void fuel_loop();
  private:
    // This maps keypad matrix coordinates to the number of the key
    static const uint8_t keypadMap[3][3] = { {1, 4, 7}, {2, 5, 8} , {3, 6, 9} };
    // The answer sequence that needs to be entered to unlock
    static const uint8_t answerSequence[12] = {2, 2, 3, 7, 6, 4, 7, 5, 7, 6, 8, 8};

    uint8_t scanKeypad();
};

// Fuel.cpp

#include "Fuel.h"

Fuel::Fuel(){
  pinMode(KEYPAD_A, OUTPUT);
  pinMode(KEYPAD_B, OUTPUT);
  pinMode(KEYPAD_C, OUTPUT);

  digitalWrite(KEYPAD_A, HIGH);
  digitalWrite(KEYPAD_B, HIGH);
  digitalWrite(KEYPAD_C, HIGH);

  pinMode(KEYPAD_1, INPUT_PULLUP);
  pinMode(KEYPAD_2, INPUT_PULLUP);
  pinMode(KEYPAD_3, INPUT_PULLUP);
}

void Fuel::fuel_loop(){
  int key = scanKeypad();
}

uint8_t Fuel::scanKeypad(){
  uint8_t pressed = 0;
  // drive columns low, and check for low rows indicating closed button
  for (uint8_t i = KEYPAD_A; i <= KEYPAD_C; i++) {
    digitalWrite(i, LOW);
    // check each row for a closed circuit
    // note since input is pullup, a closed circuit pulls LOW
    for (int j = KEYPAD_1; j <= KEYPAD_3; j++) {
      if (digitalRead(j) == LOW){
        // index for keypad array has to be offset by keypad pin number
        pressed = (keypadMap[i - KEYPAD_A][j - KEYPAD_1]);
        // ignore held buttons
        while (digitalRead(j) == LOW){
          delay(50);
        }
      }
    }
    digitalWrite(i, HIGH);
  }

  return pressed;
}

6: Map Prototype

The map prototype was incredibly satisfying to get working, since it pulled in several different components into one unit that was actually in some sense portable. This prototype consisted of the lithium battery powering the entire device, with the GPS module and NeoPixel grid working together to update location in real-time. To display the current location, I printed out the line art of Milwaukee I had drawn for engraving on the final project, and taped it over the NeoPixel grid. As I moved around the city, the position of the lit LED would change corresponding to my location.

It was not very precise as the bounds of the map I had printed were not quite aligned with the coordinates I had programmed as my edges, but it worked. This test proved the viability of the GPS module including realtime updating, the NeoPixel grid, and the entire battery power system. This was the point where the project really felt not only technically feasible, but within my immediate grasp. The entire thing was carried around in a plastic food container, and still delicate since some connections were simply stripped wire pushed into the female headers on the Arduino, but still, it really actually worked.

map prototype

Note in the final version of the project the GPS module would be attached to the hardware serial port on pins 1 and 2. However at this stage, USB debugging is still needed so this test module uses the SoftwareSerial library.

/*
   This is a test bed for GPS data while out and about. The GPS updates
   on an interrupt and then blinks an LED on the neopixel grid to show
   where you are located. This should roughly correspond to the map of
   the final box, but there's twice the linear resolution here than there
   will be.
*/

#include <Adafruit_GPS.h>
#include <Adafruit_NeoPixel.h>
#include <SoftwareSerial.h>

#define GPSECHO false
#define GPS_TX 3
#define GPS_RX 2

#define PIXEL_PIN 6
#define NUM_PIXELS 64

SoftwareSerial mySerial(GPS_TX, GPS_RX);

Adafruit_GPS GPS(&mySerial);

float root_lat = 43.070;
float root_lon = 87.910;

struct lat_lon_coord{
  float lat;
  float lon;
};

Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUM_PIXELS, PIXEL_PIN, NEO_GRB + NEO_KHZ800);
int curr_pixel = 0;


void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  GPS.begin(9600);
  // get minimum only data from the GPS since we don't need anything more
  GPS.sendCommand(PMTK_SET_NMEA_OUTPUT_RMCONLY);

  GPS.sendCommand(PMTK_SET_NMEA_UPDATE_1HZ); // 1hz update rate

  // interrupt stuff, I copied from Adafruit "parsing" example
  OCR0A = 0xAF; //sets compare register for timer0
  TIMSK0 |= _BV(OCIE0A); //sets timer interrupt bit


  pixels.begin();
  delay(1000);
}

SIGNAL(TIMER0_COMPA_vect){
  char c = GPS.read();

  #ifdef UDR0
    if (GPSECHO)
      if (c) UDR0 = c;
  #endif //UDR0
}

uint32_t timer = millis();

void loop() {
  if (GPS.newNMEAreceived()){
    if (!GPS.parse(GPS.lastNMEA())) return; //failed to parse sentence
  }

  if (timer > millis()) timer = millis();

  if (millis() - timer > 2000){
    timer = millis();
    Serial.print("Fix: "); Serial.println((int)GPS.fix);

    if (GPS.fix){
      lat_lon_coord c;
      c.lat = GPS.latitudeDegrees;
      c.lon = GPS.longitudeDegrees * -1.0;
      Serial.print("Latitude: ");
      Serial.print(c.lat, 3);
      Serial.print(" Longitude: ");
      Serial.print(c.lon, 3);
      Serial.print(" Pixel index: ");
      update_pixel(coord_to_pixel_index(c));
      Serial.println(curr_pixel);
    }
  }
}

int coord_to_pixel_index(lat_lon_coord c){
  int y = round((root_lat - c.lat) * 200);
  int x = round((root_lon - c.lon) * 200);
  Serial.print("X: "); Serial.print(x);
  Serial.print(" Y: "); Serial.println(y);

  // lazy way of keeping in bounds, do better Owen
  if (x > 7) x = 7;
  if (x < 0) x = 0;
  if (y > 7) y = 7;
  if (y < 0) y = 0;

  // converts to a single int for pixel addressing
  return (x + (8*y));
}

void update_pixel(int new_pixel){
  if (new_pixel != curr_pixel){
    pixels.setPixelColor(curr_pixel, pixels.Color(0,0,0)); // turn off old
    pixels.setPixelColor(new_pixel, pixels.Color(40,0,0)); // turn on new
    curr_pixel = new_pixel;
  }
  pixels.show();
}

7: Solar Prototype

8: Enclosure Design

9: The Mixtape, and a new puzzle

As a gift, I recorded an actual honest mixtape onto a cassette. My selection of songs only filled one side of the cassette, leaving the second side blank and open to possibilities.

I had long been interested in the possibilities of encoding arbitrary data as a sound wave. Back in the Commodore 64 days, programs used to come on cassette tapes which would have to be decoded into data. Tons of encoding methods were possible, and I pored over documentation of these various methods.

In my pursuit of this, I began development on an audio encoding format that I came to call WAVD. It would be encoded like a traditional uncompressed PWM .wav file, but would contain arbitrary data rather than audio intended for listening. The WAVD codec is still under development, and its implementation using overlapping frequencies to encode a byte at a time would prove too challenging to decode on an Arduino Uno, since it requires discrete Fourier transforms. Details on it will be published some time.

Ultimately I settled on the Manchester coding which uses a high-low transition to encode a binary 1, and a low-high transition to encode a binary 0.

The mixtape ended up being themed “sushi” which also led to it potentially becoming a puzzle for Kawa Sushi, one of the locations I had been unable to come up with a deserving puzzle for. As such, I also named the encoder the “sushi encoder.” With a little work it could be turned into a generic CLI to encode any passed file, but since I was looking to quickly encode just a single file, I just hard coded the file names. I also did not write a python decoder.

The data to be encoded was an 8x8 pixel art image of a salmon nigiri sushi. As the data was read into Project GamOS, the pixels would light up on the NeoPixel matrix to show the sushi image, and then the puzzle would complete.

# sushi encoder
# encodes binary data in manchester coding to a wav file
# written with love by owen monsma
# with respect to g e thomas

import wave, struct

def encodeByte(byte):
    encoded_byte = []
    for b in bits(byte):
        # g e thomas convention: 1 bit high-low, 0 bit low-high
        if b: # high 1 -> high-low
            encoded_byte.extend([1,0])
        else: # low 0 -> low-high
            encoded_byte.extend([0,1])
    return encoded_byte;

def bits(b):
    for i in range(8):
        yield (b >> i) & 1


sample_rate = 44100 # Hz
clock_rate = 300 # Hz

amplitude = 30000 # almost full volume, max is 32767

# Number of samples per encoded bit
bit_width = int(sample_rate / clock_rate)

ifile = "img.sush"
ofile = "sushi_wav.wav"

# probably a bad habit if the file is big, but it isnt
bytes_in = open(ifile, 'rb').read()

bytes_out = []

for byte in bytes_in:
    bytes_out.extend(encodeByte(byte))

with wave.open(ofile, 'wb') as w:
    w.setnchannels(1) # mono
    w.setsampwidth(2) # 16-bit audio is standard
    w.setframerate(sample_rate) # 44100 is probably overkill

    for b in bytes_out:
        if b: # if it's a 1 write peaks
            for i in range (bit_width):
                data = struct.pack('<h', amplitude)
                w.writeframesraw(data)
        else: #if 0 write valleys
            for i in range (bit_width):
                data = struct.pack('<h', -amplitude)
                w.writeframesraw(data)

The decoding then was the next step of the puzzle. The Arduino has analog in but expects a signal normalized to one of several standard voltage levels, e.g. between 0 and 5v. The voltage that is used to drive headphones through an aux cable is actually centered around 0, with the waveform going positive and negative. The Arduino would not be capable of reading the negative voltages.

I explored building a DC offset that would boost the signal by 2.5v, centering it in the default 0-5v range. After further thought I determined this likely should not be necessary: the encoding cares about the point of 0 crossing so the negative portions of the wave don’t really matter. They could be trimmed and I simply would have to track the point of when the signal crosses to and from 0.

Briefly I toyed with the idea of handling the decoding with an FPGA that could feed nice clean digital data into the Arduino. I had backed the Fipsy project on Kickstarter, and had two Fipsy FPGA boards sitting in my shoebox of electronics waiting for a project. This idea never coalesced much further than just being a thought.

The audio encoder was one of the last major pieces of the project to be completed, during a time it was becoming increasingly more likely that I would end up cancelling development. As such the Arduino decoder was never implemented, and I had further concerns about its overall viability in the project. Would the Arduino be fast enough to read the data, would the signal survive the process of being recorded to cassette and then played back into the Arduino. As a backup, the aux jack also has a pin out that grounds when a cable is connected. If the true decoding did not work, I planned to fake it by just marking the puzzle as solved when a cable was connected for the right amount of time.

10: The Manual

Part of the complete package was a pair of physical manuals that would contain diagrams and descriptions of the box, similar to the users manuals that come with most electronics. It was going to be a 5" square and spiral bound. The first manual was a users guide, and the second a repair and service guide with detailed diagrams of the circuitry and assembly process. From the start, I had an obsessive drive to document every aspect of the box. Since the manuals needed the box itself to be in a mostly completed state before they could really be developed, they never ended up coming together.

Closing Thoughts

The project was ambitious, as I hope is evident here. Perhaps too ambitious. The first documentation of the project dates back to November of 2018. It spans sketches in notebooks, folders of source code, boxes of electronic components. It was described enthusiastically to friends and coworkers. It was a central obsession of mine for a very long time.

I have a hard time of knowing when to give up. Perhaps that's a flaw of mine. Even in the face of setbacks, I continued, convinced I could make it work. Sometimes that belief is all you have, even despite massive contradictory evidence.

In the end, I'm not even sure the project would have done its job. Perhaps it was overengineered, and its heart and soul too mechanical to truly represent the expression of human love that it was meant to. But I'll never know.

Even now, I want to finish the project, just to prove that I can. But it would exist forever in a state that it could never fulfill its only purpose. As sad as it is to end a project knowing it will never be completed, it is just as sad to imagine it completed, but meaningless. It was given meaning by the task it was meant to accomplish, a task that no longer exists.