In the spirit of Halloween and current political events, Marco and I decided to build a talking cyborg Trump head. The head is meant to detect someone’s presence in front of it and speak. We’ve been having problems hooking up the audio playback, and although it is currently not perfected (an amp is needed to increase the output volume, and a solution to the SPI chip selection causing an unresponsive servo has yet to be found), this is it’s current state:

Mr Cyborg Trump provocatively speaks a line originally spoken from Mrs Clinton: “Apologizing is a great thing, but you have to be wrong. I will absolutely apologize sometime, in the hopefully distant future, if I’m ever wrong.”

Parts

• Rubber Trump mask
• Stuffing (cotton, bubble wrap)
• Arduino UNO
• IR motion sensor
• 8 ohm speaker
• Servo motor
• Micro SD card reader and 1gb micro SD card
• 9v battery pack

 

Code (Arduino)

#include <SD.h>
#include <pcmConfig.h>
#include <pcmRF.h>
#include <TMRpcm.h>
#include <SPI.h>
#define SD_ChipSelectPin 4
TMRpcm audio;

//using a O O G resistor (3.3 ohms) for IR sensor
//dont delay less than 250 between openMouth & closedMouth

#include <Servo.h>
Servo mouth;

int speakerPin = 3; //digital out
int servoPin = 2; //digital out
int sensorPin = A0; //analog in

int closedMouth = 45;
int openMouth = 180;
int sensorStrength = 500;
unsigned long lengthOfPhrase = 9000; //in milliseconds
unsigned long timePhraseBegin = 0;
boolean movementDetected = false;
boolean speakOnce = false;

void setup() {
  audio.speakerPin = 9;
  Serial.begin(9600);
  mouth.attach(servoPin); 
  mouth.write(closedMouth);

  if (!SD.begin(SD_ChipSelectPin)) {  // see if the card is present and can be initialized:
    Serial.println("SD fail");  
    return;   // don't do anything more if not
  } else {
    Serial.println("SD success"); 
  }
  audio.setVolume(5);
}

void loop() {
  unsigned long timePassed = millis() - timePhraseBegin;
  
  int sensorRead = analogRead(sensorPin);
  
  if(sensorRead > sensorStrength){
    if(!movementDetected){
      Serial.println("ON.");
      timePhraseBegin = millis();
    }
    movementDetected = true;
  } else {
    if(movementDetected && timePassed >= lengthOfPhrase){
      Serial.println("OFF.");
      movementDetected = false;
      speakOnce = false;
    }
  }

  if(movementDetected){
    
    //move mouth
    mouth.write(openMouth);
    delay(250);
    mouth.write(closedMouth);
    delay(250);

    //speak
    if(!speakOnce){
      audio.play("0000.wav");
      speakOnce = true;
    }
    
  } else {
    //close mouth
    mouth.write(closedMouth);

    delay(250);
  }
}

Synesthesia VR

Synopsis

Escape into another world.

Synesthesia VR is a device that enables you to experience your surroundings in a new and exciting way. By immersing yourself in this isolated virtual space, you are liberated from the shackles of order and illusions of comprehension, and freed into the higher realm of pure data.

The headset equips you with your very own prosthetic aural and visual sensory inputs which exchange data before feeding into your native senses, in order to ensure complete lack of comprehension of, and thus disconnection from our mundane reality.

Parts

• Arduino UNO
• SparkFun Sound Detector
• Serial Miniature LCD Module – 1.44″ (uLCD-144-G2 GFX) (x2)
• SparkFun RGB Light Sensor
• 3.5mm Stereo Audio Jack (Panel Mount)
• 9V battery and battery pack with power switch
• Cardboard VR viewer

Code

The final Arduino code is below. It uses the Goldelox 4D library (Goldelox is the name of the graphics processor for the LCD displays used, this library implements what the 4D company calls “4DGL” which makes interfacing with the displays very easy), as well as a dedicated library for Sparkfun’s SFE_ISL29125 RGB light sensor. Code for the sound detector can be referenced from here.

/*
 * __________//______//______//____/////____/////__
 * _______////____///__///__//__________//__________
 * ____//____//__///////__//__________//__________
 * ___//////__//__/__//__//__________//__________
 * __//____//__//______//____/////____/////__
 * ___________________________________________________________
 * __________ Copyright (c) 2016 Andrew McCausland __________
 * ________________ <andrewmccausland.net> _________________
 * ________________________________________________________
 * 
 * To upload new code:
 * 1. Disconnect main display (the one that's directly hooked to RX/TX)
 * 2. Disconnect TX connection to display 2
 * 3. Upload
 * 4. Reconnect main display, wait for it to begin visualization
 * 5. Reconnect TX connection to display 2.
 * 
 */

// ------------------------------ visual output (display stuff)
#include "Goldelox_Serial_4DLib.h"
#include "Goldelox_const4D.h"
#define DisplaySerial Serial // The compiler will replace any mention of DisplaySerial with the value Serial 
Goldelox_Serial_4DLib Display(&DisplaySerial);
int width = 127;
int height = 127;
int visOutSwitch = 0;

// ------------------------------ sound input stuff
#define PIN_GATE_IN 2
#define IRQ_GATE_IN  3
#define PIN_LED_OUT 13
#define PIN_ANALOG_IN A0

// ----------------------------- visual input (rgb sensor)

#include <Wire.h>
#include "SFE_ISL29125.h"
SFE_ISL29125 RGB_sensor;

// ----------------------------- sound output

int auxOutPin = 6;
int auxOutSwitch = 0;

void setup() {
  
  // ------------------------------ visual output (displays)
  Display.Callback4D = mycallback;
  Display.TimeLimit4D = 5000;
  DisplaySerial.begin(9600);

  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  
  delay (10000); //back up buffer time to let the display start up

  Display.gfx_ScreenMode(LANDSCAPE);

  // ------------------------------ sound input
  pinMode(PIN_LED_OUT, OUTPUT);
  pinMode(PIN_GATE_IN, INPUT);
  attachInterrupt(IRQ_GATE_IN, soundISR, CHANGE);

  // ------------------------------ visual input (rgb sensor)
  
  RGB_sensor.init();
}

void loop() {

  // ------------------------------ for sound input
  int value = analogRead(PIN_ANALOG_IN);

  // ------------------------------ for visual output (display)

  unsigned int blueColors[4] = {MIDNIGHTBLUE,BLUE,DEEPSKYBLUE,LIGHTBLUE};
  unsigned int greenColors[4] = {DARKGREEN,GREEN,GREENYELLOW,LIGHTGREEN};
  unsigned int redColors[4] = {DARKRED,CRIMSON,RED,LIGHTCORAL};
  int brightness = map(value, 0, 600, 0, 3);

  if(value > 80 && value <= 100){
    
      Display.gfx_RectangleFilled(0, 0, width, height, blueColors[brightness]);
      visOutSwitch = 0;
      
  } else if(value > 100 && value <= 200){
    
    if(visOutSwitch == 0){
      Display.gfx_RectangleFilled(0, 0, width, height, blueColors[brightness]);
      visOutSwitch++;
    } else {
      Display.gfx_RectangleFilled(0, 0, width, height, greenColors[brightness]);
      visOutSwitch = 0;
    }
    
  } else if(value > 200 && value <= 600){
    
    if(visOutSwitch == 0){
      Display.gfx_RectangleFilled(0, 0, width, height, blueColors[brightness]);
      visOutSwitch++;
    } else if(visOutSwitch == 1){
      Display.gfx_RectangleFilled(0, 0, width, height, greenColors[brightness]);
      visOutSwitch++;
    } else {
      Display.gfx_RectangleFilled(0, 0, width, height, redColors[brightness]);
      visOutSwitch = 0;
    }
    
  } else if(value > 600){
      Display.gfx_RectangleFilled(0, 0, width, height, WHITE);
      visOutSwitch = 0;
  }

  if(value > 5){
    Display.gfx_Cls(); //clear the screen
  }


  // ------------------------------ for visual input (rgb sensor)
  unsigned int red = RGB_sensor.readRed();
  unsigned int green = RGB_sensor.readGreen();
  unsigned int blue = RGB_sensor.readBlue();

  // ----------------------------- for sound output
  if(auxOutSwitch == 0){
    tone(auxOutPin, red,200);
    auxOutSwitch++;
  } else if (auxOutSwitch == 1){
    tone(auxOutPin, green,200);
    auxOutSwitch++;
  } else {
    tone(auxOutPin, blue,200);
    auxOutSwitch = 0;
  }

}

// ------------------------------ for display
void mycallback(int ErrCode, unsigned char Errorbyte) {
  // Pin 13 has an LED connected on most Arduino boards. Just give it a name
  int led = 13;
  pinMode(led, OUTPUT);
  while(1){
    digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
    digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  }
}

// ------------------------------ for sound input
void soundISR() {
  int pin_val;

  pin_val = digitalRead(PIN_GATE_IN);
  digitalWrite(PIN_LED_OUT, pin_val);   
}

I want to use what I’ve learned in this class to focus on approaching the use of video and sound in unique ways. So for my final project I want to build some sort of visual and/or auditory headset that transforms the user’s surroundings in unexpected ways, perhaps enabling them to see an aspect of reality that the naked human body isn’t capable of perceiving.

In the spirit of current VR hype, I will build a headset that simulates an audio-visual synesthetic experience, for any curious individuals interested in having a temporarily destroyed sensorium while running the risk of bumping into walls, stubbing their toes or meandering out into traffic in a manner similar to how someone in the throes of an intense psychedelic experience might do so.

The headset will consist of a visual input (camera) and an audio input (sound detector) hooked up to an Arduino Uno, which will run a program that will take the sound data and output it to a color LCD display or two, and take the camera image data and output it to a standard (3.5mm) headphone jack with which the user listen to with their own headphones.

Video to audio: I will translate the color information from the camera image at each frame to tones — a tone for each color. It could either calculate the most prevalent color and translate that into one tone, or calculate all existing colors in each image and translate them into a polyphonic set of tones, or anything in between.

Audio to video: Amplitude of sound detected will translate to image opacity on the display, and pitch/frequency will translate to hue.

Bill of Materials:

System Diagram:

Roadmap:

Nov 16:

• Have purchased all parts (or at least enough to begin assembly)

Nov 23:

• Have begun writing and testing code portion
• Continue circuit assembly
• Begin circuit housing / headset assembly

Nov 30:

• Have completed code portion
• Finalizing circuit assembly, begin user testing
• Continue circuit housing / headset assembly

Dec 7:

• Apply changes from user testing
• Finalize circuit housing / headset assembly
• Prepare presentation, finalize documentation

Dec 14:

• Present

Servo

Variable resistor (FSR) triggering servo rotation using the code given in the lab.

A display pedestal, activated via push-button. Behold — coffee.

---

 

Tone

Following the tone lab I created a noise instrument using two 8 ohm speakers, four buttons hooked up to analog pins, one button hooked up to digital, and three potentiometers.

Controls from top to bottom: volume, four notes (A2, C2, E2, G2), a fifth note or variable frequency controlled by the potentiometer just below it, and finally a delay effect.

The code for this setup (pitches.h file on this page):

#include "pitches.h"

const int threshold = 10;
const int speakerPin = 8;
const int noteDuration = 20;
 
int notes[] = { NOTE_A2, NOTE_C2, NOTE_E2, NOTE_G2 };

void setup() {}
 
void loop() {
  for (int thisSensor = 0; thisSensor < 4; thisSensor++) {    

  int sensorReading = analogRead(thisSensor);
   
  if (sensorReading > threshold) {
        tone(speakerPin, notes[thisSensor], 20);
    }
  }

  int sensorReading2 = map(analogRead(4), 662, 1023, 50, 2000);
  if(sensorReading2 < 0){
    sensorReading2 = 0;
  }
  if(digitalRead(3) == HIGH){
    tone(speakerPin, sensorReading2, 20);
  }

  int sensorReading3 = map(analogRead(5), 662, 1023, 0, 100);
  if(sensorReading3 < 0){
    sensorReading3 = 0;
  }
  delay(sensorReading3);
}

 

Response to The Art of Interactive Design: A Euphonious and Illuminating Guide to Building Successful Software by Chris Crawford, and A Brief Rant on the Future of Interaction Design by Bret Victor.

---

What is physical interaction?

Interaction is essentially two-way communication. Chris Crawford likened human-computer interaction to a conversation, breaking down the communication cycle into discrete parts: listening, thinking, and responding (input, process, and output, respectively). “Physical interaction” specifically emphasizes the listening and responding aspects of the cycle, as the only way to listen and respond is through some form of physical contact.

What makes for good physical interaction?

An important point Crawford makes regarding his definition of interaction is that not all interactions are created equal — there are varying degrees for which things are able to receive, process, and respond to given information effectively; some things are considered to have high interactivity while others low. Our current landscape of interactive tech is dominated mostly by the visual and less so the audible information in the form of “two-and-one-half dimensional” (as Crawford accurately poised the common display as “a stack of partially overlapping planar images”) UIs. Touch has also been a vital aspect of such software interaction, but still with extremely low interactivity according o Crawford’s logic. This is exactly the point designer Bret Victor makes in his famous A Brief Rant on the Future of Interaction Design essay — touch interactivity is abysmally low, while the emphasis on vision as the main communication medium is unwarranted. Victor points out the reality of how our senses function together: “when working with your hands, touch does the driving, and vision helps out from the back seat.” Instead, our hands (and, by extension, the rest of our bodies) are not nearly being used to their full potential.

Good physical interaction would be that which utilizes the human sensorium to its full potential, incorporating it as effectively as possible into the listening and responding aspects of the interaction cycle.

Does digital technology have anything new or revolutionary to offer which isn’t interactive?

I agree with Crawford’s statement that interactivity is software’s competitive advantage against other mediums, but digital technology as a whole has absolutely improved on the non-interactive mediums that predate it, through its fundamentally algorithmic properties. Its influence can be seen across the board, from music to visual art. It has also enabled various mediums to interact with one another in ways that weren’t necessarily able to be realized before.