take1:

take2:

screen version:

For the remainder of this class I wanted to explore the possibilities in creating some sort of online performative visual experience. I wanted to see what could be done by harnessing the masses of content available on the web. I was initially interested in thinking about the internet as something that was “collapsing” time from a cultural perspective facilitated by its massive and cheap capacity to preserve and distribute media—

What was your first experience of going online? … What about Google image search? “I specifically remember my first image search. I searched for trees. Like, “green trees.” And it was really overwhelming because there were so many pictures of trees, all at once. Before, you could only look at books.” (src)

art history has been “flattened” for artists of a certain age. “When they typed in ‘tree’ ” in a search engine, … “they got a thousand pictures of a tree: a picture of a tree made in the 18th century, a tree made last year, a cartoon of a tree. You have this flattening of time.” (src)

But what if this isn’t actually the case? When we Google something or look at one of our many content feeds on various social or news platforms, it’s all about *now*:

Digital media is truly a time-based medium, which, given a screen’s refresh cycle and the dynamic flow of information in cyberspace, turns images, sounds, and text into discrete moments in time. These images are frozen for human eyes only.(src)

References:

Edward Shenk “Theorist” — Visual artist leveraging right-leaning conspiracy theorist type of content that he apparently finds often on facebook and can probably be recognized to be more or less prevalent on the web by most people. He breaks this style of image-making down to its formal qualities, revealing the sort of sinister underlying tone… “There’s this manic connection-making like in the darker parts of A Beautiful Mind. If something looks like something else then that is proof enough. There is no such thing as pure coincidence, and that’s a hallmark of paranoia.” (src)

XTAL fSCK video performance — The visuals these performers create exemplify the “default” visual effects inherent in the MacOS UI: mashing [ctrl+alt+cmd+8] to use the color-inversion accessibility feature to create a strobe effect, using Quicktime’s screen recording feature to create endless recursion, swiping between desktop spaces, opening and minimizing windows with genie-effect in slow-motion, emojis and iconography are all overlaid and mashed together ad nauseum. The way the MacOS UI is exposed in this formal performative context reveals and accentuates visual qualities that were hidden in plain sight, forcing the audience to realize age-old questions of artistic merit and technological control in a context that we find ourselves in every day but perhaps overlook.

Anne Hirsch “Horny Lil Feminist”

Aaron David Ross “Deceptionista”

Tiri, Lin and I really wanted to use the 9th floor studio (or a similar setup — we ended up using the micro studio) and decided to explore the possibility of livestreaming with 360 video in this context. The idea we thought would be most interesting was the first that came to mind: presenting a how-to for what we were doing in the form of a Coding Train episode.

Our final production pipeline ended up being pretty simple: Theta S (360 camera) -> OBS (streaming client) -> YouTube. Before getting to this point we had tried many different avenues. Our first approach was to embed the stream into our own webpage and use a javascript library called Aframe to VR-ify the stream content. The first problem with this was cross-origin header issues while embedding. We spoke to Rubin about this and he explained to us that YouTube delivers streams via GET as HLS which doesn’t have native support in Chrome, but more importantly solving the cross-origin issue required each user to install browser extensions which isn’t ideal. We were able to embed Twitch streams but couldn’t block ads on load and couldn’t get a proper .mp4 or .m3u8 (HLS) stream url to work with Aframe because Twitch embeds a webpage via iframe, effectively obscuring the actual video url. At the end of the day, Youtube has a built-in VR feature and essentially plug-and-play 360-video streaming capabilities, so it made no sense to build our own custom implementation.

The livestream imagery is composed of several layers of images chroma-keyed together with several green screens.

Our first test, simply getting a livestream going:

Then managing a 360 livestream:

The final output (viewer’s POV):

Final output (flattened, OBS POV):

I really appreciate the non-antialiased, jagged style of digital imagery so my goal for this assignment was to create some sort of collaborative tool for exploring glitchy, abstract visuals taking advantage of this style.

The user draws horizontal or vertical lines (even numbered connection counts are assigned horizontal lines, odds are vertical) along the small 200px canvas by dragging the mouse button. So for users creating vertical lines, at each mouse position along the x-axis of the canvas, a for loop iterates through each pixel in the column and fills it with a 1px black rectangle. In order to keep the canvas from just being filled completely black I made it so that pixels that are already black are turned white.

The result is a lot of unexpected behavior, rendering each canvas image unique, even between users on the same canvas.

The algorithm for filling pixels is as follows:

canvas.addEventListener('mousemove', function(evt) {

	//when mouse is pressed
	if (isMouseDown) {

		//draw pixel at current mousepos:
		if (numUsers % 2 == 0) { //if even number of users
			//set the user to horizontal
			drawLineHor_(evt.clientY, context);
		} else { //if odd number of users
			//set the user to vertical
			drawLineVer_(evt.clientX, context);
		}

		//send drawing
		var objtosend = {
			x: evt.clientX,
			y: evt.clientY,
			px: px,
			py: py,
			userNum: numUsers
		};
		socket.emit('drawing', objtosend);
	}
	//receive drawing
    socket.on('drawing', function(receivedData) {
        if (receivedData.userNum % 2 == 0) { //if this user is even
            //set the user to horizontal
            drawLineHor(receivedData.y, context);
        } else { // if this user is odd
            //set the user to vertical
            drawLineVer(receivedData.x, context);
        }
    });
});

//these functions could probably all be consolidated somehow
function drawLineHor(y, ctx) {
    for (var x = 0; x < canvas.width; x++) {

        //get color
        var p = ctx.getImageData(x, y, 1, 1).data;

        //compare color
        if (p == "0,0,0,255") { //if black
            //set to clear
            ctx.fillStyle = white;
        } else { //if clear
            ctx.fillStyle = black;
        }

        //fill this pixel
        ctx.fillRect(x, y, 1, 1);
    }
}

function drawLineVer(x, ctx) {

    for (var y = 0; y < canvas.height; y++) {

        //get color
        var p = ctx.getImageData(x, y, 1, 1).data;

        //compare color
        if (p == "0,0,0,255") { //if black
            //set to clear
            ctx.fillStyle = white;
        } else { //if clear
            ctx.fillStyle = black;
        }

        //fill this pixel
        ctx.fillRect(x, y, 1, 1);
    }
}

function drawLineHor_(y, ctx) {
    for (var x = 0; x < canvas.width; x++) {
        ctx.fillStyle = black;
        ctx.fillRect(x, y, 1, 1);
    }
}

function drawLineVer_(x, ctx) {

    for (var y = 0; y < canvas.height; y++) {
        ctx.fillStyle = black;
        ctx.fillRect(x, y, 1, 1);
    }
}

I found that there is no way to get perfect, 1px non-antialiased lines in a canvas with the built-in drawing methods. One trick is to translate the entire canvas half a pixel (ctx.translate(0.5, 0.5)) to offset the interpolation that causes 1px lines to fill 2px, but this still doesn’t keep lines 1px when drawn at angles. So what I did instead (which also allowed me individual pixel control) was make 1px sized rects with  ctx.fillRect(x,y,1,1). The canvas thus is so small because this requires large for loops which is detrimental to performance.

For the remainder of this class I’ll attempt to discover the sort of “hidden style” of not just the structures of interfaces in digital media (windows, buttons, menus, scrolling… GUI paradigms in general) but of the content itself that fills these structures, specifically on the web. Because content on the web (image, video, and text, interactive or not) encapsulates objects, people, places, and ideas, from arguably every period of human history, could the resulting aesthetic vernacular really be considered an overarching style of everything, a mess of clashing styles and perspectives from wildly different paradigms? Is this what defines our current zeitgeist, and if so, what does that mean for the future? What could possibly come after this massive collapse of time and space in aesthetic understanding?

This kind of flattening seems to have an affect on many things, like political perspectives, cultural diversity (global homogenization in fashion, language, icons, etc), and maybe even more. So not only could the web be considered an “artistic medium” but it’s also arguably the primary source of information dissemination and media consumption for many people all over the world.

I will realize this using appropriated media as facilitated by the use of web APIs for various content aggregators and media platforms. I’ll try to focus on content “hubs” to minimize bias but it will be interesting to see the bias inherent in my choosing regardless, if not instrumental in helping to discover to what degree the web today acts as a window into any sort of objective paradigm of everything.

References:

What was your first experience of going online? … What about Google image search? “I specifically remember my first image search. I searched for trees. Like, “green trees.” And it was really overwhelming because there were so many pictures of trees, all at once. Before, you could only look at books.” (src)

art history has been “flattened” for artists of a certain age. “When they typed in ‘tree’ ” in a search engine, … “they got a thousand pictures of a tree: a picture of a tree made in the 18th century, a tree made last year, a cartoon of a tree. You have this flattening of time.” (src)

Edward Shenk “Theorist” — Visual artist leveraging right-leaning conspiracy theorist type of content that he apparently finds often on facebook and can probably be recognized to be more or less prevalent on the web by most people. He breaks this style of image-making down to its formal qualities, revealing the sort of sinister underlying tone… “There’s this manic connection-making like in the darker parts of A Beautiful Mind. If something looks like something else then that is proof enough. There is no such thing as pure coincidence, and that’s a hallmark of paranoia.” (src)

XTAL fSCK video performance — The visuals these performers create exemplify the “default” visual effects inherent in the MacOS UI: mashing [ctrl+alt+cmd+8] to use the color-inversion accessibility feature to create a strobe effect, using Quicktime’s screen recording feature to create endless recursion, swiping between desktop spaces, opening and minimizing windows with genie-effect in slow-motion, emojis and iconography are all overlaid and mashed together ad nauseum. The way the MacOS UI is exposed in this formal performative context reveals and accentuates visual qualities that were hidden in plain sight, forcing the audience to realize age-old questions of artistic merit and technological control in a context that we find ourselves in every day but perhaps overlook.

I’d like to take this opportunity to put the tools I’ve learned in this class thus far toward a project I’m starting to develop for Design for Discomfort. The idea is to build a series of interconnected web pages filled with visual tactics commonly found all over the web to invoke extreme sensory discomfort toward its users. I also want to address our interactions in these systems and the resulting feedback loops that affect our actions. I aim for it to serve as somewhat of a timestamp for the current state of our digital landscape, and my hope is that it will prompt people to think about this kind of emotional/behavioral stimuli we all subject ourselves to.  What comprises our contemporary digital landscape? Is it different from, say, 10 years ago? If so, how is it different? What has changed, why, and what does it say about us in terms of what we are using the web for, what we want the web to be, and what it might become in the future?

For this assignment I was most interested in understanding how image data was encoded into the base64 format. I spent a great deal of time attempting to understand exactly how this was done so that I could alter the image pixel data with code. So far I’ve been unsuccessful, and I have no idea why my current code doesn’t work, but perhaps I’m close, or maybe I’ve led myself completely astray.

First of all I learned that images are a binary data format, meaning the color of each of its pixels can be interpreted via a 1 or 0, (black or white), but in this case the png I am sending as base64 is not just 1 bit per pixel but rather 24 (I think), meaning there are 24 bits being used to represent a whole spectrum of rgb values. Knowing this, I then read up on how base64 encodes such binary data. The way I understand it is that the image takes the binary data in 24-bit chunks and divides each of those into four 6-bit chunks and assigns each of those 6-bits (which has 64 possible values) to an ascii character based on the base64 index table. In the end the long ascii string representing all the data in the image is “padded” with = or == to indicate whether the remainder of bits in the divided up binary image is either 1 bit or 2 bits.

So in order for me to alter the pixel data I thought I’d need to

1. Decode it from base64 back to raw binary
2. Do stuff to the binary (here I applied a random shuffle function)
3. Encode it back to base64

I am able to do this successfully, except that the resulting base64 data just returns a broken image. I’m not sure why this would happen assuming the new base64 data should contain the same exact number of bits (right?). I also made sure that the final base64 string begins with `data:image/png;base64,` and ends with an = or ==. Here’s the corresponding code:

index.html:

<!DOCTYPE html>
<html>
	<head>
		<title></title>
		<script type="text/javascript" src="/socket.io/socket.io.js"></script>
		<script type="text/javascript" src="skrypt.js"></script>
		<style>
			#imgcontainer {
				position:relative;
				float:left;
				width:100%;
				height:100%;
				margin:0 auto;
				border:solid 1px #000;
			}
			#imagecontainer img{
				position:relative;
				float:left;
			}
			#txtbox{
				width:100%;
				word-wrap: break-word;
			}
		</style>
	</head>
	<style>
		canvas{
			border:solid 1px #000;
		}
	</style>
	<body>
		<video id="thevideo" width="320" height="240"></video>
		<canvas id="thecanvas" width="320" height="240" style="display:none"></canvas>
		<div id="imgcontainer">
			<img id="receive" width="320" height="240">
		</div>
	</body>
</html>

skrypt.js:

// HTTP Portion
// var http = require('http');
var https = require('https');
var fs = require('fs'); // Using the filesystem module
// var httpServer = http.createServer(requestHandler);

const options = {
    key: fs.readFileSync('my-key.pem'),
    cert: fs.readFileSync('my-cert.pem')
};

var httpServer = https.createServer(options, requestHandler);
var url = require('url');
httpServer.listen(8080);

function requestHandler(req, res) {

    var parsedUrl = url.parse(req.url);
    console.log("The Request is: " + parsedUrl.pathname);

    fs.readFile(__dirname + parsedUrl.pathname,
        // Callback function for reading
        function(err, data) {
            // if there is an error
            if (err) {
                res.writeHead(500);
                return res.end('Error loading ' + parsedUrl.pathname);
            }
            // Otherwise, send the data, the contents of the file
            res.writeHead(200);
            res.end(data);
        }
    );
}


// WebSocket Portion
// WebSockets work with the HTTP server
var io = require('socket.io').listen(httpServer);

// Register a callback function to run when we have an individual connection
// This is run for each individual user that connects
io.sockets.on('connection',
    // We are given a websocket object in our function
    function(socket) {

        console.log("We have a new client: " + socket.id);

        // When this user emits, client side: socket.emit('otherevent',some data);
        socket.on('image', function(data) {
            // Data comes in as whatever was sent, including objects
            console.log("Received at server: " + data);

            var buf = Buffer.from(data, 'base64'); // Ta-da //encode the base64 to binary?

            // //turn buffer object array into uint8 array
            var uint8 = new Uint8Array(buf);
            console.log("uint8 array: " + uint8);

            // //re-sort binary array
            var arr = uint8;
            var sortedArr = shuffle(arr);
            console.log("sorted: " + sortedArr);

            //turn back into base64
            var newB64str = (new Buffer(sortedArr)).toString("base64");
            console.log("newB64str = " + newB64str);

            //try adding another `=` to end of string
            var finalB64Str = newB64str + "=";

            socket.broadcast.emit('image', finalB64Str); //send to all except sender
        });

        socket.on('disconnect', function() {
            console.log("Client has disconnected " + socket.id);
        });
    }
);

function shuffle(array) { //https://stackoverflow.com/a/2450976/1757149
    var currentIndex = array.length,
        temporaryValue, randomIndex;

    // While there remain elements to shuffle...
    while (0 !== currentIndex) {

        // Pick a remaining element...
        randomIndex = Math.floor(Math.random() * currentIndex);
        currentIndex -= 1;

        // And swap it with the current element.
        temporaryValue = array[currentIndex];
        array[currentIndex] = array[randomIndex];
        array[randomIndex] = temporaryValue;
    }

    return array;
}

server.js (this is where it’s all done):

// HTTP Portion
// var http = require('http');
var https = require('https');
var fs = require('fs'); // Using the filesystem module
// var httpServer = http.createServer(requestHandler);

const options = {
    key: fs.readFileSync('my-key.pem'),
    cert: fs.readFileSync('my-cert.pem')
};

var httpServer = https.createServer(options, requestHandler);
var url = require('url');
httpServer.listen(8080);

function requestHandler(req, res) {

    var parsedUrl = url.parse(req.url);
    console.log("The Request is: " + parsedUrl.pathname);

    fs.readFile(__dirname + parsedUrl.pathname,
        // Callback function for reading
        function(err, data) {
            // if there is an error
            if (err) {
                res.writeHead(500);
                return res.end('Error loading ' + parsedUrl.pathname);
            }
            // Otherwise, send the data, the contents of the file
            res.writeHead(200);
            res.end(data);
        }
    );
}


// WebSocket Portion
// WebSockets work with the HTTP server
var io = require('socket.io').listen(httpServer);

// Register a callback function to run when we have an individual connection
// This is run for each individual user that connects
io.sockets.on('connection',
    // We are given a websocket object in our function
    function(socket) {

        console.log("We have a new client: " + socket.id);

        // When this user emits, client side: socket.emit('otherevent',some data);
        socket.on('image', function(data) {
            // Data comes in as whatever was sent, including objects
            console.log("Received at server: " + data);

            var buf = Buffer.from(data, 'base64'); // Ta-da //encode the base64 to binary?

            // //turn buffer object array into uint8 array
            var uint8 = new Uint8Array(buf);
            console.log("uint8 array: " + uint8);

            // //re-sort binary array
            var arr = uint8;
            var sortedArr = shuffle(arr);
            console.log("sorted: " + sortedArr);

            //turn back into base64
            var newB64str = (new Buffer(sortedArr)).toString("base64");
            console.log("newB64str = " + newB64str);

            //try adding another `=` to end of string
            var finalB64Str = newB64str + "=";

            socket.broadcast.emit('image', finalB64Str); //send to all except sender
        });

        socket.on('disconnect', function() {
            console.log("Client has disconnected " + socket.id);
        });
    }
);

function shuffle(array) { //https://stackoverflow.com/a/2450976/1757149
    var currentIndex = array.length,
        temporaryValue, randomIndex;

    // While there remain elements to shuffle...
    while (0 !== currentIndex) {

        // Pick a remaining element...
        randomIndex = Math.floor(Math.random() * currentIndex);
        currentIndex -= 1;

        // And swap it with the current element.
        temporaryValue = array[currentIndex];
        array[currentIndex] = array[randomIndex];
        array[randomIndex] = temporaryValue;
    }

    return array;
}

I also attempted to perform the shuffle to the base64 encoding itself (without transforming to binary first):

// HTTP Portion
// var http = require('http');
var https = require('https');
var fs = require('fs'); // Using the filesystem module
// var httpServer = http.createServer(requestHandler);

const options = {
    key: fs.readFileSync('my-key.pem'),
    cert: fs.readFileSync('my-cert.pem')
};

var httpServer = https.createServer(options, requestHandler);
var url = require('url');
httpServer.listen(8080);

function requestHandler(req, res) {

    var parsedUrl = url.parse(req.url);
    console.log("The Request is: " + parsedUrl.pathname);

    fs.readFile(__dirname + parsedUrl.pathname,
        // Callback function for reading
        function(err, data) {
            // if there is an error
            if (err) {
                res.writeHead(500);
                return res.end('Error loading ' + parsedUrl.pathname);
            }
            // Otherwise, send the data, the contents of the file
            res.writeHead(200);
            res.end(data);
        }
    );
}


// WebSocket Portion
// WebSockets work with the HTTP server
var io = require('socket.io').listen(httpServer);

// Register a callback function to run when we have an individual connection
// This is run for each individual user that connects
io.sockets.on('connection',
    // We are given a websocket object in our function
    function(socket) {

        console.log("We have a new client: " + socket.id);

        // When this user emits, client side: socket.emit('otherevent',some data);
        socket.on('image', function(data) {
            // Data comes in as whatever was sent, including objects
            console.log("Received at server: " + data);

            var slicedData = data.slice(22); //slice off data:image/png;base64,
            console.log("sliced data: " + slicedData);

            var b64string, numEquals;
            if (slicedData.length - 2 === "=") {
                b64string = slicedData.substr(0, slicedData.length - 2); //remove last 2 `==`
                numEquals = 2;
            } else {
                b64string = slicedData.substr(0, slicedData.length - 1); //remove last `=`
                numEquals = 1;
            }
            console.log("sliced data without == : " + b64string);

            var b64array = Array.from(b64string);
            var shuffledb64array = shuffle(b64array);
            var newB64str = shuffledb64array.join('');
            console.log("newB64str: " + newB64str);

            //try adding another `=` to end of string
            var finalB64Str;
            if (numEquals == 1) {
                finalB64Str = newB64str + "=";
            } else if (numEquals == 2) {
                finalB64Str = newB64str + "==";
            } else {
                console.log("error calculating number of equal signs");
            }

            socket.broadcast.emit('image', finalB64Str); //send to all except sender
        });

        socket.on('disconnect', function() {
            console.log("Client has disconnected " + socket.id);
        });
    }
);

function shuffle(array) { //via https://stackoverflow.com/a/2450976/1757149
    var currentIndex = array.length,
        temporaryValue, randomIndex;

    // While there remain elements to shuffle...
    while (0 !== currentIndex) {

        // Pick a remaining element...
        randomIndex = Math.floor(Math.random() * currentIndex);
        currentIndex -= 1;

        // And swap it with the current element.
        temporaryValue = array[currentIndex];
        array[currentIndex] = array[randomIndex];
        array[randomIndex] = temporaryValue;
    }

    return array;
}