🎞️ Videos → TypeScript Theatre: Coding Art Performances with Algorithms, AI and Assembly

Introduction: Exploring Creative Coding Projects0:00

Let us get started. เดี๋ยว talk นี้จะเป็นภาษาอังกฤษนะฮะ This talk would be in English. Let me introduce myself a little bit. My name is ภูมิ, ภูมิปรินทร์ มะโน นะครับ Right now, I work as a software engineer at Metabase. It's a data visualization company. But in my spare time, I like to create side projects and explore different areas.

Art is one of the areas that I got into this year. I just wanted to share on that a little bit. But before we start, just curious, how many of you have heard of Minecraft or have played Minecraft before? I think a lot of us. Actually, I think Minecraft and video game modding in general is how a lot of my friends in my generation got into coding. A lot of them started modding GTA, modding Minecraft. Then they started to get this feel for when you are able to modify the code of a game and make magic happen.

I think this notion of following the fun is still how I do programming up until today. I have been organizing some events for quite a while now. Actually back from seven years ago, and I think every event I have done, it's always about one question: what can you do with code? There's a lot of frameworks, there's a lot of technologies we have today. But I think the important question is, what are you going to build with it? What are the kind of things that you can create with code? I just wanted to share my journey in this past year. I'm actually not organizing events much anymore, but I have started to get into creative coding projects.

First one is Cyber Subin, which is AI that, actually more like a dance project I did with a Thai dancer where we build software that humans can dance with machines, with algorithms. Second project is a project I did with a Singaporean artist, Ho Rui An, which is about history of human

and history of AI. The last project is actually my own side project. This is Visual Assembly Canvas. This is where I want to explore being able to write my own assembly to do audio visual projects. Let's actually get started.

Cyber Subin: Art Performance and Technology's Role2:18

For Cyber Subin, this is a project where we want to create an art performance. Art performance, if you're not sure what that is, it's basically trying to encode some meaning or some question that you want to pose to people and encode it in the form of art. In this case, the main question of Cyber Subin is, can technology and human work together? Like is AI going to replace all the Thai dancers in the world? Do we still need humans if we have AI? This is the kind of thing that we want to explore, essentially the cybernetics between human and machine.

The Cyber Subin Team and Project Origins2:55

This is the folks on our team. It all actually got started back in 2021. I was working on this hackathon called Hack Natasin or Hack Cybernetic Arts. In this hackathon, it's actually a project that was made, initiated by two persons. First is PPP or Pat. He's a researcher at MIT Media Lab. He calls himself a cyborg scientist. He's interested in how human and machine interact with each other. And then we have Pichet, which is a choreographer. His role is he wants to deconstruct Thai dance.

Deconstructing Thai Dance: A 100-Year Perspective3:32

This picture really fascinates me. This is the first picture of Natasin Thai, first capture photo of Thai dance. And in this photo is 1923, right? But what year is it right now? 2024. So it's actually 100 years apart. The interesting thing about Thai culture is it hasn't really been trying to decode or try to understand it. Like if you think about K-pop or J-pop, you see a lot of dance on TikTok, right? You see a lot of dance where people are making their own dance. They are making the dance related to the character they're interested in. But in Thai dance, the meaning is always 𝚘𝚋𝚓𝚎𝚌𝚝.𝚏𝚛𝚎𝚎𝚣𝚎. It's always frozen in time, and you cannot really do much about it.

Using Algorithms to Generate New Dance Moves4:14

In this project, we want to use algorithms to try to

decode the dance moves that we have into a series of

programs that we can use to generate new dance. That means you are not limited to the number of dance you have like Thepphanom, for example, but you're able to continuously create new dance.

The way he did this is through the systems called the six elements. He thinks every dance in the world, like African dance, like K-pop, is composed of different elements of dance. For example, the energy in your body, or external body space, which is the space that surrounds you, or access point, which is the joints, the parts that connects. He thinks this is how you can compose these elements into different dances.

Domain-Driven Design in Dance and Technology5:07

What do you do when you start a new project with people in that domain? Of course, you do domain-driven design. You need to get into that domain. What we did is we found this, basically the very first photo of Thai dance, and we found a diagram that the choreographers were using to decode Thai dance. Of these choreographers, they are people who,

in Thailand, there's not much of creating dance, but you have to understand it first before you're able to create it. What we first started to do is we started to write a simple React app that you can use to write a diagram that you can use to denote how the dance works. And there is also another project by other group. For example, my friend Pat, he did a project where he tries to understand and plot the dance from the video into a graph, just to see if there are any patterns that we can see.

Goals: Creating New Dance and Human-Machine Interaction6:04

Based on those projects, we think of two goals. First thing is we want to create new dance, because if you cannot create new things, you're always just copying. You're always just replicating what's there. And the second thing we want to explore is the interaction, the feedback between human and the machine. Can they work together or no, or they're antagonistic to each other?

Cybernetics: Feedback Loops in Dance and Missiles6:28

Let's talk about cybernetics quickly. It is actually about feedback loop. So imagine if you're shooting a missile, right? What happens if you just shoot it in a straight line? It's not going to hit the target. So you need to get the input, get the sensor data, and refine the output.

This is all about cybernetics. It's all about understanding what's around you, taking that feedback in, and then changing the course. We want to explore that concept between the dancers as well. For example, we want to take in existing dance, and we want to try to change it, modify it in some way so that we can create new dance. And we want to see how dancers would react to that. Like would they copy the dance? Would they dance with it? Like what would they do?

Technology: Three.js and Motion Capture7:16

The technology we are using today is going to be Three.js that we decided to use for this project. Three.js is actually very suitable for this kind of, let's say, POC work or the work where you want to quickly explore with the data.

This is like a CodeSandbox example with Three.js.

But the problem is this model is not ours, right? So we need some kind of existing data. And the way we do that is with motion capture. So we use a motion capture device and connect it to your joints, connect it to your legs. So we're able to capture a pretty detailed pattern

of the data.

And then we have the motion capture. So what? If you just have the motion capture, then it is like, if you have the motion capture then you can't really do much with it until you know how to modify it.

Modifying Dance with Keyframes8:09

So the question right now is how do you modify the dance? How do you create new dance? Okay, so I'm going to play Dora the Explorer here a little bit. So we have four API we can use to modify the dance. So, fog, materials, keyframe and mesh. Which one do you think it is? Who thinks it is fog, raise your hand? Who think it is keyframe, raise your hand? Okay, who think it is mesh, raise your hand? Alright. Okay, so this is a bit of a trick question. But if you think about it, each position of your dance is actually a series of keyframes, right? So it is a series of changes in position, in rotation, that are kind of interconnected. So we have this API called 𝙺𝚎𝚢𝚏𝚛𝚊𝚖𝚎𝚃𝚛𝚊𝚌𝚔, which is actually you can use to basically

represent your animation. And more importantly, we can modify it. So think of it as a JavaScript function where you get in a set of keyframes, you put it some parameters, then somehow you get a keyframe out. Well, it sounds very easy, right?

Six Elements of Dance as Transformation Functions9:12

But wait, how do we write that function? Luckily there's the work that our choreographer have done for us. Remember, the six elements? The theory is with these six elements you can compose any kind of dance. So theoretically we should be able to use it as our transform. If we can write these sets of algorithms

in a way that you can represent it in computer, then yeah, you get a dance generating machine. But how does it work? So let me show you a very simple example. So this is the one we actually use, called overrideEnergy. So some elements you can interpret it very simply. For example, if you think about it, what happens when the energy goes down? Your body kind of slows, right? The movement kind of slows. So you can basically just divide the keyframe a little bit to basically make the movement in each part slower. So your hand might move a little bit slower, but your legs might be very fast. So you can do keyframe manipulation with that.

Demo: Manipulating Dance with Energy Parameters10:14

And let's see if it works. So here's a little quick demo. So if I do energy, upper body,

and modify it a little bit to be 12%,

then you can see the movement kind of stops. It becomes like very, very slow.

So that's very simple, right?

Representing External Body Space in Code10:31

But what if you go a little bit more complex? What if you want to represent something like external body space? So in Thai, this is ช่องว่างระหว่างร่างกาย. It's the space that surrounds your body. How do you represent that in code? I don't know. But of course we have to make an interpretation, because this is art. You have to basically try to interpret it into what computer can do. And an idea that the team came up with is freeze frame. What happens if you freeze your body? You get to see all the parts that surrounds you, the parts between your body. So the choreographer thinks when you're freezing, or when you're slowing down the movement, you see more external body space. So algorithmically what you have to do is you have to find a point where there's not much activity, where the movement is kind of very slow, and then you freeze it in time. You freeze the moment.

Coding Dance Transformations: From Curves to IK11:25

So how do you write that in JavaScript? It's actually not easy.

It's actually like a lot of code, but what you're doing is you try to find a valley in a graph with low change. And finally what you do is you add a little offset. So instead of the movement being fast, it's actually have some delay where it's not doing anything. So those two are quite simple, but there are something that might be a little bit more difficult. For example, how do you make a movement less curvy? What you might do is you might apply a filter, like high-pass filter to the graph. So instead of something being very curvy, if you apply a high-pass filter, it would get rid of some of the curve, and maybe you get a straight line. So by manipulating those kind of keyframes, you can do a lot of programming already, and you can already create new dance. You can also use some more complex algorithm like inverse kinematics, or IK, which is what they use in animation, to move something toward a point.

But this is obviously a lot more complex than the other. But if you think of it like function composition, you have the first dance, the original Thai dance as an input, and then as you add it to your transformation function, you're composing a function. And the end result of that function is new dance.

Interactive Dance Manipulation with Voice Commands12:43

So that's basically how the program works. You have a transform function that takes a old dance, apply some parameter, and you get a new dance. And of course we have to make this quite parametric because you can make the dance, for example, you can add a lot of curve, or you can make it perfectly straight. So you can add some parameter to these, or set some value constraints.

But a big question, how do you make it interactive? How do you make it so that people in the show can play with it? Because it's a dance, it's a performance. So what we decided to do is to go with a very simple stepper UI, where you can basically talk into a microphone like, "Energy, body, 12,"

and the program would understand that. Okay, I have another Dora thing for you. What API do you think it is? WebOTP, Speech, MIDI, or Webพนัน? Okay, who thinks it's WebOTP, raise your hand? Who thinks it's Web Speech? Who thinks it's Web MIDI? And who thinks it's Webพนัน? Okay. Oh yeah, that one would make the most money, but sorry, it's not the correct one. So you can actually use Web Speech for this. So at first I thought we need to do some WebSocket thing to get the audio data, but actually no. There's an API that kind of takes in your sound, and you can already do speech recognition. The bad part is, is actually not very good with something like a stepper command, because the choices is very restricted. So I end up having to write like a thousand lines of regex to actually correct the command.

So it's a little bit hacky, but it works. So just to summarize, we have the pre-recorded movement, and we apply some transformation, for example, applying energy, applying delay. And then with those set of operations, we're able to create new dance.

Demo: Live Dance Manipulation14:34

Okay, so just to show you another demo again. So what happens if, okay, so in the show we would have dancer selection. So you would be able to select the dancer to your stage, and then you would do commands like rotations, Y, and

For example, 94. And then it would kind of change that live. But I think the interesting thing for me is not the dance itself, but the people who play with the program. Because if you have the program, I think it's very boring. But in this case, we have microphone, for example, for me, and we pass the microphone to the audience. And I think the amazing thing is you get to see the people try to dance with the avatar, try to give new commands, like energy or circle and curve. And then you get to see people just play with the technology. And I think my goal as a programmer would be to create software where people would be able to think about it, or where people would try to basically have fun with it. So I think this is one of the projects where I'm very happy with because I got to see that. All right.

Singapore History and AI: A Performance Lecture Project15:41

So that's our first part done. For the second part, it's going to be a little bit shorter. So this is a project I did with my Singaporean artist friend, Ho Rui An. So he's very interested in Singapore's history.

So Rui An is actually a Singaporean artist, and he's interested in basically how the history of Singapore kind of maps into the history of AI.

And I think this is actually really interesting, and as an artist, he kind of do this in the form of a performance lecture.

Okay. So a performance lecture is kind of like a lecture that you might see in the university, except that it's a lot more energy, like a lot more performative. You are trying to do it as if you're doing it as a form of performance. So it is pretty nuanced, and you can use some a lot of interactive things with it. But when I met Ruian, I actually did a little different project.

Diffusion Project: Exploring Singapore's History through AI16:36

So I did this basically a diffusion project where I take in the photo from the National Archive. So for example, in Thailand, we have something like หอจดหมายเหตุ or we have something like หอสมุดแห่งชาติ, and you would have a lot of photo where you are able to take those photo and put it into a diffusion model. And you start to see some patterns in the data. For example, I started to see a lot of trees because there's a lot of photo with tree in Singapore's National Library because they want to go green or whatever. So you get to see a lot of these interesting patterns. Another thing that I also get to do is an automated exhibition because the talk there might

only be done a couple of times a year, but you want exhibition to go on all year.

Technology: Hugging Face Diffusers and WebSockets17:22

The technology here is actually quite simple. So we use Hugging Face diffusers, which is like a Python library that you can use to generate images. And then we basically open a WebSocket tunnel connection. So you can do a lot of cool things with it. The first thing we did is live transcription. So as Ruian, the artist, is speaking into the microphone, the Web Speech API takes in that, transcribes that, and generates new image. So you get to see this juxtaposition of what Singaporean artist is showing in his slide versus what the Western AI kind of thinks about it. So because of the training data, the image that's being generated is very weird and very out of context.

Visualizing Diffusion Steps in Real Time18:07

Another thing that we did is we want to visualize diffusion steps. So if you don't know what that is, it's something like this. So when you're actually generating image with a diffusion model, first it has a noise image, and then it tries to convert that noise little by little into a target image. So we want to be able to showcase this kind of effect. So people that are visiting the exhibition or the gallery, they started to understand how the diffusion works. So we use a pretty simple algorithm here that basically takes in what the AI step is doing in

that latent space and convert it to RGB. So you get to see something like this happening in real time.

Yeah, so you get to see this kind of nice step of how it works.

Automating the Exhibition with Captured Actions19:00

Yeah, and a final thing that I did on this project, which is quite interesting, is exhibition automation. So, because Rian only gives a talk maybe five times a year, we wanted to be able to show all year round. But the problem is there's like a lot of keystrokes that are happening. So, what we did is we basically tried to capture every action that he did in the performance, and we tried to capture that into JSON. And then once he's doing something, for example, he's moving the slider in the program, the program is basically trying to regenerate that. So, it's basically trying to replicate what he did in the installation. So far, I think this project is a very short one, but I think it's kind of interesting to see how you can use basically a very simple program to demonstrate your points or arts.

Visual Assembly Canvas: Writing Assembly for Audio-Visual Projects19:53

Okay, so here's my last project, and I think this is my personal side project which I'm very proud of, and it's called Visual Assembly Canvas. So, the idea is what happens if you can write assembly, but in this assembly, you can connect it to everything in the world. You can write assembly to generate new music. You can write assembly to basically write your own custom program, and you're able to use it with your media hardware.

And I think this project actually started

Inspiration: Old Computers and Shenzhen I/O20:25

with me interested in assembly. So, for me, I'm very interested in old computer, and I think these days, when we write JavaScript, we're very far from the origin of the computer. We have a lot of abstraction, and it's very different from people who are in the homebrew computer club of the past. Like a lot of them, they're writing very low-level tools. So, there's a game I used to play called Shenzhen I/O. So, in this game, you can write custom assembly code

to solve a puzzle, and this is very similar to real assembly. And it is a very weird game. Like you can print out a printout chart, and you can try to construct your own little circuits. So, my question is this,

Writing a Custom OS and Stack Machine21:11

how hard would it be to write your own OS? And honestly, at first, I think it was going to be very hard, but I see a lot of people, for example, the guys who did uxn and varvara. So, it's kind of like a group of people who try to create their own operating system and try to write things with assembly first. So, for example, these programs, like the clock, the calendar, and the calculator, they're all written with assembly, and the OS is written with assembly. And he's trying to basically try to do a personal compute environment where you're not dependent on compilers, where you're not really dependent on much software. So, you can see you can write all these with assembly, which is very hardcore. But I think this is going to help me to learn a lot as a developer. Like I think it's going to help me to learn how the computer works and just improve my skill. So, I first set a goal for myself,

which is, can you write your own stack machine?

But at first, I thought it was going to be a quick project, but it took me 320 hours, which is a little bit longer than I expected and a lot of commits. But what I ended up doing at first is this, you're able to write some very simple stack machine assembly. For example, push 10, push 25, you multiply together, and the program is able to execute that assembly.

Okay, cool.

Using React Flow and Rust for Assembly Programming22:38

And the software I used with this is React Flow. So, this is a React library that you can use to create a diagram, and you can create new nodes and connect them together. Yeah, and for this one, I used React and Rust. So, the reason I went with Rust is I want my assembly to be able to run on actual hardware. So, someday I want to have something like an Arduino and use my assembly in there.

Yeah, and with React, I think it's quite straightforward. I think one interesting thing is Lezer, which is a library that helps you to write your own language. So, here you can see we have some syntax highlighting here. This is thanks to the power of CodeMirror and Lezer. Okay, so how do I create this program? Let's go with the very first. So, we want to build a computer, right? How do you do that very quickly? Of course, you're going to need a memory. So this is like a raw memory, if I understand correctly. So what I did is I basically allocated a part of memory. I divided into chunk. So one chunk would be for the code, one chunk would be for the data, one chunk would be for the stack. And then you basically write some stack manipulation program. You create some register so that you know where in the program is pointing to. And then you do a fetch, decode, execute cycle. So it is able to read what instruction you are at.

For example, you want to add two numbers together, you want to multiply them, and then it's going to run that instruction. So I written this in Rust and basically I try to write like for each instruction, what is it going to do? Is it going to add? Is it going to push? And I think this is really all you need to make a very functional, like simple stack machine.

Yeah. So this is how it looks like at the very start of the project. So I made it so that it kind of highlights like where in the execution is going on. So now it works like as an accumulator. So kind of like a for loop.

Inspiration: EDM Concerts and Algorithmic Music24:39

Okay, so we now have a computer, but is this cool? For me, not yet. I think like there must be something more that we can do. So I went to an EDM concert. I know, very weird tangent. So in this EDM concert, I see people that use data

to basically create new music, to create new sounds, to create new visual. And I think it's like, wow, this is like a new world for me. You can see like playing with numbers can result in so many cool sounds. And even Creatorsgarten themselves, we organize an event called Algorave, which is an event that we invite people to use algorithms, to use JavaScript, to use P5,

to basically create their own sounds or create their own visual.

Writing Assembly for Music and Visuals25:23

So now I feel inspired. What happens if you can write assembly to do all that? That's weird, but maybe it works. Because in the world of people who are doing live coding, they already use tools that look very similar to this. So for example, a lot of artists use something called Pure Data, which is you basically create individual sine waves, you create individual square waves, noise, and connect them together. A program like Max MSP. And I think if they can do it, then I think I can do it too. So I started basically diagramming a little bit how I want the program to look like.

Actor Model and Message Passing in Visual Programming26:00

And another thing I noticed is that a lot of these programs, they use actor model because each block has to send a message to the other block. So I program and I'm also inspired by this example in Minecraft where each block can send message to another block and you can create a fully functional program. So I started to like this message passing system

where you can connect one machine to the output of another machine. So output of this to the input of this. Then you're able to basically compose a more complex program. And with actor model, it's very simple. You are able to send some message, you are able to receive a message. And then I built some kind of little inspector. So because I think when you're writing assembly, it's very difficult to know like, what the hell is this memory? What is that memory? Like, I don't know any of it. But if you're able to drag it into a workspace, then you know like what that piece of memory is for and how it's changed.

And I started to add like some simple function, for example, an oscillator.

So now you're able to take in a clock input and you're able to basically play with it.

Adding a Command Palette and Web MIDI Integration27:13

One of the last thing I did is add a command palette. So now you can basically add new blocks together. You can add a plotter. You can plot that data. You can connect it together. So this is like a sine wave. You can select what waveform you want and then you plot it. So in here, like in a very small amount of code, you can already create like all sorts of complex things. And I think this really helps me to be able to prototype. Like if I want to create a program, I don't have to really write assembly, but I can kind of connect things together. Okay, and I think this is the last demo. So I start to see like the Web MIDI API,

which I know like you don't need to use MIDI only for music. No. You can use MIDI for everything. So I started using Web MIDI as like a control or as an input where you can enter something into the launchpad and then it can be interpreted into the program. And finally, this is like kind of the last thing I did with this project,

Bad Apple Challenge: Running Animations with Assembly28:15

which is like, do you know Bad Apple? Anyone here know Bad Apple? Okay. So I, not a lot of you know. So it is kind of like a programming challenge. The challenge is very easy. There's a couple of video frames from the music video of Bad Apple. Can you make it run anywhere? Can you make it run on your toaster? Can you make it run on your fridge? And a lot of people try to do this challenge. So, yeah, at the end of this project, I was able to get this working.

Okay, but I think this is now not working. But we're at the last slide, so it's going to be fine.

I think we're lucky because this is the last slide.

Does it work? Okay, it works.

Yeah, so that's like the last thing I did on this project.

Reflections: Following the Fun and Doing Hard Things29:23

And it's very fascinating how you can get from "I want to create a computer" to write your own assembly to create a set of things that allows you to play. So, this is the end of these sections.

So I think the two things that I learned from working on these really fascinating things. First thing for me is follow the fun, or more importantly follow what you're nerdy about. Like I'm interested in dance someday,

next day I'm interested in history, the next day I'm interested in how computers work or how music work. And you're kind of able to tie that together. And I think the second most important thing is try to do hard things. Like when we are at work, I think a lot of us we just work on crud projects. We work on create, read, update, delete. We work on something that are maybe not very fun or not very technically challenging. So I think in your side project is kind of an opportunity where you can play with any technology you want. You can go as crazy as you want with your libraries. So I think with this private side project,

if you just have fun with it, if you just try to do things that are challenging for you, then I think it results in so much fun and you can learn a lot from it.

And I think this is it for today. Thank you for listening. ขอบคุณครับ

Okay ฮะ Okay, so before we end, two more things. First thing is the feedback form, if you could scan it. This is for the whole conference, not for this talk. So if you have some feedback for the whole conference,

like how do you feel about the conference, what isn't quite good for you, what you like about the conference, you can share it in the feedback form.

I'm going to pause for one more minute.

Assuming everyone have already did that. And because this is the last session of today,

I think let's take a photo quickly, shall we?

For the feedback form, maybe we can take a photo first quickly. Or maybe whenever you feel like it.

Yeah, we take photo together.

So this is the last session, so let's do some photos.

It was really nice. Thank you.

อ่ายืนเรียงกันเลยค่ะ รบกวนไม่บังสไลด์นะคะ จะได้เห็น - สไลด์เต็มๆ นะคะ - โอเคครับเดี๋ยวยืนเรียงกันนะครับ Could you stand in one row, I think? Maybe one or two rows, so that it doesn't block the slide