Videos → Regular Expression as an Irregular Musical Expression

Talk starts0:00

Hello, I'm Jim. My profile is not very interesting, so I'll skip it. The topic of the session is "Regular Expression as an Irregular Musical Expression". If I translate it, it will be confusing. so, I won't translate it. But you will understand what it is, eventually. My background is a graphic designer. I'm also an amateur electronic musician.

I'm interested in writing codes. I just wrote a code a few years ago. so I'm quiet a relatively newbie in this field. Let's start with the word "music".

music0:42

since, It weaved the world of "graphic design" and "codes" together, personally.

What is music? According to Edgard Varèse, he is a french composer also known as the "Father of Electronic Music".

"Music is a sound that is organized". however, some might not fully agreed with that definition, it's debatable. but I'll use his term as a starting point so, what is a fundamental of "organized sound" then? fundamentally, "Sound wave" is the basis of any sound.

Sound is the vibration of the object through the medium. for example, When I hit the table the object will vibrate and the molecule in the air will be pushed into the ear of the audience. so the air act as a medium and you can hear the sound

properties of sound wave1:57

let's look at the properties of sound wave. What is it? the first thing is wavelength. the moment before the wave is repeated itself. The next one is amplitude. Ampitude is the rate of displacement.

normally, we perceive it as a "loudness" the other one is the frequency. Frequency is the number of repetitions of the wavelength in one second. for example, note A is 440 Hz.

It will vibrate 44,000 times per second. human will perceive it as a note A the next is time-period, similar to wavelength the difference is wavelength more about a "space" but time-period is about "time" (before it's repeating itself) "velocity" is the speed at which the sound travels. so, the sound that happen in the water. the velocity will be different from the sound travel through the air. The next video will be an inspiration for the software I wrote.

I will pull out one of the properties.

In the video, despite there is no sound at all. but it can be reminisced to one of the element I've just talked about I want everyone try to think, what is that property?.

This is a real event. I means, inspired by real event When I search for information on the internet, I clicked and clicked. What is related to the properties of sound wave is "time period" the time before it repeats itself. If you go back to see it, there will be a starting point. Stop and rewind.

time period4:10

Let's look at the word "time period". When it repeats itself, it is called periodic event. It is an event that repeats itself. The event is one of the methods that musicians use.

I will give an example. Does anyone have no idea what a fizzbuzz is? Everyone probably knows. Fizzbuzz is a game, sort of. It is a game that teaches children how to do division. If it is divided by three and remains is zero, it is called "fizz" if it is divided by five and remains is zero, it is called "buzz" if it is divided by both three and five, it is called "fizzbuzz"

you'll notice, I didn't do anything in the first event. also the second event I didn't make any sound. The third event I spoke "fizz" and so on.. What happens is an "interval (of time)". Interval is the time period between one stage and another stage. One of the main element of music is rhythm.

in this sense, rhythm is nothing more than just a management of interval (of time). This is a hardware model.

This is a software model (step-sequencer). Everyone might be familiar with if you have played GarageBand, you will notice the timeline. ultimately says, timeline is just an event management (interval of time).

“anu”6:00

This is the software that I wrote.

"anu" is a Thai prefix that is used with other words. "anu" means "small", "tiny", "sub" eg. "anupak"(molecule), "anukun"(support), "anukrom"(series). metaphor to the software that can prefix with others.

objectives I want it to be just a "frontend" for any "backend" that can understand the supported protocols currently, it supports OSC, Open Sound Control and MIDI. The second one is the deterministic / stochastic processes. basically, a semi-random semi-deterministic I will explain it later. The third one is "(live) performance-oriented" initially, I wrote this for live performance but eventually it never been performed anywhere so

next one is "low dependencies" means it should depends other library as less as possible since I don't want to keep updating dependencies or some "breaking change" also it is easier to see how it is broken. so it's easier for maintenace the last one, since I am from a "graphic design" background. It must have a connection to its vision.

development7:12

Let's see how to develop the application that I wrote. Let's look at the overall image first.

This is the software that I wrote using JavaScript only.

I did not use any framework, React, Svelte or anything. When you click search, it will match the index.

It will add a class "current". It will be actived this does not work at all, because it manipulates the DOM directly. By manipulating the DOM(document object model) directly it will cause a lot of performace problems. (for step-sequencer application) I will explain further. This is the amended version by sending a msg to trigger a sound.

I use SuperCollider as a backend sending out an OSC(open sound control) since, I cannot compose other part while clicking

so a "step" was introduced it will be the result of the next video. oh ok, It will checked-out one of the objective of the project. It is a "visualize syntactic connectedness" the audience will get a clue what's going on

this is the amended version

the problem is that it cannot compose a complex pattern when I use normal searching so, I added the function of the regular expression. for anyone who don't know about Regular Expression basically, it is just an another method of searching so you can define a complex pattern for searching eg. I can validate if the text message is an email or not. I can search if the text message is an email or not. phone number or not, etc. for example

the letter "?", will match preceding character('u') 0 time or 1 time

the letter "+" will match preceding character('r') 1 time or more times

This is an implementation of a RegEx(Regular Expression). so we can creating more complex patterns

ok so, What is deterministic and stochastic? Stochastic is used in Iannis Xenakis's meaning Xenakis is a Greece composer he was nearly in the same period with John Cage but Xenakis is known to be an "Avant-garde" composer while John Cage is experimental composer there are similarities, but different in details. Avant-garde, adopts an extreme position within a certain tradition eg. controlling some element by creating system that let it determine itself that allow the system to create the flow.

For example, Xenakis's work "Achorripsis" (1956)

he used the algorithm well, not really, its a formula called "Poisson distribution". It's the distribution of events that occur randomly

to make a score(notation) so I use the word "stochastic" in this meaning. what I means is, Even though you know what's going to happen with these RegEx pattern, but in the context of music, you can't say that music matches what you want.

There will be a half-random and half-deterministic

let's try implementing it with the hardware.

This is a modular synthesizer. The message is CV control voltage. It's the hardware language.

this will checked-out the objectives's "back-end agnostic" with hardware this is integrating with another software called Pilot.

Now we have achieved 3 objectives. only 2 left

Tech stack12:49

let's talk about "Tech Stack" who uses Electron?

"anu" was written in 2019. using Electron but I changed it to Tauri (in 2023) anyone using Rust? cool, lol

The result is the size has been reduced drastically from ~120 MB to ~ 9 MB. but I have to set up the optimization profile. But to be fair, even if I didn't set it up, it will be reduced by 10 times anyway.

Application architecture13:26

Let's look at the architecture. The main part of Tauri is the Rust code. for backend The frontend is JavaScript. I use the backend to store modules that do not have on the frontend. I use the backend to store modules that do not have on the frontend. Since Tauri does not use Chromium at all.

that's why the (application)size was very small. instead It uses WebView. but WebView doesn't have requestMidiAccess method so I have to write MIDI module in backend(Rust) instead then call the function via `invoke`

this is the architecture of the frontend. If you have ever written Processing or OpenFramework, It will use the similar structure there will be an `install`(function) that runs once. There will be an `init` also runs once. and `run`, and `update` that will run as a loop

There will be no dependency except for the CLI of Tauri.

I will checked-out "low dependency / less obscurantism"

Regex implementation14:38

next, let's see how to implement the regular expression. In the first version, I used a very straightforward method. In the first version, I used a very straightforward method. The problem is this. This is it.

If you have implemented the regular expression, You will understand that it will cause a case like this case. The case called catastrophic backtracking. The case called catastrophic backtracking. basically, RegEx cannot prevent us to use a pattern that will cause a long recursive finding weather it can find or cannot, since it will takes a long long time it doesn't work with my application at all otherwise, it may be stuck (freezed). If we send a empty capture group if we send a capture group. oh, spoiled, sorry The question is, How will we prevent this case? How will we prevent this case? my question was

how will we know whether the program (RegEx solver) will runs forever or stop at some point? does it sound familiar? yes, it is a Halting problem classic

What I did was, the first thing I did was sending a message to the worker thread. the thread that handle RegEx send "onload" when worker recieved "onload" msg the first thing it's going to do is it will throw out the "timeout" back to main thread. If the task takes too long, I set it to after 250 ms. If it's not too long, just clear the timeout. And send the message back.

Whether it's an error message or whatever.

Back to the main thread

how do we know that the RegEx pattern we wrote will run all the time or leave the loop? so I use a technique called 2-pointer. Slow pointer and fast pointer. If anyone used to play Leetcode you'll be familiar with it. eg. Circular Dependencies sort of thing it's used to check if two position is "meet" or not there should be a gap between the slow one and the fast one, like a rabbit and a turtle. When the turtle follows the rabbit, there will be a gap between what follows. if not, or a slow index = fast index its means, the loop occurance.

External connection17:10

Now, the external connection currently support just two, MIDI and OSC(open sound control) as I said earlier,

it can't request MIDI access on WebView. So, I had to write MIDI module in Rust simply, we just had to declare the procedural macro here and then we register it in the main app. now we can call via `invoke` we can invoke Rust code from JavaScript side.

We can invoke it. Open sound control or OSC is a protocol. This is quite long. I can't explain it. basically, I use nom as a parser combinator. It's similar to PEG but with PEG you have to write a gramma it will pass the message (byte-array) directly. The speed is good. tbh, I don't know the difference that much it's just looks interesting

so, I used nom.

Clock scheduling18:23

Another important part of the application is the sequencer. The timing of the sequencer shouldn't be jittery. it have to be stable There must be an robust implementation of the clock. What will I use between these three? Date.now(), setInterval, setTimeOut. The first thing I used was setTimeOut. very straightforward and naive eventaully it didn't work. I asked myself again. What should I use?

Date.now, setInterval, setTimeOut. setTimeout already tried so there were two left. But the all options was wrong. The right question should be "where to run the clock?" because the main thread has the duties of rendering to disturb the timing of the clock. I used to use the event loop. Event loop is another topic that can be found on the Internet. basically, Event-loop it doesn't guarantee that the task you put in the queue will work as planned. It's not very suitable for the timing that requires high precision.

The next method I used was setInterval on worker thread. This method almost works. But the problem is when you wanted to change the tempo in the middle of the song, it will cause a problem.

The last method I use is audio context current time. Current time is double precision 64 bits. It means that the accuracy of the floating point will be very high therefore, it is very suitable for high precision handling such as timing and tempo. This is the updated version. There is a "start" msg, and "interval", and "stop" from the first one, there is only a start. what's been developed from previous version was this function the actual function quite long but it can be boiled down to understand as this. While it sets the...

It has to keep checking the queue. It has to check the nextNoteTime. It's time to "is it ready for the next note(tick)"?. There is a scheduleAheadTime. We will check the time until the next note. the results is the timing will be very precise this is a bit tricky (to implement). since, setTimeOut although you already have a precise time, guaranteed time to play precisely but it's not I tried using setTimeOut but it's not works so I use the audioContext's time instead. The audio contact has a function called "start()" which accept "timestamp" as an argument tell it when to "start" and when to "stop" since we already have a very precise time and audio contact that can run the right time. "start" then "stop" immediately. Then use "onended" as a callback to run the application it's a little tricky and the flow is strange but I can't think of a better way. If you know how to improve the code, please let me know.

Latest iteration22:23

This is the latest version. you might notice that I completely removed the form that has input. personally, I like Vim's workflow.

I don't wanted to use the mouse while performing. quite distractive and annoying. This is the console.

so this checked-out "(live)performance-oriented". I use it to perform.

the objectives is now completed 5 options.

It's done. "anu" is an open-source so anyone can PR or open an issue or anything

Demo23:24

now I'll show you the demo

That's about it.