Videos → Just go for it: The story of dance-mat.js

Please give a big round of applause to Mr. Ramón Huidobro.

Thanks a lot for that wonderful introduction and hello everyone, good afternoon. How about this conference, huh?

What a day it's been, so much wonderful knowledge shared, so much wonderful people, and here I am to ruin it. I'm just kidding. Listen, if you were having some trouble seeing my slides or if you want to get a peek at what's coming up, I've left a link there, ramonh.dev/dance-mat-js.pdf.

You can follow my slides there just in case. So, hello. Quick word of warning before we begin. There is going to be some music in this talk, so you'll be hearing some nice stuff. My name is Ramón. I'm so happy to meet all of you. Some of you I've already met and talking to you throughout the day. I live in Austria, I said. I'm originally from Chile, but I've been living in Vienna for 20 years, so I don't really say I'm Chilean anymore. Yeah, I do some freelancing and working with children, but you're not interested in that. You probably saw my web, the picture of me on the website and thought, "Hey, I saw a dog there. Can we get more pictures of the dog?" Yes, don't worry, I've got you covered. This is Fiona. I love her so much. She came into our lives like three, four years ago and she's got these eyes. She looks at you and says so many things. I particularly love this photo at the top right where she's judging my wife as she eats some tasty mango.

Yeah, I love her so much. Well, enough about me, let's talk about dancing, in particular dancing video games. Now, I want to get some hands up from everybody. Does anybody know Just Dance?

I see. Oh, I see a couple enthusiastic hands over the back. Yeah, that's I never played it myself actually, but it seems pretty fun. Let's try something a little more recent. Does anybody know Beat Saber?

Yes, some enthusiastic people, that's awesome. I tried it recently. It's super fun. I'm a little more old school myself. I grew up with a game called Dance Dance Revolution, or as I'm going to call it from now on, DDR. So let's see a show of hands. Who knows DDR?

Yeah, nice. If you don't know it, it's a dancing video game. You can see here in this handy gif. You've got some arrows on the floor. Each of these are a button that you press to the rhythm to make it look like you're dancing semi-competently. It's a lot of fun. It's pretty old. If you see the screenshot, it's from the PlayStation 1 days, but it's a lot of fun. And you know, given that it's from the PlayStation 1 days, you might realize the music is well, it's something else.

Let me tell you. I'm going to play a demo for you of what the music sounds like. Hopefully this will work.

Oh my god, it's just so good. So, my wife Big Ed and I, we wanted to play some DDR recently. And while getting a PlayStation and some mats or going to an arcade, we don't have a lot of arcades in Austria. So, the best option for us was to download an open-source piece of software called StepMania. Does anybody know it? Yeah, cool. A couple of people know it, that's awesome. And it's a lot of fun and what's great is, as I said, it's open source, which means you can put any song you want on it, program it in order to have your dance moves ready for it, which is pretty fun. And for the hardware itself, you can have these pretty cheap foam mats. Now, the way they work is that instead of being a button physically per se, it's more of a contact that when you push it, it presses the button. And it's great and they're pretty affordable. Problem is that after some time, especially if you're using it as intensely as we are, they wear out, right? So we started getting used to seeing this screen a lot. So what we thought instead is I remember when I was a child, they had these metallic mats that you could play with at home. And I looked these up and they're pretty expensive.

Now, you might think, huh, $300, it's about 10,000 baht if I'm not mistaken.

$300 is a lot of money, but notice that's a discount from $900.

That's a lot of money for just a mat. So we thought, okay, well, we forgot about it. Then after some time, we were at this event in Vienna called the Maker Faire. Does that exist here? Yeah? Excellent. Well, what it is, is a showcase for lots of people making their own tinkering stuff. And I'm not very familiar with hardware myself at the time. So I thought, oh, I'll go check it out. And then I discovered a piece of hardware, if we can call it that, called Bare Conductive paint. Anybody heard of it? I see a hand.

What it is, is you might have seen on like Facebook videos or something. What it is, is a paint that allows you to draw your own electronic circuits. So you could draw like a line from a battery to a light bulb, back to the battery, and it'll light the light bulb, which is cool and all. But as you can see in this photo, what I did not know is it's also capacitive, which means it can carry a charge if you touch it. So when I saw this, we started thinking like,

hmm, I could make something with this. So I checked it out. It turns out they have some nice open source software that you can use in order to interact with it. And it even works with a Raspberry Pi. And my favorite part, a piece of code that I can copy and paste directly to try. So I looked at that and I thought, hmm.

Could definitely make something with this. So I took a look, another look at StepMania, and it had customizable controls. That's pretty cool. So we thought, okay, my wife and I thought, alright, let's take a yoga mat,

some of the paint, a Raspberry Pi Zero, one of the hats for interacting with the paint, a little Node.js, we've got ourselves project dance mat.

And what an adventure it was, and this is what I'm here to share with you today. Quick disclaimer, this is my first ever hardware project. So, I'm going in a little bit unprepared for it.

So here's the plan. You plug the dance mat into the computer, the computer thinks, "Aha! This is a keyboard. I shall treat it as such, and then I can customize my controls on StepMania." Fantastic. So, let's get started. We've got our piece of code here. What it does is you set it up so that it's linked up to the hat that's connected to the Raspberry Pi,

and when it receives data, we have that big comment that I left myself there, "Send data to PC."

And that's where I found my first piece of trouble, because I didn't know this going in, but if you just plug a Raspberry Pi into a computer, the computer's not going to immediately go, "Oh, this is a keyboard, of course." Now, some of you might be thinking, "Ramón, did you seriously not research this before going in?" And to that I say, may I remind you, this is my first time trying something like this, okay?

But I did some research. So it turns out, because the Raspberry Pi, which if you don't know, is a, at least this model here, is a $10 piece of hardware that runs Linux and allows itself to be completely modded.

Haha! It can be configured to act like a USB gadget. I found some documentation here, I started reading it, I gave up immediately, and did some Googling, and I found this, a nice handy dandy guide to teach me how to turn a Raspberry Pi and tell a computer, "Haha! I am a keyboard." Now this takes place in two steps. Step number one is to enable what is called Raspberry Pi On-The-Go mode, which I really like the name. Step number two is to add a script so that when we

plug the Raspberry Pi into the computer, the computer says, "Huh, wait a minute, what's this?" And the Raspberry Pi will have a little script that says, "No, no, don't worry, don't worry, it's cool. I'm a keyboard."

Now, the way the USB on-the-go mode works is that,

there's two USB ports, mini USB ports on the Raspberry Pi. The first is for power, the other one's for USB. Data travels through the second port, but if you enable this mode, it can also act as a power, which is pretty cool. So all I had to do was run these three commands, don't worry about what they say, I'll tell you what they do. What they do is essentially be copy-pasted and let you do all the things you want. No, I'm just kidding. What they do is they allow you to configure your Raspberry Pi to say, "Hey, I'm a device."

Right? So that's step one. Step two was having a script that runs on boot

that tells the Raspberry Pi, "Hey, when I get plugged into a computer, I'm going to tell it I'm a keyboard." And it's this nice big chunk of code here that I have no idea what it does, except for three very important parts, which is this here. It, where I can tell it, this is my serial number, this is my manufacturer. You know, when you plug it in and Windows says, "Configuring blah blah blah by such and such manufacturer." That's all that mattered to me.

So, I learned some things about, from this, by the way. I didn't know that there's a virtual file system that gets mounted on the Raspberry Pi automatically and lets you configure hardware stuff. And there's also so many configuration options that lets you mess around with what is essentially a $10 piece of hardware. I am learning already so much from this.

So, once I had that running, the most exciting part came when I plugged it into my Windows machine and it made the most incredible sound. It went something like,

It's an exciting sound, believe me. So, okay. Next up was to look into the USB human interface devices guide and learn how to send binary data from the Raspberry Pi to the computer to tell it, "This is a keystroke." And so I went to this website, I downloaded the 300-page PDF, and about five minutes later, I made that sound that like big, like a basset hound makes. You know what a basset hound is? It goes kind of like this.

That's a lot of reading. Turns out I was very lucky and I was able to Google some stuff and talk to some friends who have more experience with hardware than I do. And it was a reminder to me that if I get stuck, it's fine to ask for help. People are happy to help, and if it's something they know about, they'll be enthusiastic to help you.

So from that, I learned that each keystroke of a keyboard through the USB human interfaces guideline is an array of eight hexadecimal bytes. The first two are which are reserved for modifiers, like shift and all that. So, I found a list of hexadecimal codes that correspond to each key, which is great because then I was able to move back to my JavaScript and say, "All right, this is left, this is right, this is up, and this is down," which is represented by four hexadecimal codes which represent the letters A, B, C, and D. I was already started and I was so excited. So I was able to plug that into a piece of code that says, okay, when I press some data, you see, the hat itself has 12 little nodes where I can attach an electrode. And each of these electrodes can correspond to a member of a list. So I can parse each electrode and tell it, depending on which one is pressed, pass onto that the corresponding key. So, if the first one is pressed, press add the left key. If the second one is pressed, put in the right key. And then up, and then down, and so on and so forth. So when I did that, I ended up with an array of touched codes.

Now, you might notice this allows me to press more than one at the same time because with DDR, there might be sometimes where we might be doing a little bit of this, a little bit of that. So it's good to have. Once I've done that, I need to actually build a keystroke because as you saw earlier, it's made up of eight hexadecimal codes. So, the first two are made up of modifiers.

So we'll skip those, and then I'll just put in the rest of the keys and put in a bunch of zeros just so it makes up a nice list of eight. Once I do that, I can actually start the script, plug in my Raspberry Pi, and start just looking at what's coming out. And you'll see I get a nice little A, or a B, or a C, or a C and a D, and all that. So, next up is converting that into a byte array. Now, again, documentation, super helpful, MDN allowed me to find this function here called 𝚄𝚒𝚗𝚝𝟾𝙰𝚛𝚛𝚊𝚢. And from that, I was able to create a keystroke from that

that is a byte array. Next, I got to send that over to the computer.

And here I got stuck again.

Now, as I said, I'm not very experienced with Linux and hardware and all that, but after some research, I found something called file descriptors. So for those of you who don't know, everything in Linux is a file in the file system. Everything from hardware devices to network devices to documents, they're all files. And I thought, all right, so I can pipe this byte array into the

file descriptor of my USB device. And I did some searching around, and of course, because there is something for everything there, there's an npm package for this.

And I was able to plug this npm package, say, all right, open up that USB file interface and with that, I'm able to simply write to it. Isn't that amazing? But of course, I'm a good citizen. Once the program ends, I have to close the device. Got to clean up after yourself. So let's try it out. Start up the program. Plug it in onto my computer. Open a text editor in my computer. Look at that. I've got A's, B's, and C's, and D's.

It doesn't look like much, but I'm just tapping bits on an electrode. I've basically made a keyboard here. And here comes the fun part. Next up is the painting. My wife, Big Ed, and I got to work. We've laid it out. As you saw, we bought a yoga mat. We even made two players because you got to have two players in DDR.

We then realized if you have two players that close together playing DDR, it could lead to some trouble. So we cut it in half and there's Big Ed painting everything. And we tried it. And did it work? Yes. Were we excellent? Kind of.

But it worked.

Now we could play StepMania or DDR with our very own dance mat that we made from scratch. Then the question becomes, are we set for life now? And honestly, it depends on how you look at it. You see, over time, if you're stepping on a yoga mat made of paint, something happens. Actually, I'll just show you. Computer, enhance. Thank you.

If you put paint on a yoga mat, it's going to crack over time. So through this and some other problems, we started compiling a list of things that I like to call, "If we could do it all over again." So when I was talking to some friends and some bigger hardware people than I, the first thing they told me is, "Ramón, you should really just use an Arduino to do this." And I said, "Hey, that's a great idea. Quick question, what is an Arduino?"

It's a microcomputer, by the way.

We should use a smoother surface to do this. Turns out a yoga mat, sure, it's nice to be able to roll up and put away, but it cracks the paint. The other problem we had is that if you're using paint as a cable, it creates a lot of surface error potentials. So we started soldering on some cables onto our

Raspberry Pi. The important thing though, is that through all this, we had a great time and we learned a whole bunch of stuff. And to prove that, here's a picture of us looking really happy.

Now, you've been hearing me ramble on about this for a while, you're probably wondering, does it work? Is there a demo? The answer is yes-ish. You see, I can't bring a whole yoga mat all the way from Austria. So, I built one out of a smaller piece of cardboard. Here's the Mk Ⅰ. As you saw, it's using a lot of cables as for the paint, and it did work. And I was going to present this at HalfStack Vienna, and about five minutes before, the Raspberry Pi came loose and the cable wasn't working anymore, so I tried to repaint it, but then it connected two of them together. So I tried to fix it with a little bit of water, and that kind of smudged it. So I tried using a little bit more water and that smudged it even more, and ended up being a mess. So what I have for you here today is Mk Ⅱ.

This one uses crocodile cables, which I never considered using. Now what you see here, and this is what I have here on stage, is four arrows and a cat.

The cat button is start.

Now, what is this, you're probably thinking, oh, this is super high-tech, I wonder what's underneath. I'll tell you, computer, flip! It's a pizza box!

So let's try it out! I hope it's working. Good thing I have StepMania here.

Alright, let's start it up. Start the music. Now, normally I would ask a volunteer to come up and play for me, but we're kind of short on time, so I'll just do it myself.

Alright, here we go. Wish me luck.

You get the point.

You're too kind. That was a disgrace, and if I had continued, I would have gotten an F. Well, so that was me for today. For those of you who aren't experienced with hardware hacking or think it looks daunting, it is, but there's so much support out there. People are so helpful. There's so much to do. And I wouldn't stress about it. Yes, it's a bit more brittle than a piece of software, but those same principles that you love about developing software, be it web, native, mobile, and so on,

they can be applied here. There's lots of JavaScript things out there. There's lots of Node things out there. There's lots of things in your favorite programming language, Ruby.

Those are Ruby socks, by the way. There's just so much out there. And it's so important to just have fun and most importantly, just go for it. You know, you never know what will happen. Thanks a lot, folks.

Thank you, Ramón. One more thing, I really want to see you dance over here on the stage. You do a real dance.

Anyone would like to see Ramón to dance, real dance, on the stage?

Come on, you got a big fan here in fact, in Thailand.

I don't know. Can I have some music? Yeah! Wow! I'll be your dancer, too. Let's do it along together.

Everyone, big warm applause for Ramón!

Thank you, we love you from Thailand. Thank you, Ramón.

So, I really love him for one of his main concepts,

one of his main idea he would like to give you as a takeaway, which is play around with things. I think it's really amazing and it's good that you can apply this kind of idea for in every aspect of your

life, too.