🎞️ Videos → Write Less, Achieve More: How to build a living document with Cucumber JS and PACT

Introduction and Initial Hiccups0:00

Hello, everyone.

Yeah, this is my very first time here as a speaker in the Oops, in the like event like this as well. Yeah, it sounds like something is wrong with my presentation.

The Importance of End-to-End Testing0:22

So we're gonna have on this session, right? So, of course, we I begin the topic without the test name, the same way we write the romantic names. So I mean, we write the romantic song, right? So the

Writing Stable and Fast End-to-End Tests0:40

topic is about how we write the end-to-end test, but not about writing it. How we make sure that whatever end-to-end test that we are going to run is gonna be super stable and fast, right? So, anyone here like there's someone with the like that works with the role of the like

Software Automation Engineer, right? The Test Automation Engineer? Anybody here?

Why Software Testing Matters1:22

Cool. It sounds like, hm,

You never test it?

Oh, yeah. That is a good thing, but how do you know that your software works?

But you didn't test it.

Keep up to update. It's not bad. Oh, okay.

Okay. That sounds like good idea to

build the software as well, right? So, Obviously, yeah. Like, the same motto can also do with the

elevator. So today, if we don't test the elevator, what happened? Would like to jump into the elevator that does not get properly tested and risk your life as a testing material?

Sounds interesting. So, yeah, opt me out.

The Cost of Testing and the Limits of Unit Testing2:23

I probably don't want. So testing the software is a matter of take care and make the software reliable for the user like, despite they are customer or non-customer, right? So we being responsible for what we deliver to them, right? So wrong software rule or automation. Or I mean wrong software, rule or functional can cost the business loss. and it can also

go back as a punishment on our backlog, on your backlog, on my backlog, as back to fix. So we begin like

our unending journey of

fixing the bugs.

Introduction to Effective Test Writing with Cucumber3:12

But I do understand one thing that software testing is costly. Really costly. It's it's not easy start to be conduct with, right? So we have the easy way that we try to do and everyone probably familiar with it, which is the unit testing, right? Right. The real problem with the unit testing is it actually tests one thing in too much isolation, but the software itself does not work in isolate. Right. We can test how our

JSON stringify works. But we never test whatever real credit card transaction like

that got performed on our system. That can be the case, as well, right? So, we don't want to deliver the software that we'll end up as creating the damage to the user or to the to ourselves in the long run. So, But how can we write the test effectively, right?

Session Goal: Fast and Reliable End-to-End Tests4:23

So, yeah, so I begin the session, right? This session is about conducting the test here. So, a lot of text here, you can capture and read it later. I already mentioned, right? So today we, like, what I am going to show you

is the ability to run

whatever end-to-end test,

with one minute per test case, within one minute. And end-to-end means, if you're working with the website application, you start your website, you start your BFF, and you configure everything and run your UI test, right? The full UI test, right? And also, in the matter of

really, really high reliability. So it means that when your test fails, most of the time, it's because of your functional, not the system or environment overall, right?

Right. When we, and then

it can also make it parallel. So I'm not sure I can share, but today, at the team I work with, we run these kind of tests, like I think, more than 100. I don't remember the number, and the pipeline that we write it and run these tests, it's done within 15 minutes with 90% plus stability. This is the number that I know, right? Right.

Documenting Business Rules and System Functionality5:57

So, and the second thing, second thing is Second bullet is something that I think everyone is familiar with, right? When you work with the system, right? you develop a set of knowledge on how your business works, right? Or how your system performs with the business, on how the system dynamic,

meaning how systems

group to each other and they will deliver the result, right? So The problem about this thing is if you don't have the proper way to record the functionality as well as validating your functional

you you become a bottleneck. What does it mean by bottleneck? Think about the situation that everyone has to run to you asking you how these things work, how this feature works, how the login works, how the inventory works, right? How this particular set of the function works. And they need to rely on you, and you cannot do anything else because this question will go to you every time, right? And at some worst case, when you leave or the person who built, who worked on this particular function left the company or left the team, this code becomes something we call legacy because we don't know what it is. We just know that if we don't touch it, everything works. And if we touch it, something funny happens. Like, don't touch it on Friday evening. Everything is okay. So

Escaping the Bottleneck: Documentation as Testing7:45

So, but how can we get out of this situation? How can we document the business rule, the software functionality rule and use it as our testing? Why?

It have a less coupling to the system. Right. Coupling means, think about unit tests, right? When you refactor all your unit tests probably can compile, and you need to change all your unit tests to conform with whatever you refactor, and you start to wonder whether you fix the test correctly or not. This is where the unit test

introduce a coupling to the system.

Black Box Testing and Reducing Coupling8:37

So, how can we make a test to be more like a black box, that has less coupling? Of course, we still have need to have some coupling like the same tech stack, some kind of same platform, but at least we want to have a test that we, number one, we understand, and everyone who reads this understands. and it performs separately this regard on

our systems implementation.

Using Cucumber and Gherkin for Requirements and Testing9:09

So, to avoid it Like I think some some of you already know the answer, right? So, to write the requirement we use Cucumber and Gherkin. Right. Oh

Sorry. Gherkin is the language, and then Cucumber is with the more like the name of the platform, right? Right. I think you probably know the term called Given-When-Then. Right. On this attempt, what we are going to do is we are going to use Cucumber to specify the system environment in the way that you want it to be.

Using Cucumber to Precondition Backends10:02

For instance, if you are working on the class application or some application with the server side and client side, right? Your application will be stacked in a way like you have the web client or Angular client, whatsoever with client, then you have backend for frontend that deals with the data fetching and a lot of backend that you don't control. Right. So, we will use Cucumber to precondition those backends in a way that we can be sure that whatever we mark as a mock or precondition are properly validated with the

provider or can be, at least it can be, right? So, which comes into the second step. Means we want to use Normally when we write the

Mocking Servers with PACT11:00

test, right? You use Jest and then you probably mock some like some class, some instance, some function to return what you expect, right? So but at this time, we will use PACT to mock the whole server. And we will start the server on the test every time, right? So think about your test case, one of your test case, one of your scenario, you configure how your backend is going to behave.

And then you perform some action and then you can expect the immutable

response. Right. So, in this Indeed, on this, the On this session, I will use just Elysia I think I heard a good name here. Bun and React to demonstrate the idea. Right, but this idea also works with another kind of application. We also use this similar idea on our webpage that we build today.

Common Mistakes When Using Cucumber: It's Not Just Test Cases12:09

So, like, before we go, right, one big mistake when we are using Cucumber is All right. Cucumber itself is not the test case. is the matter of how you document your requirement and use it as your test case. You probably work on the company that asks you to, hey, yeah, just write Cucumber, and maybe you have a lot of the long Cucumber file that you don't understand what what what it is, right? So, like, Cucumber is there you only and always have three stages, right? So, other things else are considered as anti-pattern, right? So, we describe the immutable system condition before action happening on events. So, you start your test case on given. You describe how your database behave, how the backend behave, how your browser is going to be, right? Then describe everything under given.

And then, you describe the action that the user if you build the end user application, or the event if you build the event-driven application or backend, right? Let's describe the action. on when. Preferably, use only a single when clause. Right? Because multiple when clauses most of the time will make your test become something else, right? So, and it most of the time creates the anti-pattern. And then, on the last on the last section, it's just then. Then is the expectation. So, you know, precondition, you set given. Given I have this, this, this, this, when this happens, then this is my result. For instance, I am a user. I

I have a credit card. When I put my credit card on the system, then my credit card will be saved on the system, right? So, just simple, precondition and then action and then

outcome. Or if you are familiar with another form of test, especially in the unit test

Why Cucumber? Addressing System Dynamics in Integration Tests14:48

pattern, it is called arrange, act, and assert. Same pattern, different name. Right? And why, why Cucumber? I already spoke about it, right? The problem about the integration test is the dynamic. Anyone here work with a game company before? Like, anyone here build a game?

Yeah. No. Uh, on the game industry. Like, if you build a game, there is a concept. It's called, like, mechanic, dynamic, and aesthetic, MDA. Right. So, the concept is easy. Mechanic is the rule of the system or your game. Dynamic is how the user is going to use the mechanic or how to use the mechanic. So, they give the dynamic. So they give the aesthetic or feeling, right? So, the dynamic in this context, it means that the system when it got glued together or work together, right? They have many pre-conditions, many stuff can happen. And we can have many, many, many outcome based on each condition. Right? Let's say server down, or our server starts to return CAPTCHA. And then we need to

figure out how to deal with it, right? So, on Cucumber, it provides

very, very good pattern on how we deal with the dynamic or address the non-dynamic. Right? So, based on given, when, then clause, and with those given, when, then, it also allows us to run some linting rule to ensure that all of the

test cases, or the way we write or address the test cases are properly conducted. Right. So, Yeah, of course, you can also use this on another form, like, but also to make sure that your integration test works in the way that you can use this for document, like this pattern of arrange, act, assert

are required.

Right.

Model-Based Testing and Its Complexities17:06

Otherwise, if you do the model-based testing, Anyone here know the term called model-based testing?

Anyone here used to work with the QA, and QA draw you a figure

on the feature, and then draw the mind map on how things are going to work.

Yeah, yeah, you can start that. So, yeah.

So, I used to work with that, like mechanic as well. We call this model-based testing where you model your problem, and you try to understand the like every possible outcome. And the problem with that outcome is is too complicated. Like let's say you build a simple Agoda website,

It's not simple, by the way. So So you have to start from the first page, right? You search for the hotel, and then you go to the room, you select room, you go to booking form, you do the payment, you check the after booking or like MMB page. This is what we call like I don't remember the full name, right? So, you check your bookings, and then this is the end of the test case. Think about you need to write the automation from the first page to the last page. How long is it going to take? And how hard is it going to make you need to precondition your application.

Right?

The Need for Effective Testing Patterns18:41

Like it's going to be difficult, right? So that's why we need some pattern, some right pattern to ensure that we know what we are testing, right?

So, I think I speak a lot of boring principle,

I want to build something.

Building a Rock-Paper-Scissors Game: A Practical Example19:01

now, and that is the way I speak. So, yeah, one motto that I love

about being agile is working software is the measure of progress. So, yeah.

So, we want to build something. Oops, so small. We want to build a basic online rock-paper-scissors game using server-client, right? So, the mechanic The game rule mechanic is simple. We have rock win scissor. Everyone know it. Everyone know how to play, right? So, and some constraint to make sure that we don't or I don't overcomplicate the example I'm going to demonstrate to you is we're going to have

server. Server side will randomly pick

the answer and then will decide whether the game who wins the game, to be precise, right? So, and then we have a client side that just sends the information on what you pick to the server and server will tell you whether you win or lose the game.

Easy, simple mechanic, right?

Tech Stack: PACT, Test Containers, CucumberJS, Bun, and Elysia20:13

I put the QR code here. You can scan. It's not the prompt pay, it's the link to the GitHub. which is like later, sure. Right. So, yeah, the tech stack is simple. We have Pact. We have This blue box, it's called test container. Anyone here know Docker? Yeah, I think everyone knows, right? Docker Test container itself is the way that you write the program on your favorite language that talks to the Docker engine. The same way that you write the Docker file. Right, it allows us to dynamically create the container at the need or programmatically create the container at the need of our like requirement. And we're going to use it to create our mock server. Right. Yeah, just a small thing. Kubernetes also does the same thing as the test container, but with larger scale, right? So, another thing we're going to do is the CucumberJS. Actually, they have many languages. But since we are BKK.js, so I use JS at this session. And also I think everyone knows this icon? Like we're going to use Bun to build the application. Right. And the reason I chose Bun is because it looks cute, nothing in particular. Yeah. And we use another

framework, a popular framework from Thai engineer, right? I don't remember who built it. It's called Elysia. Nice name, I like it. Yeah. Cool. So let's see how we can build this software and piece the information together, right?

Building the Rock-Paper-Scissors Application22:05

So, client side handle the user input, send to the server, server decide whether you win or lose. Right. And server randomizes. In this situation, the server we are going to build will produce the random output. So if we want to build a test case that's end-to-end, it's going to be very difficult to predict the server outcome, right? Which, luckily, is the same way as the software dynamic that we are working on, right? And when you work on many teams, servers team can change the behavior of the server without informing you. So, in this example, I'm going to demonstrate the way we can mock those servers, the way that we can ensure that the server knows what they are doing. Alright And then everything besides the server side where the client will be served under dev server because I don't want to make stuff super complicated, right? So, the client side will have the BFF, back-end for front-end as a dev server. Right. So, how can we make it happen?

At first, right? I want to be BDD. I really want to be BDD. I am a big fan of BDD. So, I start the project. I hope it like, I think it's a little bit too small.

Perfect.

Okay.

Perfect.

Okay.

Project Structure and Consumer-Driven Contract Testing24:07

The battery is out for both microphones. So, yeah. So, we

like, I don't know how to make it bigger, but, yeah, the example project contains three folders, right? The first folder is the acceptance means when we do BDD, we create the acceptance test, right? So, and the second server This server is a little bit tricky. It should be on different GitHub, but I want to make it simple, so I put it here. Right, the server side, consider the server side as some server that another

person not you or not your team or not your company built. Right. You just write the client that consume the data from them, right? So, and then client is what we are going to test, right? So, what we are going to do is two things. We want to do BDD, and we want to do the second technique called consumer-driven contract testing. All right. Anyone know this term before?

Oh, yeah. Done. Who knows? Right.

Yeah. So, done.

Maybe you need to zoom in your IDE or VS Code. It's kind of too small. Okay.

But I think I can, I need to open it here. Command? Control command?

I see. How to make it bigger?

Yeah. มัน มันไม่ใหญ่ขึ้น

ขอโทษ อืม เห้ย ไม่ขึ้น โอเค ไม่ขึ้น มันต้อง หรือว่าเราดูใน GitHub เลย ครับ VS Code ใช่มั้ยครับ เปิด VS Code หน่อยได้มั้ยครับ

VS Code. Okay. ขอบคุณครับ ขอโทษนะครับ

ฮะ ใช่ครับ ขอบคุณครับ

Beginning the Acceptance Test and Consumer-Driven Contracts26:37

Okay. Yeah, it's a little bit of the technical hassle. So, yeah, going back to our beloved server. So we want to begin the acceptance test because we we want to write a test first. So we want to be a real test driven. Right? And also we want to make sure that our test servers are very stable. So we want to use the consumer-driven contract testing. Like under the hood, it's just a simple way to write the mock server and it's going to generate the the rule of that mock server means this request is going to respond like this. And both request and response will be called contract and it's going to get used by the

server side to validate that to validate that change against us.

So the concept is pretty simple. It's just if you're working as a backend engineer, you probably have a lot of questions on whether I changed the behavior or changed the API, if it is going to break my website or someone who consumes my API or not. Like, or, if you are working with the front-end API you probably want to make sure that server does not send something strange, right? and consumer-driven contract testing works in that way. But we will begin first with the

Writing Feature Files and Defining Rules and Scenarios28:00

feature files. Here is how I write the feature file. Very simple. I begin first with the rule. Rule, simple, simple term, not fancy like scenario. Rule is, you describe what can do, what cannot do. Right? So this is what the rule said. A match will decide the winner based on the option player and NPC pick. Right? So, scenario will be easy, right? You pick something, server decides whether you win or lose, and then you show to the user, right? So, yeah. You pick. Yeah, I put the bullet on given and when and then like this, so it's probably hard to understand, but let me show you the real working software, right?

Okay, this is the real working software. Like when you Yeah, the UI is pretty rough. So, rock-paper-scissor. You pick one, and then

Yeah, if you are lucky, you win. Yeah, if you are not lucky, you lose. Server will randomly give the response.

The Challenge of End-to-End Testing with Random Server Responses29:18

And now, the challenge is we need to make sure that we create the test that we be able to

run it properly. Right. And it should be end-to-end test. It could Like we can make the test very stable by writing the unit test on each component by rendering the component and then like or write some mock, right? But now, I don't want that component. I want to be end-to-end. So, The game works like this. The rules are on the server side and we have to test it. So what we do first, we create the feature file this like this. Right. If you work with the team with the business or with the customer, this is where you need to write the document on. And you need And you This document will serve as your

Yeah, we have We have We have other sessions before this, so they are They might already drain all batteries. The battery is out. So,

Yeah. So we want to continue after that, so

Building the UI and API for the Game30:40

we have the feature file already here, right? So we build the server and client by the then we start building the code, right? So we want some UI since it's tangible and we can like touch it and feel it. We build the UI first. So we start build a UI, we bootstrap. This one we are bun in its command and then boom! everything everything happened here. Right. Then we go into app.tsx.

Right. So next, going to be very simple. What we're going to do is we create one we create the app component An app component has one function that call handle play. In each handle play,

it is the event that got fired after onClick on each button that we create below. Right, we create three buttons, rock paper scissor. We put data-testid, so it is easier for like automation test. Then, this one After you click on each button, it sends the request to the API.

Normally, normally this one you probably done it to your backend for frontend. Like your frontend code probably doesn't look like this. But since I want to demonstrate to you fast, so like The API itself has one function, extra function, so we can override the target API on each, each time you start the, server, right? Or start the UI. Now the UI is done, you just send the API, and the API is API match action. Like, just, we send the action to API on the match, right? So then, respond. Nothing fancy, unless it's error, we don't, we do not, we just set the, like, the data that responds from the API. API decides, right? So the response here. No logic in the client side. Then, the API we do it. So let's say we are on different teams, we build this API. Then we start it on the Elysia server, right? So we create one action here, one post action here. receive the body, receive the player action and then randomly pick the NPC choice, means NPC non-player character, right? NPC choice. And then it evaluates whether you or them or NPC are the winner, right? Evaluation is like not that big deal. So we just do some small calculation here. in the server side logic, so we will skip like but this thing works. So

Yep.

Setting up the Automation Tests with Cucumber.js and PACT33:35

After that, the most challenging part is here. So we want to build the the automation, right? So we use

this package. So at first, what we do is by

So we use bun add cucumber,

@pact-foundation for our consumer driven contract testing. And test library, this one is not needed, it just use it for boilerplate code. And then yeah, this one is as IO, we need it. for doing some magic behind Get port as well to doing some magic behind. And then another one, test container. Right. Another one is the node, obviously, right? So we have Playwright. So everyone, fancy about Playwright. This part, it's just the wrapper of the Selenium, but it's cool wrapper, by the way. So, yeah, we have Playwright here. So, how do we define the test? Right. So, in

Cucumber.js, the pattern that we we work with the Cucumber.js, they tend to be different than other languages. So like,

Yeah. So what we do here is we create the step binding. Right. Every time when you have the feature file, but don't have the step binding and you run the Cucumber, it's gonna tell you automatically the snippet that you can copy and paste into the new and into your bindings, which I can show you for a moment. So let me rename this. So, like

ครับ So, yeah. So we run it with bun. Run

Yeah, you see. Oops. So it will generate the

snippet of the step binding. Like pretty different than Java, or C# if you work with C# or other language that have the code generator. This one it generate on on during your run and you need to copy and paste on your step bindings. And after that you need to do some like template thing to make sure that you you you can pass the value

with the template properly, right? So, Yeah, this one is too detail. You can check the document later. Right. So, what I did here is I copy everything. And then the next thing is on the given cost I have the global where is global

variables address here or world object. If you're familiar with JavaScript, right? So, TypeScript does not have world object. So I need to declare it like this. Then this one we collect the test state. Fortunately, on each Cucumber run this global will be different. This this object will be different on each run. Like, thanks for the Cucumber folks, right? So, we collect the precondition from the given cause here.

And we we do nothing fancy, right? We don't do we don't perform any direction. We don't even open the browser here. Right. on our given code is more like builder pattern, if you familiar with it, like you configure your object this way, this way, this way. and you'll be with

The 'When' Clause: Performing Actions and Setting Up Mock Servers37:36

And now we're going to go to the more exciting part, which is when Right. on when This is where the action got performed. We set up the server. We set up the mock server. And then we perform the action on this method.

How can I set up the server here? So, let's start with our first function. There are a few steps that I create here, right? Number one,

we want to build the Pact contract. consumer contract. Then we start test. Sorry to disturb. You are running I'm afraid you already ran out of time, so maybe you can take like another five minutes. Oh, yeah, yeah. Sorry for running out of time. So, yeah. Let's continue fast. Then we have the server here. We have the server and then this is where we start the PACT server.

How PACT Works: Building and Verifying Contracts38:38

Okay. So, how does PACT work? Let's go to build game contract, right? Your game contract itself will invoke PACT. What we did here are the First line is bootstrap. You can check how PACT works. The most interesting parts are here. So, you configure how your server is going to behave here.

Then, you run verify and finalize, so PACT is going to dump the thing called contract.json at this state.

Right. So, I'm afraid I don't have that much time, so yeah.

Starting the PACT Stub Server and Game Client39:16

We have the contract.json, and then next we start the PACT stub server. Simple container code. that we create the container, Docker container, and from PACT foundation, and then mount the contract path

to the container and start it on the random port, right? We start a random port. And then we start the game client. Since it's like the most dirtiest way to start the game client because we want to start the browser, right? We don't want to render our test. We start the browser, we execSync, we create the command line, we override the server here. And then, boom, the magic happened. Below, we launched the browser. that talk to the client side. or webpack-dev-server here. Then, it's not webpack, it's bundle server. Then after that, we just click on the button, if you notice.

Right? And then, yeah, just expect the result. Right.

Reviewing Test Results and the Importance of Contracts40:26

So, Alright.

Cool.

I think something is going to crash. Yeah.

Now, we have two test cases, or three. Yeah. Well, one test case failed. Maybe I did some modification, but two passed. Yeah. So we try to validate the rule based on this thing. Right. So, one test case failed, which is the draw. Maybe I put the wrong term here. And after that, yes, in the end. So I want to show you one thing, very cool.

This is what we call contract. And this is why the magic happens. This contract is the configurator of your microservice or your environment. And this contract can be consumed and used by the server. Right. So we can load this file and use it as their own test. So, server side, can be sure that when you change something, when you refactor something, you know what to test against your consumer. Right. So, unfortunately we're running out of time. So I have, we've crossed the session at this point.

Q&A and Closing Remarks41:44

So, does anyone have a question?

Right.

Does anyone have any questions? You can raise your hand up. I will send a mic to you.

Nah.

Uh I wonder how many test cases can it be? It depends on how we write the Cucumber file. Yeah, it depends on how many Cucumber or how many

scenarios you want. Oh. Yeah. And you can create this full stack application per one of your scenarios. Oh. Yeah. everyone at your run host machine allow to do so as well. But it's kind of like we don't need to specify all the test cases by ourselves. But like give it some example and Yeah. Oh. So the rest will be generated. Yeah, the rest will be generated. Right. So it's like we need to

do some manual work of course if the interactions are not there. But if the interactions are there, you just reuse whatever the builder has and then just change the parameters. Oh. Yeah. Thank you. Yeah.

So Yeah, so please give an applause. Thank you.

Okay.