
In a past life I spent some time working as a Build Engineer for a co-development studio, so I was fortunate enough to get a good look at a bunch of different games and a bunch of unique (and common) challenges.

There was one challenge that was evidently consistent; game builds can be HUGE and you need to move them between build machines, to cloud storage, onto servers, down to developer machines, onto player's machines, etc., and do it all performantly and securely.

_TIP: skip to the [In Action](#in-action) section below if you need some convincing to keep reading!_

---
<aside>
__A quick aside about Build engineering:__

Build engineering is hard; the iteration cycles are long, the problems are vast, the requirements are many, the knowledge is niche and often not talked about or shared, and the compliments few and far between. When the build is green the build guy is a phantom, a ghost, a myth, a legend, but when the builds are red, they are on everyones' speed dial and the center of attention. If your team has a build guy, buy em' a beer, they could probably use one.
</aside>
---

This common problem of moving large files around is nothing new to the Game industry. Long gone are the days where players would <a href="https://en.wikipedia.org/wiki/Doom_(1993_video_game)#:~:text=The%20team%20was%20unable%20to%20connect%20to%20the%20FTP%20server%20at%20the%20University%20of%20Wisconsin%E2%80%93Madison" target="_blank">download Doom via the FTP server at the University of Wisconsin-Madison</a>. Today most studios have a solution for performant build distribution, whether it be public like Blizzard's Battlenet Launcher, or Epic's Epic Game Launcher, or the incredibly popular Steam and itch.io platforms, or internal only via some special tools only used by the development team, most teams have this covered in some way. That said, I've also seen and heard of plenty of teams that just pass around `.zip` archives via Google Drive. I think teams can do and deserve better than that though.

## Why am I even thinking about this?

Building your own launcher/patcher probably doesn't make sense for most teams, if you can ship on an existing platform like Steam then you probably should, but when it comes to distributing internal only binaries used by the development team, or when you need to copy large files (or large quanities of files) between build machines in the cloud, you'll need something more custom. Game patchers are most synonymous with distributing builds to players, but they can be used for these other purposes as well.

Note that in the case of Unreal Engine you can utilize something off the shelf like UGS (Unreal Game Sync) provided by Epic, but that doesn't cover distribution of binaries between build machines & servers, so you'll find teams that build something custom or build ontop of something like rsync. 

These are the usecases which have me thinking about game patchers, how they work, and if I could iterate on any of the approaches here. Additionally, I wanted a reason to learn Rust and this seemed like a good reason. Also... I'm hard stuck Iron in League of Legends, and this feels like a more constructive use of time than dying over and over to Teemo top lane cheese.

### Unreal Game Sync & Editor Binaries

Lets talk briefly about UGS.

Unreal Game Sync is a utility application distributed by Epic as source code along with the Unreal Engine source. When a game team pulls down the source for Unreal Engine and uploads it to their VCS (Version Control System), one of the first tasks the build guy might tackle is a build job to compile Unreal Game Sync and save the artifact into Perforce (or whatever VCS they are using). Their team can then pull down the UGS installer, install UGS on their machine, and then each morning use it to "sync editor binaries" built from the nightly build including the Engineering team's changes from the previous day. Sounds great right? Its definetly a good place to start, but P4 as a mechanism for distributing these binaries sometimes falls short, and the perforce server is likely already busy syncing workspaces on all those build machines you have.

__sync editor binaries__

Running "sync editor binaries" when using UGS's default mechanism (which pulls from Perforce), can easily take minutes (5, 15, 30, etc.) depending on how large a game's editor binary artifact is, and what other tooling they ship with it. Doing this only once each morning might not be that big of a deal, but if a engine team is constantly shipping new editor changes, then this time adds up and it can be a real distraction and source of friction in the development process, can easily turn into a DDOS on your perforce server, and can be a signifcant source of stress and hair loss for your build engineer.

During my time as a Build Engineer, it was a common struggle trying to optimize this process, especially for teams that wanted to pull binaries more frequently.

## What is osyrs

At this point you are probably thinking; Ok, cool story bro -- thanks for the history lesson and UGS / UE refresher. Now what is osyrs? Or maybe you left and aren't even reading anymore... but if you are, thanks for sticking it out this far.

osyrs is my "Binary Patcher" pet project. Meaning it is a tool that aims to support the "patching" of binary files... which means it enables the migration of a file from the current version to a later version of the file (well it will someday).

### Functionality

Currently osyrs has this REVOLUTIONARY (/s) functionality:

- Patching: Create a binary patch file representing the difference betwen two versions of that file
- Patching: Apply a binary patch to upgrade one version of a file to the _next_ version of that file
- Chunking: "Chunk" a file into pieces defined by its contents
- Chunking: Reconstruct a file fully from its previously computed chunks

### Patching (bsdiff)

Patching is the first approach I took, and is somewhat simple/straightfoward. Patching supports creating a "patch" file that represents the binary differences between two versions of a file. With a patch file you can upgrade one version of a file to its next version by "applying" the patch file to the original. osyrs leverages the `bsdiff` algorithm through a Rust crate to accomplish the diffing and patching.

While straight forward (apply this one file to upgrade file `X` from version `1.0` to `2.0`), it has its downsides and limitations, such as when you have version `1.0` of a file but need to go to version `3.0` of that file. How do you "skip" versions like that? Well, you have to compute a patch that represents the difference between `1.0`, and `3.0`, which works but over time if you have many versions of your application/game, these patches going from `1.0` to _some other version_ are going to add up, and you need to have a patch for each potential starting version leading to the latest version, computed each time you create a new latest version. Thats alot of work. Additionally, there are times when a seemingly small change to a source file can result in a rather large patch file being generated, so the related IO and applying of patches isn't incredibly performant either. Riot Games has written some great articles on these topics, which I've linked below in the references section. I highly recommend you give Riot's tech blog a read (finish this article first though of course!).

### Chunking (Fast CDC)

Chunking is the second approach I've tried, and it is much more complicated and rigorous than the simpler patching approach. Chunking aims to re-use portions of the original file (which is already present on disk) and only apply changes to the sections of the file which are different in the next (or several versions later) version of the file. This might sound very similar to the above patching approach but there are some very important differences, namely supporting "jumping" versions of a file, meaning you can update a file from version `1.0 -> 3.0` without first updating from `1.0 -> 2.0` and then `2.0 -> 3.0`. Additionally, chunking can help optimize the network IO involved with fetching the differences that need to be applied during the upgrade process, as the aim is to ownly fetch the chunks which are actually missing from your current file, nothing more. Again, Riot does a great job of covering these concepts in more depth on their technology blog.

To accomplish this approach I am leveraging the "Fast CDC" (Content-Defined Chunking) algorithm (also through a Rust Crate) to compute the "chunks" that make up a file. Using Fast CDC I am able to compute the list of chunks in a file, logically group them for _theoretically_ performant retrieval from disk (and later network) and access in memory, and then re-assemble the file with those chunks.

Practically speaking, what this looks like is grouping chunks into `.occz` (osyrs chunk container zip) files, and then referencing all `.occz` files composing a file into a `.occmz` (osyrs chunk container manifest zip) file. `.occz` and `.occmz` files are stored as binary files using the Message Pack interchange format, and then compressed with Brotli. I'm using Message Pack because I knew I would want to store these files in a Binary format, and I knew I didn't want to spend time writing my own serialization and deserialization code. I may do a small bit of custom framing in the future to support different types of serialization for the data payload portion of these files, but I'll explore that later if I get there.

__Rust Structs__

```Rust
#[derive(Debug, PartialEq, Deserialize, Serialize)]
pub struct Container {
    pub total_size: usize,
    pub map: HashMap<String, Vec<u32>>,
    #[serde(with = "serde_bytes")]
    pub data: Vec<u8>,
}

#[derive(Debug, PartialEq, Deserialize, Serialize)]
pub struct ContainerManifest {
    pub chunk_ids: Vec<String>,
    pub chunk_hash_to_container_hash: HashMap<String, String>,
}
```

An interesting note is that the aggregate size on Disk for the various `.occz` and `.occmz` files is relatively about the same size as the source file. For example, using a 28 MB zip as the source file, the resulting directory on Disk on Windows is 28.4 MB, contianing 253 `.occz` files, and a single `.occmz` file. I would love to get this down to be smaller than the actual source file (maybe through better compression). Serializing as Message Pack does adds some size to each file though, and I add new data into each file to describe its contents, so there is naturally more data than just the source contents.

That said, what I am optimizing for is the reconstruction and patching of a file either from scratch or from one version to another, not necessarily the storage size for the remote patching system, so I believe this is a reasonable tradeoff at present.

<a href="#in-action" id="in-action">
## In Action
</a>

In the below examples I first chunk a file, and then reconstruct it using the processed chunks. The input file (`2d-rpg.zip`) is a zip archive of a small 2d Unity game compiled and built for Windows. This zip archive is only about 28 MB compressed, which isn't anything impressive. I'll need to run these tests against a much larger file(s) (such as Unreal Editor Binaries) in the future.

Chunk the file:

```bash
RUST_LOG=info ./osyrs.exe chunk "./2d-rpg.zip" -o "./out/chunks"
```

<img src={"/images/articles/2024-12-02-osyrs/osyrs-ff0aff323-chunk.gif"} />

Reconstruct the file:

```bash
RUST_LOG=info ./osyrs.exe reconstruct \
-m "./out/chunks/manifest.occmz" \
-c "./out/chunks" \
-o "./out/reconstructed/2d-rpg.zip"
```

<img src={"/images/articles/2024-12-02-osyrs/osyrs-ff0aff323-reconstruct.gif"} />

Demonstrate that the file was reconstructed correctly (shoutout to Windows for taking forever to decompress the file):

<video controls>
  <source src="/images/articles/2024-12-02-osyrs/osyrs-ff0aff323-reconstructed-game.mp4" type="video/mp4"/>
</video>

Notice that in this example the chunking and reconstruction operations each take less than 1 second to complete. Seems pretty good to me, even with the test file being pretty small (28 MB).

## What's Next

Now that I have what I feel is the basic functionality (recreating a file from nothing), the next step is to be able to upgrade files rather than just reconstruct them from scratch. This will involve inspecting an existing file, detecting which chunks the current file has, determine which chunks are missing, and then apply those chunks in the correct way so that the file is patched properly.

Once I have this ability to patch/upgrade a file from one version to another, I'll then either approach networking and fetching the `.occz` and `.occmz` files from a CDN or Cloud Bucket (S3, etc.), and/or explore applying these concepts to a directory of files, rather than a single file. This directory recreation/upgrading will be necessary functionality for upgrading an application/game on disk from one version to another without needing to first have a compressed archive or single file version of the application/game available.

I'm excited to keep exploring this project, and to some day escape Iron in League of Legends. Keep an eye on this blog and my other socials for updates if you want to follow along.

Cheers!

## References

- <a href={"https://technology.riotgames.com/news/supercharging-data-delivery-new-league-patcher"} target="_blank" rel="noopener noreferrer">Riot Games: Supercharging Data Delivery: The New League Patcher</a>
- <a href={"https://fasterthanli.me/articles/efficient-game-updates"} target="_blank" rel="noopener noreferrer">Efficient game updates</a>
- <a href={"https://www.daemonology.net/bsdiff/"} target="_blank" rel="noopener noreferrer">daemonology.net/bsdiff</a>
- <a href={"https://joshleeb.com/posts/content-defined-chunking.html"} target="_blank" rel="noopener noreferrer">Content-Defined Chunking</a>
- <a href={"https://msgpack.org/index.html"} target="_blank" rel="noopener noreferrer">Message Pack</a>
- <a href={"https://github.com/google/brotli"} target="_blank" rel="noopener noreferrer">Brotli</a>
- <a href={"https://rsync.samba.org/"} target="_blank" rel="noopener noreferrer">rsync</a>


I've been playing a fair bit of [Valheim](https://www.valheimgame.com/) recently, and I wanted to take a stab at hosting my own server on AWS EC2. Like any other sane and not overengineered solution, I tried to leverage good infrastructure as code and configuration as code practices so that my server was quickly reproducible (this has been very helpful). Given that, I decided to use Ansible to configure the EC2 instance.

The rest of this article will describe how I used Ansible to check if save files existed in S3 while configuring the EC2 instance with Ansible.

__Why is it necessary to check if an Object exists?__

The `amazon.aws.aws_s3` module will throw an error when downloading a file that doesn't exist. Because of this, it can be helpful to check if that Object exists before downloading it. Another scenario is that you may want to check if an object exists before replacing it with a new or default object. The sky is the limit!

## 1. List Objects in the Bucket

Ansible provides a helpful module for interacting with S3 buckets. The module is called `amazon.aws.s3_object`. Using this module, we can list all the Objects in a bucket and then check if a specific Object exists. The module is well documented, and you can find the documentation [here](https://docs.ansible.com/ansible/latest/collections/amazon/aws/s3_object_module.html).

Using `mode: list,` I list all of the objects in the `valheim-saves` bucket and store the result in a fact called `objects_in_saves_bucket`. 

```yml
- name: List Objects in Saves bucket
  amazon.aws.aws_s3:
    bucket: "valheim-saves"
    mode: list
  register: objects_in_saves_bucket
```

## 2. Store the Desire Object Name in a Fact

Storing the subject Object name in a fact is a convenience step. This way, I could easily change the name of the Object I was looking for and not have to change it in multiple places.

```yml
- set_fact:
    world_save_fwl_file_name: "save-file-name.fwl"
```

### Dynamic Object Name Example

 Ansible Lint discourages using interpolation in conditional clauses, such as `when`. When I want to use a dynamic object name, I get around this by storing the object name in a fact, which makes Ansible Lint happy and makes the code more readable.

```yml
- set_fact:
    world_save_fwl_file_name: "{{valheim.world}}.fwl"
```

## 3. Store the State of the Object in a Fact

I wanted to store the state of the Object in a fact. This way, I could easily reference the Object's state in a later conditional.

```yml
- set_fact:
    s3_bucket_has_fwl_save_file: "{{world_save_fwl_file_name in objects_in_saves_bucket.s3_keys}}"
```

## 4. Download the S3 Object if it Exists

If the value of `s3_bucket_has_fwl_save_file` is true, then we will download the Object from S3 into the desired location.

```yml
- name: Download World Save FML File from S3
  when: s3_bucket_has_fwl_save_file
  amazon.aws.aws_s3:
    bucket: "valheim-saves"
    object: "{{ world_save_fwl_file_name }}"
    dest: "ubuntu/valheim/saves/worlds_local/{{world_save_fwl_file_name}}"
    mode: get
```

## Putting it all together

After putting it all together, we have the following playbook with steps that will check if an object exists in an S3 bucket, and if it does, download it to the host.

```yml
- name: List Objects in Saves bucket
  amazon.aws.aws_s3:
    bucket: "valheim-saves"
    mode: list
  register: objects_in_saves_bucket

- set_fact:
    world_save_fwl_file_name: "save-file-name.fwl"

- set_fact:
    s3_bucket_has_fwl_save_file: "{{world_save_fwl_file_name in objects_in_saves_bucket.s3_keys}}"

- name: Download World Save FML File from S3
  when: s3_bucket_has_fwl_save_file
  amazon.aws.aws_s3:
    bucket: "valheim-saves"
    object: "{{ world_save_fwl_file_name }}"
    dest: "ubuntu/valheim/saves/worlds_local/{{world_save_fwl_file_name}}"
    mode: get
```

## Conclusion

I hope this article was helpful. If you have any questions or comments, don't hesitate to get in touch with me via any of my social media [links](https://linktr.ee/viglucci). Thanks for reading!


This article will provide an overview of how you can build and release an [oclif](https://oclif.io/) application via GitHub Actions. The pattern we will implement supports the following workflow:

- Create a release in a project's repository on GitHub
- Publish the release, which will trigger a "workflow" in GitHub Actions via the Release Published event
- Workflow attachs results to the release as "artifacts," allowing them to be downloaded later

## Technologies & Concepts

First, lets talk alittle about the technologies/concepts we'll be using.

__oclif__

"The Open CLI Framework" ([oclif](https://oclif.io/)) is an open-source framework for building CLI (command-line interface) applications with standard technologies such as Node.js, JavaScript, and TypeScript.

__oclif packaging support__

oclif is especially appealing when you wish to produce binaries that work natively on multiple operating systems. This support is provided through several commands available on the `oclif` CLI. You can learn about this in more detail in the ["Release"](https://oclif.io/docs/releasing) section on the oclif documentation website. Still, we'll give a brief overview of the strategies below.

__Standalone Tarballs__

For projects where you can not expect users to install node.js and npm, oclif provides functionality for producing builds that will output standalone tarballs. Standalone tarball builds can be created for multiple supported operating systems. For the sake of this article, we will be working to produce an installable Windows executable. 

__NPM__

Because oclif is built around node.js and npm, an oclif app can be published directly to npm using the standard `npm publish` command. This strategy allows existing node.js and npm users to easily install the application globally using the `npm install -g <app_name>` command.

This strategy is likely sufficient and the most straightforward for public projects or projects where the target user base is expected to have node.js and npm installed.

__GitHub Actions__

[GitHub actions](https://docs.github.com/en/actions) is a CI/CD solution provided directly from GitHub that allows you to run automation tasks based on various "events" that occur on repositories hosted on github.com.

__Building with GitHub Actions__

GitHub actions can run tasks and commands on a repository in response to events. One supported event is the "release published" event, which we will use to trigger a workflow.

__GitHub Releases__

Releases on GitHub allow maintainers to communicate meaningful milestones for their projects and distribute "artifacts" related to those milestones. For this example, we will be creating a new Windows installer for a CLI application and attaching it to the desired release as an "artifact."

To view the releases for a repository on GitHub, you can visit the releases page at `github.com/REPOSITORY_NAME/releases` or click the "releases" header on the repositories index page.

The releases header can be challenging to find, so I've provided an example below.

![GitHub releases header example image](
  /images/articles/2022-02-21-how-to-build-oclif-for-windows-with-github-actions/releases-header-example-2.jpg)

## Getting Started: Create a workflows file 

To get started, we need to ensure that we have a `.github/workflows` directory created in the repositories root directory.

example: `project-folder/.github/workflows`

Next, inside the `.github/workflows` directory, create a `release.yml` directory. This YAML file will configure the "actions" that will run in response to the events on the repository.

example: `project-folder/.github/workflows/release.yml`

### Setup event triggers

This configuration will trigger the workflow anytime a new release is published on the repository.

```yaml
# .github/workflows/release.yml
on:
  release:
    types: [published]
...
```

### Configure jobs

This configuration provides a few bits of functionality to the workflow:

- Creates a list of ["jobs"](https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow) that will run when the workflow triggers
- Defines that the job should run on and use the latest "ubuntu-latest" image as a base
- Defines a "strategy," which allows us to define variables that can be referenced later

```yaml
# .github/workflows/release.yml
...
jobs:
  release:
    name: release ${{ matrix.target }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - target: win
            artifact_glob: "./dist/win32/*"
...
```

### Strategy matrix variables

The `strategy.matrix` is a configuration that allows for running workflow jobs with a combination of variables. Matrix is generally helpful if you wish to run the same set of tasks with different combinations of variables, such as versions of dependencies, or in our example, arguments passed to a CLI command.

For our purposes, we are using a `strategy.matrix` that targets a Windows build but could be extended with additional entries to target other platforms later.

You can learn more about matrix configurations [here](https://docs.github.com/en/actions/using-jobs/using-a-build-matrix-for-your-jobs).

### Steps

Last we will configure the "steps" that will be run when the job is executed.

```yml
# .github/workflows/release.yml
...
    steps:
    - run: sudo apt update
    - run: sudo apt install nsis p7zip-full p7zip-rar -y
    - uses: actions/checkout@v2
    - uses: actions/setup-node@v2
      with:
        node-version: '16'
    - run: npm install -g yarn
    - run: yarn
    - run: yarn global add oclif
    - run: oclif pack ${{ matrix.target }} -r .
    - name: Attach artifacts to release
      uses: svenstaro/upload-release-action@v2
      with:
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        file: ${{ matrix.artifact_glob }}
        file_glob: true
        overwrite: true
        tag: ${{ github.ref }}
```

A description of what each "step" accomplishes is provided below:

 1. `run: sudo apt update`
    - Updates the libraries already installed on the `ubuntu-latest` base image to ensure we have the latest and greatest
 2. `run: sudo apt install nsis p7zip-full p7zip-rar -y`
    - Installs several dependencies required to package the application as a Windows installer using `makensis`
 3. `uses: actions/checkout@v2`
    - Uses a prebuilt GitHub action to check out the repository source code
 4. `uses: actions/setup-node@v2`
    - Sets up node.js and npm, specifically using version `16` of node.js
 5. `run: npm install -g yarn`
    - Installs `yarn` globally so it can be used in later commands
 6. `run: yarn`
    - Calls `yarn` to install the dependencies listed in `package.json`
 7. `run: yarn global add oclif`
    - Installs the `oclif` CLI globally via `yarn`
 8. `run: oclif pack ${{ matrix.target }} -r .`
    - Calls the `oclif` CLI to package the application using the "target" previously defined in the matric config
    - example: `oclif pack win -r .` will package for Windows
 9. `uses: svenstaro/upload-release-action@v2`
    - Uses a prebuilt GitHub action ([svenstaro/upload-release-action@v2](https://github.com/svenstaro/upload-release-action)) to attach the artifacts from the `oclif pack` command onto the GitHub release with the tag denoted by `tag: ${{ GitHub.ref }}`
    - Notice the `file` attribute with the value `matrix.artifact_glob`? This value references the variable we originally defined in the `matrix` configuration. Practically speaking, this will result in every file in the `/dist/win32/` directory being attached to the GitHub release.

### Putting it all together

Below is an example of the entire `release.yml` file.

```yml
# .github/workflows/release.yml

on:
  release:
    types: [published]

jobs:
  release:
    name: release ${{ matrix.target }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - target: win
            artifact_glob: "./dist/win32/*"
    steps:
    - run: sudo apt update
    - run: sudo apt install nsis p7zip-full p7zip-rar -y
    - uses: actions/checkout@v2
    - uses: actions/setup-node@v2
      with:
        node-version: '16'
    - run: npm install -g yarn
    - run: yarn
    - run: yarn global add oclif
    - run: oclif pack ${{ matrix.target }} -r .
    - name: Attach artifacts to release
      uses: svenstaro/upload-release-action@v2
      with:
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        file: ${{ matrix.artifact_glob }}
        file_glob: true
        overwrite: true
        tag: ${{ github.ref }}
```

## Conclusion

Once the GitHub Action workflow file has been added to the repository, future published releases will trigger the workflow. The produced artifacts from the workflow will be attached to the release and available for download.

![GitHub releases page with attached artifacts](
  /images/articles/2022-02-21-how-to-build-oclif-for-windows-with-github-actions/release-attachments-example.jpg)

Note: Ensure that release versions are tagged with a corresponding Git tag, and the appropriate tag is added to the release on GitHub, as these tags are required by the *svenstaro/upload-release-action@v2* action. Tagging a release version is as easy as using the [npm version](https://docs.npmjs.com/cli/v8/commands/npm-version) or [yarn version](https://classic.yarnpkg.com/en/docs/cli/version) commands.



This article will overview practices and areas to focus on when navigating a systems design interview at a high level. I expect this information to be most helpful to engineers in the 1 - 3 years of experience range or those who have never done a systems design interview.

Additionally, this article is expected to be used as a jumping-off point, and readers will likely need to leverage other resources to dig deeper into the referenced concepts and terms.

## What you can expect

The shape of a system design interview will likely depend heavily on the company you are interviewing with and the level of the role you have applied for. For instance, an intern or an associate-level candidate might be asked to diagram how multiple classes or modules in a system would interact with each other. In contrast, a more experienced candidate might be asked to design an entire distributed system.

### Whiteboarding

These interviews are generally conducted in a "whiteboard" style, where the interviewer will ask the candidate to present their answer as a diagram. Before the increased prevalence of remote work, this would be conducted on a physical whiteboard; however, as more of the engineering workforce moves to remote work, these interviews are more commonly being performed with tools like [Excalidraw](https://excalidraw.com), [tldraw](https://www.tldraw.com/), [diagrams.net](https://app.diagrams.net/), and similar online whiteboarding tools. Ideally, the employer will let you know which tools they use for their interviews, and I highly recommend familiarizing yourself with the tool ahead of time. If the employer doesn't let you know which tool they expect you to use, don't fret, the interviewer shouldn't be grading your proficiency on using a specific tool.

### Questions

System design questions are designed to understand a candidate's ability to think deeply about systems and their requirements. These questions generally require a candidate to talk about a system they've already worked in, or more commonly, about a theoretical scenario.

Below are examples of questions that I've been asked, others have been asked, or that I've asked myself:

- Can you describe a system you've worked in recently?
- How would you design the backend for a social media app?
- How would you design the system for a parking garage?
- How would you design an e-commerce site?
- How would you design the content management system for a large blog?

For specialized industries, an interviewer might ask you a question specific to the domain they operate in, so it wouldn't hurt to understand some of the complexities of their industry. However, it is just as likely that an interviewer will ask you a generic question, so try not to deep dive too heavily into specialized solutions during your preparation.

## Prepare for the interview

The systems design interview can and should be something you deliberately prepare for, just like any other interview.

An easy way to practice systems design is to diagram the system you are currently working in and identify the components or areas that are undefined or unknown to you. This exercise should help you identify additional areas of preparation. For instance, your system may use a specific type of database or leverage a cloud provider for block storage. Do you know why the original designers chose a particular database technology? Or the benefits and tradeoffs of block storage? Looking into these and related topics more deeply should be part of your interview preparation.

### Your current work

If the system you are currently working in isn't very complex or exciting, you may be inexperienced and thus disadvantaged when designing a system from scratch. If this applies to you, it is recommended to spend time answering some of the example questions above, watching mock interviews on platforms like YouTube, and even asking your friends or trusted colleagues to run a mock interview with you.

## Seek out the customer

A system is only helpful if it serves a customer's needs.

### Identify the customer

A "customer" is any user of the system. These "customers" could be the actual human being using the mobile app that interacts with your system, or they could be other services inside a company's microservices architecture. As the system designer, you will need to ask yourself, "who is my primary customer?", and then keep the customer in mind as you design the system.

It will be vital that you identify who the primary customer of the system is and what requirements those customers have of the system. If it is not apparent who the customer is during the interview, it is recommended to be direct and ask the interviewer, "Who is the primary customer of this system?". The interviewer should be receptive to these types of questions.

Additionally, you can identify the primary customer by making assumptions based on statements made by the interviewer. For instance, if the interviewer asks you to "design TikTok" (a popular video-sharing app), you could assume that the customers are global users who primarily interact with the system using their smartphones. Once you think you know who the primary customer is, you should validate this assumption with the interviewer; "Is it safe to assume that my primary customer is global users who are using a smartphone application?".

### Identify customer needs

The "needs" of the customer can generally be defined as either "functional requirements" or "non-functional requirements." It is imperative to identify and validate requirements (both functional and non-functional) with the interviewer before you start providing your solution.

### Functional requirements

Generally speaking, you should start by defining the system's functional requirements, and your interviewer should be receptive to your questions about these requirements. For instance, given the question to "design TikTok," you might ask your interviewer if the below list of functional requirements satisfies the customer's needs. The interviewer will likely reply, "looks great to me!" and if not, you should try to dig in on what functional requirements you might have missed.

- upload short videos (length is 3 minutes or less)
- view a "feed" of videos
- "like" videos
- comment on videos

As you dig into your system design, these functional requirements should also expand to include the fundamental ways they will be satisfied. For example, it would be a good idea to list out the various API endpoints needed to satisfy each requirement and maybe even which database structures (tables, documents, etc.) might be required to support those API endpoints. Later on, we'll touch a bit more on when you should go into detail and also when you should leave things vague.

### Non-functional requirements

Along with functional requirements, non-functional requirements should be identified and vetted with the interviewer before you dig into or decide on system architecture and start creating a diagram. These non-functional requirements could include concepts such as:

- High Availability
- Low Latency
- Eventually Consistent (or Strongly Consistent)

When determining which of these are relevant to your system, you should always ask yourself how this requirement would serve or impact your customer (the end-user). For example, in the question to "design TikTok," you might presume that this system could support an architecture that is "Eventually Consistent." You could argue that it is more important that users can quickly access videos in their feed than for those videos to be the latest or most up-to-date. This argument would suggest that eventual consistency is sufficient.

When making these arguments or assumptions, you should again validate them with your interviewer. If you fail to validate your assumptions, you may be leaving yourself open to a "curveball" later if the interviewer decides to introduce a constraint contrary to an assumption you made and that your design does not support.

To validate your assumptions, you may pose the question to your interviewer; "Is it safe to assume that it is more important for the video feed to load quickly than it is for the videos to be the most up to date?".

### Knowing the appropriate level of detail

A common misstep when answering system design questions is when you provide too much or too little detail. Knowing and ascertaining the correct level of detail expected from your interviewer can be vital to answering your question correctly and sufficiently. Remember that interviews are a two-way conversation between the candidate and the interviewer, and becoming comfortable asking these questions will make you more successful. When describing a particular component of the system, take a moment to pause and ask your interviewer, "is this the level of detail you were expecting, or would you like me to give more or less?".

For example, prematurely giving too much detail can hinder you, especially when digging deep on a particular portion of the system in a sufficiently more detailed way than others. An easy example of this can be when describing databases. The chances are that the system you are designing will require some database or data storage, and you might be eager to blurt out the first database technology that comes to mind, or maybe whichever is your favorite. Unless you've established with the interviewer that they are interested in these finer details, try your best to avoid doing this. If you feel the urge to recommend a specific solution, be sure to confirm with the interviewer if they would like those details at that time, but it can also be advantageous to defer that decision.

By avoiding specifics where they aren't yet necessary, you allow yourself the flexibility to make this decision later when you potentially have more information. You also avoid showing a potential bias towards a solution or technology that may not fit the system's requirements. If you do go as far as to state that the database will be a NoSQL database, and even worse, will specifically use Cassandra, you've made a strong indication that you think these technologies would provide significant value to the system. The interviewer would be well within their rights to go deep on questioning these decisions.

### Don't show up with a solution

You've likely worked in a few different architectures, and you might even have a favorite design or pattern that you think works best. Experience and opinions are great, but this can be a trap when answering system design questions. It is best to avoid approaching systems with rigid pre-existing solutions or designs.

For example, suppose you are a big fan of event-driven architectures and are already set on or fantasizing about how you'll apply this pattern to whichever scenario the interviewer describes. In that case, you'll quickly fall short when the functional or non-functional requirements of the system don't lend themselves to the limitations of your predetermined solution.

Rather than showing up with an architecture already in mind, it is best to understand standard components and approaches and where they can be employed to satisfy different requirements of any given scenario. While the list below is not comprehensive, and the best components and approaches will vary heavily based on the system, it is recommended to understand where each offers value and which tradeoffs they present. These topics are likely to be expected knowledge for mid-level and higher backend engineering interviews.

- Vertical vs. Horizontal Scaling
- Monolith vs. Microservices Architecture
- Database Approaches (Relational, NoSQL, Graph)
- Event/Message Queues
- Load Balancers
- Block/Cloud Storage
- Content Delivery Networks (CDN)
- Serverless/Lambdas

## Next Steps

I hope this article gave you some insights into what to expect from a systems design interview and strategies for successfully navigating them. As stated at the beginning, this article likely isn't where your preparation ends but where it has just started. I highly recommend digging deeper into the above concepts via other resources and watching a mock system design interview (or ten) to gain broader exposure to common questions and suitable approaches.

If you landed on this article while preparing for an upcoming interview, I wish you the best of luck. Please feel free to leave any questions/comments below. If you enjoyed this article, checkout my other [social media accounts](https://linktr.ee/viglucci) where I share content!



The "buffer is not defined" error is a common error that can occur when trying to use the [Buffer](https://nodejs.org/api/buffer.html) Node.js API in an environment where it is not natively supported. This article will cover how to fix this error by providing a polyfill with Webpack 5.

### Why do we need to polyfill Buffer?

When working in the context of a browser environment, Node.js APIs, such as [Buffer](https://nodejs.org/api/buffer.html), aren't available. There are sometimes comparable APIs available in the browser environment. However, the author of 3rd party libraries and code you may be using would have had to either replace the Node.js specific APIs they are using with the browser versions or provide another mechanism to provide your API to satisfy the libraries' need.

Unfortunately, library authors often do not consider browser environments, or their library may not have been intended to be used in a browser environment. In these cases, it is popular to use a build tool such as Webpack to replace the Node.js specific APIs used by the library with a version that provides a browser environment compatible API, known as [polyfilling](https://developer.mozilla.org/en-US/docs/Glossary/Polyfill).

### Webpack v5

Before Webpack v5, the polyfilling of many Node.js APIs would be performed automatically. With v5, [Webpack will no longer polyfill Node.js APIs](https://webpack.js.org/blog/2020-10-10-webpack-5-release/#automatic-nodejs-polyfills-removed) automatically.

### Solving the Problem

__Install a polyfill__

First, we will need to install or otherwise provide a browser-compatible Buffer implementation. For my needs, I have found the [feross/buffer](https://www.npmjs.com/package/buffer) package to be suitable.

The below command will install `buffer` as a `dev` dependency.

```
npm install -D buffer
```

__Configure Webpack Fallback__

The Webpack `resolve.fallback` configuration option allows us to configure Webpack to leverage the provided package as a "fallback" for APIs which are not natively available in the target environments that Webpack is building for (ie, a Browser).

```js
const webpackConfig = {
    ...
    resolve: {
        ...
        fallback: {
            buffer: require.resolve('buffer/'),
        },
    },
};
```

The `/` at the end of `buffer/` may appear strange to you, as it did to me. However, it is recommended by the [`buffer` documentation](https://www.npmjs.com/package/buffer#usage) because of how the Node.js module lookup algorithm works.

> To require this module explicitly, use require('buffer/') which tells the node.js module lookup algorithm (also used by browserify) to use the npm module named buffer instead of the node.js core module named buffer!

__Configure Webpack `ProvidePlugin`__

The Webpack `ProvidePlugin` provides a mechanism for injecting a module/value as a replacement/API for global variables that would otherwise be undefined. For instance, in Node.js you can leverage the `Buffer` class from the global namespace without any `import` or `require` statements. The' Buffer' class would be' undefined in a browser environment and without the `ProvidePlugin.`

```js
new Buffer();

Uncaught ReferenceError: Buffer is not defined
    at <anonymous>:1:1
```

We can provide a compatible implementation of the `Buffer` class via the Webpack configuration `plugins` array to overcome this.

```js
const webpackConfig = {
    ...
    plugins: [
        new webpack.ProvidePlugin({
            Buffer: ['buffer', 'Buffer'],
        }),
    ]
};
```

The provided array (`['buffer', 'Buffer']`) informs the `webpack.ProvidePlugin` to set the global value of `Buffer` to the `Buffer` export from the `buffer` package.

For example:

```js
const Buffer = require('buffer/').Buffer;
```

__All together__

Below is an example of the combined relevant parts of your Webpack configuration to polyfill the `Buffer` class using Webpack v5.

```js
const webpackConfig = {
    resolve: {
        fallback: {
            buffer: require.resolve('buffer/'),
        },
    },
    plugins: [
        new webpack.ProvidePlugin({
            Buffer: ['buffer', 'Buffer'],
        }),
    ],
};
```

## Conclusion

Now that you've configured Webpack to polyfill the `Buffer` class globally, you should be able to reference and create an instance of the `Buffer` class without the pesky `Uncaught ReferenceError: Buffer is not defined` error. Additionally, any reference to `Buffer` in libraries included in your "bundle" should now also reference the provided polyfill.

Please feel free to leave any questions/comments below, and if you enjoyed this article, checkout my other [social media accounts](https://linktr.ee/viglucci) where I share content!



Automated testing is an essential aspect of ensuring the quality of any software product. However, it can often be challenging to test systems that rely on databases or other external resources.

Luckily, [Testcontainers](https://www.testcontainers.org/) is a great project that makes setting up external resources more comfortable by providing a programmatic interface to run Docker containers in testing scenarios. In this article, we'll be looking at a community port of Testcontainers for Node.js ([testcontainers-node](https://www.npmjs.com/package/testcontainers)), and how we can run tests that rely on both a single Redis and a Redis Cluster.

The examples below will leverage [mocha](https://www.npmjs.com/package/mocha) as a test runner, [chai](https://www.npmjs.com/package/chai) as an assertion library, and [ioredis](https://github.com/luin/ioredis) as our backing Redis client.

Before you go any further, you can skip this article altogether and go straight to some [working examples on GitHub](https://github.com/viglucci/testcontainers-node-redis-example) if that's more your style.

## Testing against a single Redis

Testing against a single Redis instance is exceptionally straightforward.

First, we will define a `describe` block with a `before` hook. In our `before` hook, we will create our Docker container that will run Redis, and we will also make our Redis client instance. The `before` hook is run before any tests are invoked.

```js
const Redis = require("ioredis");
const { GenericContainer } = require("testcontainers");

describe("RedisTest", () => {
  let container;
  let redisClient;

  before(async () => {
    // "redis" is the name of the Docker imaage to download and run
    container = await new GenericContainer("redis")
      // exposes the internal Docker port to the host machine
      .withExposedPorts(6379)
      .start();

    redisClient = new Redis({
      host: container.getHost(),
      // retrieves the port on the host machine which maps
      // to the exposed port in the Docker container
      port: container.getMappedPort(6379),
    });
  });
});
```

Once we have our Docker container and Redis client created, we will define an `after` hook to clean up the resources created in the `before` hook. The `after` hook is called when all tests are done executing.

```js
const Redis = require("ioredis");
const { GenericContainer } = require("testcontainers");

describe("RedisTest", () => {
  let container;
  let redisClient;

  before(async () => { ... });

  after(async () => {
    redisClient && (await redisClient.quit());
    container && (await container.stop());
  });
});
```

Now that our Docker container and Redis client are created, we can write a test that will invoke a command against the running Redis.

```js
const Redis = require("ioredis");
const { GenericContainer } = require("testcontainers");
const { expect } = require("chai");

describe("RedisTest", () => {
  let container;
  let redisClient;

  before(async () => { ... });

  after(async () => { ... });

  it("should set and retrieve values from Redis", async () => {
    await redisClient.set("key", "val");
    expect(await redisClient.get("key")).to.equal("val");
  });
});
```

Putting it all together, we now have a test that can invoke commands against a running Redis instance and perform assertions on the results.

```js
const Redis = require("ioredis");
const { GenericContainer } = require("testcontainers");
const { expect } = require("chai");

describe("RedisTest", () => {
  let container;
  let redisClient;

  before(async () => {
    // "redis" is the name of the Docker imaage to download and run
    container = await new GenericContainer("redis")
      // exposes the internal Docker port to the host machine
      .withExposedPorts(6379)
      .start();

    redisClient = new Redis({
      host: container.getHost(),
      // retrieves the port on the host machine which maps
      // to the exposed port in the Docker container
      port: container.getMappedPort(6379),
    });
  });

  after(async () => {
    redisClient && (await redisClient.quit());
    container && (await container.stop());
  });

  it("should set and retrieve values from Redis", async () => {
    await redisClient.set("key", "val");
    expect(await redisClient.get("key")).to.equal("val");
  });
});
```

## Testing against a Redis Cluster

Testing against a Redis Cluster can be a bit more complicated than against a single Redis. This is primarily because the Docker container's local network will not map to the same IP and ports as on the host machine. Still, luckily the ioredis library supports the needed configuration to make it possible.

Using our `before` hook from the previous example, we will need to make a few small changes to run a Redis cluster rather than a single Redis in our Docker container. These changes include:

- Changing our Redis Docker image to `grokzen/redis-cluster`.
- Creating a Docker network.
- Creating a list of Redis nodes to connect to.
- Create a [NAT](https://en.wikipedia.org/wiki/Network_address_translation) map to map from the internal Docker network to the host network.

It is necessary to create a NAT map as once our backing Redis client is connected to any node in the cluster; it will attempt to auto-discover any other nodes that belong to the cluster. Unfortunately, when the client discovers new nodes, the nodes will be referenced by local addresses to the Docker container rather than the host machine. The NAT map will allow the client to properly translate the internal Docker address to an available address on the host machine.

```js
const Redis = require("ioredis");
const { GenericContainer, Network, Wait } = require("testcontainers");

describe("RedisTest", () => {
  let container;
  let network;
  let redisClient;

  before(async () => {
    // "grokzen/redis-cluster" exposes 6 Redis nodes
    // on ports 7000 - 7005
    const ports = [7000, 7001, 7002, 7003, 7004, 7005];

    // we create a new Docker network so that we have a consistent way
    // to retrieve the internal addresses of the Redis nodes to build
    // the NAT map
    network = await new Network().start();

    // "grokzen/redis-cluster" is the name of the Docker
    // image to download and run
    container = await new GenericContainer("grokzen/redis-cluster")
      // exposes each of the internal Docker ports listed
      // in `ports` to the host machine
      .withExposedPorts(...ports)
      .withNetworkMode(network.getName())
      .withWaitStrategy(Wait.forLogMessage("Ready to accept connections"))
      .start();

    const networkIpAddress = container.getIpAddress(network.getName());

    const dockerHost = container.getHost();
    const hosts = ports.map((port) => {
      // { host: "localhost", port: 55305 }
      return { host: dockerHost, port: container.getMappedPort(port) };
    });

    /**
     * {
     *   "192.168.16.2:7000": { host: "127.0.0.1", port: 55305 }",
     *   "192.168.16.2:7001": { host: "127.0.0.1", port: 55306 }",
     *   ...
     * }
     */
    const natMap = ports.reduce((map, port) => {
      const hostPort = container.getMappedPort(port);
      const internalAddress = `${networkIpAddress}:${port}`;
      map[internalAddress] = { host: dockerHost, port: hostPort };
      return map;
    }, {});

    redisClient = new Redis.Cluster(hosts, { natMap });
  });
});
```

With our Redis cluster container running and our `Redis.Cluster` client created, we now need to make sure we remember to add our newly created `network` resource to our `after` hook.

```js
const Redis = require("ioredis");
const { GenericContainer, Network, Wait } = require("testcontainers");

describe("RedisTest", () => {
  let container;
  let network;
  let redisClient;

  before(async () => { ... });

  after(async () => {
    redisClient && (await redisClient.quit());
    container && (await container.stop());
    network && (await network.stop());
  });
});
```

Everything from this point forward is the same as consuming a single Redis. You can see it all put together below.

```js
const Redis = require("ioredis");
const { GenericContainer, Network, Wait } = require("testcontainers");
const { expect } = require("chai");

describe("RedisClusterTest", () => {
  let container;
  let network;
  let redisClient;

  before(async () => {
    // "grokzen/redis-cluster" exposes 6 Redis nodes
    // on ports 7000 - 7005
    const ports = [7000, 7001, 7002, 7003, 7004, 7005];

    // we create a new Docker network so that we have a consistent way
    // to retrieve the internal addresses of the Redis nodes to build
    // the NAT map
    network = await new Network().start();

    // "grokzen/redis-cluster" is the name of the Docker
    // image to download and run
    container = await new GenericContainer("grokzen/redis-cluster")
      // exposes each of the internal Docker ports listed
      // in `ports` to the host machine
      .withExposedPorts(...ports)
      .withNetworkMode(network.getName())
      .withWaitStrategy(Wait.forLogMessage("Ready to accept connections"))
      .start();

    const networkIpAddress = container.getIpAddress(network.getName());

    const dockerHost = container.getHost();
    const hosts = ports.map((port) => {
      // { host: "localhost", port: 55305 }
      return { host: dockerHost, port: container.getMappedPort(port) };
    });

    /**
     * {
     *   "192.168.16.2:7000": { host: "127.0.0.1", port: 55305 },
     *   "192.168.16.2:7001": { host: "127.0.0.1", port: 55306 },
     *   ...
     * }
     */
    const natMap = ports.reduce((map, port) => {
      const hostPort = container.getMappedPort(port);
      const internalAddress = `${networkIpAddress}:${port}`;
      map[internalAddress] = { host: dockerHost, port: hostPort };
      return map;
    }, {});

    redisClient = new Redis.Cluster(hosts, { natMap });
  });

  after(async () => {
    redisClient && (await redisClient.quit());
    container && (await container.stop());
    network && (await network.stop());
  });

  it("should set and retrieve values from the Redis cluster", async () => {
    await redisClient.set("key", "val");
    expect(await redisClient.get("key")).to.equal("val");
  });
});
```

Now that you are testing your applications with greater confidence using containers don't forget to let me know if you found this article helpful! Feel free to leave any questions/comments below, and checkout my other [social media accounts](https://linktr.ee/viglucci)!



When programming async operations, JavaScript developers are generally at least familiar with Promises, async/await, and callbacks. However, newer async models such as ReactiveX ([RxJS](https://github.com/ReactiveX/rxjs) in JavaScript), are not quite as popular. In this article, we dig into something similar but slightly different to RxJS; a Reactive Streams implementation called [RSocket Flowable](https://github.com/rsocket/rsocket-js/blob/master/docs/03-flowable-api.md) (rsocket-flowable on npm).

## Core Concepts

At the core of RSocket Flowable is the Observer Pattern, implemented in rsocket-flowable via the **Single** and **Flowable** interfaces.

> The observer pattern is a software design pattern in which an object, called the subject, maintains a list of its dependents, called observers, and notifies them automatically of any state changes, usually by calling one of their methods. https://en.wikipedia.org/wiki/Observer_pattern

### Single and Flowable Interfaces

```js
const { Single, Flowable } = require('rsocket-flowable');

// construct an instance of a Single
const mySingle$ = new Single(subscriber => {});

// constuct an instance of a Flowable
const myFlowable$ = new Flowable(subscriber => {});
```

### Single

The Single is an observable interface that supports the following interactions:

- emit a single value via the `subscriber.onComplete` callback
- emit an error value via the `subscriber.onError` callback
- cancellation via the `cancel` callback passed to observers through the `onSubscribe` callback

Apart from cancellation, these operations should feel familiar as they are mostly the same as interacting with Promises, as promises can only ever resolve or reject.

#### Single Example

A practical example of consuming the Single interface would be wrapping a promise API/operation, such as the `fetch` API. In the below case, we do just that; we create a new instance of Single, which when subscribed to will call to the Starwars API to retrieve data about Luke Skywalker.

```js
const { Single } = require('rsocket-flowable');
const fetch = require('node-fetch');

const luke$ = new Single(subscriber => {
  fetch('https://swapi.dev/api/people/1')
    .then(response => response.json())
    .then(data => subscriber.onComplete(data))
    .catch(error => subscriber.onError(error));
  subscriber.onSubscribe();
});

luke$.subscribe({
  onComplete: data => {
    console.log(data);
  },
  onError: err => {
    console.error('There was a disturbance in the force!', err);
  },
  onSubscribe: cancel => {
    /* calling cancel() will stop calls to onComplete/onError */
  },
});
```

### Flowable

The Flowable is an observable interface that supports the following interactions:

- emit a single value via the `subscriber.onComplete` callback
- emit one or more values via the `subscriber.onNext` callback when the subscriptions **request callback** is invoked
- emit one or more error values via the `subscriber.onError` callback
- cancellation via the `cancel` callback passed to observers through the `onSubscribe` callback

Flowable differs from Single on a fundamental level in that we expect Flowable to emit one or more values. Single is only ever supposed to emit a single or no value. Additionally, Flowable supports the concept of back-pressure.

From the Reactive Manifesto:

> ... back-pressure is an important feedback mechanism that allows systems to gracefully respond to load rather than collapse under it https://www.reactivemanifesto.org/glossary#Back-Pressure

The concept of back-pressure isn't exactly unique to rsocket-flowable, but it is simpler compared to the back-pressure support provided through RxJS. In the simplest terms, Flowable's back-pressure support allows for an observer to control the rate at which an observable emits or "publishes" values. To support this, **the Flowable interface accepts a subscriber that must implement a request method**. This request method acts as a callback that is responsible for "publishing" values as requested by an observer.

### The Request Method

The request method is responsible for publishing data at the rate requested by an observer, with the observer controlling the flow of data by passing an int value representing the number of events it can handle.

```js
const { Flowable } = require('rsocket-flowable');

const ints$ = new Flowable(subscriber => {
  subscriber.onSubscribe({
    request: n => {
      for(let i = 0; i < n; i++) {
        subscriber.onNext(i);
      }
    }
  });
});

ints$.subscribe({
  onNext: i => console.log(i),
  onSubscribe: sub => sub.request(3)
});
```

In this example, calling `sub.request(3)` would result in `onNext()` being called with the values `0, 1, 2`.

For a more complex "real world" usage example, read the detailed explanation of an algorithm leveraging Flowable labeled "Flowable Code Example Explanation," or jump straight the corresponding code sample labeled "Flowable Code Example."

### Flowable Code Example Explanation

Below we've implemented a Flowable publisher that will emit data retrieved from the Starwars API for every movie that contains the character Luke Skywalker. To accomplish this, we implement the request method of the subscription object passed to `filmsSubscriber.onSubscribe()` that roughly follows the following algorithm:

When the request method is invoked for the first time:

- Fetch data about Luke Skywalker from the Starwars API and destructure the array of films from the response. We then save the collection of films to the `pendingFilms` variable so that we can reference it on subsequent calls to `request`.

When the request method is invoked for the first time, and on every subsequent call to request:

- Loop over each URL in the `pendingFilms` array to load data about a movie with Luke Skywalker as a character.
  - Break the loop if we have requested the number of movies that the observer requested (`requestedFilmsCount`).
  - Break the loop if data for all of the movies is loaded.
- Remove a URL to a movie from the `pendingFilms` list.
- Fetch the data about the movie removed from the `pendingFilms` list, and add the resulting promise to the unsettled promises array (`fetches`).
  - Once the promise resolves, pass the resulting data to `filmsSubscriber.onNext(filmData)`.
  - If the promise rejects, pass the resulting error to `filmsSubscriber.onError(err)`.
- Once all the promises saved to the unsettled promises array (`fetches`) have settled, check if we still have movies we haven't loaded data for yet.
  - If there are movies that still haven't loaded data for yet, do nothing and wait for the observer to perform a subsequent call to `request` on its subscription.
  - If there are no more movies waiting to be loaded, call `filmsSubscriber.onComplete()`, which will signify to the observer that all possible data is loaded.

This algorithm is substantially more complex than the more straightforward case of leveraging `Single` to forward the result of a Promise.  However, the support for controlling the rate at which we pull additional data, along with the backing for cancellation (with small adjustments), makes the added complexity of Flowable a worthwhile tradeoff.

### Flowable Code Example

```js
const { Flowable } = require('rsocket-flowable');
const Promise = require('bluebird');
const fetch = require('node-fetch');

const films$ = new Flowable(subscriber => {

  let pendingFilms = null;

  subscriber.onSubscribe({
    request: async requestedFilmsCount => {
      if (!pendingFilms) {
        const response = await fetch('https://swapi.dev/api/people/1');
        const { films } = await response.json();
        pendingFilms = films;
      }

      const fetches = [];
      while (requestedFilmsCount-- && pendingFilms.length) {
        const nextFilm = pendingFilms.splice(0, 1)[0];
        const promise = fetch(nextFilm)
          .then(response => response.json())
          .then(filmData => subscriber.onNext(filmData))
          .catch(err => subscriber.onError(err));
        fetches.push(promise);
      }

      await Promise.allSettled(fetches);

      if (!pendingFilms.length) {
        subscriber.onComplete();
      }
    }
  });
});

films$.subscribe({
  onComplete: () => console.log('All films fetched!'),
  onError: err => console.error(err),
  onNext: film => console.log(film.title),
  onSubscribe: sub => sub.request(100)
});
```

### Lazy Observables

The observable interfaces implemented by rsocket-flowable are "lazy," meaning that no "work" begins until an observer subscribes to the observable. These observables can also be referred to as "cold observables," which is in contrast to a "hot observable." When working with a hot observable, the observable may emit values regardless of the presence of any observers.

```js
const mySingle$ = new Single(subscriber => {
  // closure is not invoked until mySingle$.subscribe() is invoked.
});

const myFlowable$ = new Flowable(subscriber => {
  // closure is not invoked until myFlowable$.subscribe() is invoked.
});
```

In contrast, you may already be familiar with the concept of "eager" or "hot" interfaces in the form of Promises. For promises, the callback passed to the Promise constructor is invoked as soon as the Promise instance is created (or on the next tick of the event loop if you want to get specific).

If a tree falling in the woods was a hot observable, it would make sound regardless of if anyone was around to hear it or not.

```js
new Promise((resolve, reject) => {
  const log = (msg) => console.log(`[${new Date().toISOString()}] ${msg}`);
  log('Callback called');
  setTimeout(() => {
    const randomInt = Math.floor(Math.random() * Math.floor(10));
    log(`The random int is ${randomInt}`);
    resolve(randomInt);
  }, 1000);
});
```

In the above example, the `setTimeout` method  in the callback passed to the Promise constructor is invoked whether the `.then()` prototype method is invoked or not. You can verify this for yourself by copying the above example into your browser's dev tools console, where you will see that a console logline printed immediately, followed by a random int value about one second later.

### Cancellation

Cancellation is a powerful feature of observable interfaces, such as rsocket-flowable. Cancellation allows for an observer to indicate to the observable that they are no longer interested in the result of any operations which are ongoing. Cancellation is useful when programming user interfaces with frameworks such as ReactJS, where being able to cancel in-flight async operations is essential for cleaning up state to avoid side effects when unmounting components. Cancellation support is also useful when implementing stateful web services with protocols such as WebSockets, where a client could terminate their connection at any time, and continuing to perform operations on their behalf after they've disconnected likely doesn't make sense.

### Cancellation Example

In the example below, we create an instance of a Flowable that will emit an integer value until canceled, with a subscriber requesting a random number of ints every 500 milliseconds (half a second). The subscriber will additionally cancel the stream of ints after three seconds. This example is similar to how you could implement a timeout for an async operation, such as a network request or file read.

```js
const { Flowable } = require('rsocket-flowable');

let lastInt = 0;
const ints$ = new Flowable(subscriber => {
  subscriber.onSubscribe({
    cancel: () => console.log('Stream canceled!'),
    request: (n) => {
      const upperBounds = lastInt + n;
      for(let i = lastInt; i < upperBounds; i++) {
        lastInt = i;
        subscriber.onNext(i);
      }
    }
  });
});

ints$.subscribe({
  onComplete: () => { /* onComplete is never called */ },
  onNext: i => console.log(i),
  onSubscribe: sub => {
    /**
     * Request a random number of ints every 500 milliseconds.
     */
    const interval = setInterval(() => {
      const n = Math.floor(Math.random() * Math.floor(10)) + 1;
      sub.request(n);
    }, 500);

    /**
     * Cancel the stream after 3 seconds.
     */
    setTimeout(() => {
      clearInterval(interval);
      sub.cancel();
    }, 3000);
  }
});
```

### Cancellation Caveat

It is essential to understand that canceling an observable stream only instructs the observable that the subscriber no longer cares to receive updates, it does not automatically cancel any operations which the publisher may have been performing. If it is vital for your observable to react to being canceled, then you can implement the `subscription.cancel` callback to perform cleanup as needed.

Lastly, with `rsocket-flowable@0.0.14`, to avoid the below TypeError, you must implement the cancel callback on the publisher if you intend to invoke cancel from a subscriber.

```
TypeError: this._subscription.cancel is not a function
```

### The Future of RSocket Flowable

In a previous article [The State of RSocket in JavaScript](/the-state-of-rsocket-in-javascript), we reviewed how the future of rsocket-js may be uncertain, and with a maintainer of rsocket-js stating that RSocket Flowable may be replaced in the future by a [new project](https://github.com/rsocket/rsocket-js/issues/45#issuecomment-522035252), I feel the same way about RSocket Flowable. However, the GitHub repo for the proposed replacement has not received meaningful contributions for over 16 months, so it is hard to say if this will occur. It is, of course, possible that the project could be in-development behind the scenes as a closed source project to be released as OSS in the future.

### Final Thoughts

Advancements such as RSocket support the continued adoption of observable patterns. RSocket models a stream of messages moving fluently through a system, with each leveraging a ReactiveX, or a React Streams implementation. Reactive Streams, and ReactiveX (implemented as RxJS in JavaScript), have grown on me after discovering RSocket, and I believe we will continue to see growth and adoption of these patterns over the next couple of years. In the Java ecosystem, it is already clear that Reactive Streams has become more prevalent through the popular Project Reactor.

I recommend diving deeper into these concepts if building highly scalable reactive microservices and real-time interfaces intrigues you.



RSocket is an exciting new advancement in the application protocol space with decent adoption and support in the Java ecosystem, mainly due to contributions from companies such as [Pivotal](https://pivotal.io/), and leading support in the popular [Spring](https://spring.io) framework. With respect to JavaScript, RSocket hasn't quite picked up the same level of popularity or achieved as other protocols, such as [GraphQL](https://graphql.org/).

## Stiff Competition

Let's compare RSocket's search popularity with GraphQL's to help gauge popularity of each protocol.

### RSocket

In the below graph, we see that RSocket's search popularity only started picking up steam in late 2018.

<img src="/images/articles/2020-05-23-the-state-of-rsocket-in-javascript/rsocket-search-trends.png"
    alt="RSocket Search Popularity"
    width="1440"
    height="490" />

Source: [Google Trends](https://trends.google.com/trends/explore?date=2015-01-01%202020-03-15&q=rsocket)

### GraphQL

In contrast, GraphQL has experienced a much steeper increase in search popularity, starting in 2016.

<img src="/images/articles/2020-05-23-the-state-of-rsocket-in-javascript/graphql-search-trends.png"
    alt="GraphQL Search Popularity"
    width="1440"
    height="490" />

Source: [Google Trends](https://trends.google.com/trends/explore?date=2015-01-01%202020-03-15&q=graphql)

### Drawing Parallels

If we draw a parallel between search popularity and industry adoption, RSocket has yet to experience the same search popularity growth as GraphQL, signifying that it hasn't adopted the same level of adoption.

<img src="/images/articles/2020-05-23-the-state-of-rsocket-in-javascript/rsocket-search-trends-vs-graphql.png"
    alt="RSocket vs GraphQL search popularity"
    width="1440"
    height="490" />

GraphQL in red. RSocket in blue.
<br/>
Source: [Google Trends](https://trends.google.com/trends/explore?date=2015-01-01%202020-03-15&q=rsocket,graphql)

### Popularity and Adoption

One can easily contribute the popularity of GraphQL to its conception at Facebook and Facebook's opensource contributions having high visibility in the JavaScript space. Additionally, the investment into the GraphQL ecosystem from startups such as [Apollo GraphQL](https://www.apollographql.com/) have paid dividends into popularizing GraphQL. If RSocket were to gain popularity with large enterprise companies, as well as an engaged audience of JavaScript developers, I believe it could help RSocket experience similar adoption.

### RSocket's Ecosystem is Undeveloped in JavaScript

In JavaScript, most developers have become accustomed to leveraging open source packages to solve a wide range of common problems, and it's never been a better time to avoid reinventing the wheel. However, being that there is only a single reference implementation of RSocket and no well-established supporting libraries, if you expect to be able to `npm install` your way to victory, you're going to have a bad time with RSocket.

<img src="/images/articles/2020-05-23-the-state-of-rsocket-in-javascript/npm-rsocket-bad-time-meme.jpg"
    alt="Southpark super cool ski instructor meme"
    width="608"
    height="411" />

## Support for RSocket in JavaScript

There exists enough library support to build *something* with RSocket & JavaScript, however, when it comes to running an application in production, and you start considering monitoring, telemetry, and other needs, your mileage may vary.

### rsocket-js

[rsocket-js](https://github.com/rsocket/rsocket-js) is the primary reference implementation of RSocket for JavaScript. The library provides the low level base implementations for several [transports](https://github.com/rsocket/rsocket/blob/master/Protocol.md#transport-protocol), as well as a [Reactive Streams](https://github.com/rsocket/rsocket/blob/master/Protocol.md#flow-control-reactive-streams) implemention.

When building an application, developers will find it is much more practical to consume an abstraction library, such as [rsokcet-rpc-js](https://github.com/rsocket/rsocket-rpc-js), rather than consume rsocket-js directly. However, library authors would likely spend a majority of their time interfacing with rsocket-js directly.

### Project Ownership

At first glance, it can be difficult to determine the primary maintainers of the rsocket-js project. The contributions as of late have primarily been from engineers who do not appear to be associated with Facebook, however, several of the licenses in the repository reference Facebook, and new contributors to the project are expected to accept the Facebook OSS agreement. Based on this, it appears that Facebook may not be taking a leading role in the libraries continued development, even if they did bootstrap and get the library off the ground initially. It is also important to note that the project has received varying levels of contribution from engineers at companies such as Pivotal and [Netifi](https://www.netifi.com/), as well as contributors from the open source community.

### Uncertainty

The Java implementation of RSocket for Java ([rsocket-java](https://github.com/rsocket/rsocket-java)) has received heavy and continued contribution from engineers at established well known companies, such as Netflix and Pivotal. This investment by leaders in the Java OSS space provides consumers of rsocket-java with a certain level of confidence that the project has a long life ahead of it, and as such, could feel more comfortable betting their project, and possibly their business of the technology. In contrast, it is uncertain if rsocket-js has a clear set of core maintainers, a defined road map, or is being used by any large companies.

### rsocket-rpc-js

rsocket-rpc-js is an abstraction library which consumes rsocket-js, and in practical terms, is likely the interface that most applications or developers would consume when working with RSocket in JavaScript.

> [rsocket-rpc-js](https://github.com/rsocket/rsocket-rpc-js) - "The Standard RPC implementation for RSocket" in JavaScript.

In addition to exposing the reactive streams implementation behind service interfaces that will be familiar to traditional OOP programmers, the library also ties in nicely with RSocket message brokers that facilitate message routing, which can be core to any distributed messaging system.

### Opinionated

RPC style client and server implementations are generally opinionated about the interfaces they expose and how messages are framed/formatted.

In contrast to a restful API where a consumer requests a User resource by invoking an HTTP GET request to `/users/1`, where `1` is the user ID, in an RPC API it would more common to invoke a HTTP POST request to `/UserService/GetUserById` and include in the request body the ID of the user `{ "userId": 1 }`. rsocket-rpc-js encourages these types of RPC patterns and interfaces. A method following this pattern could have a signature similar to the following: `getUserById(Number: id): Single<User>`.

### Tooling

**Opinionated APIs and specifications empowers tooling** to be built which further standardizes integrations without requiring developers to work around new ideas that present their own unique challenges. For instance, because rsocket-rpc-js and other RSocket RPC implementations follow an established spec, it is possible to generate source code for clients, servers, and message payloads in a number of languages by leveraging common tools and interchange formats such as protoc and protobuf.

An example of consuming a generated client service produced by protoc when combined with the [rsocket-rpc-protobuf protoc plugin](https://github.com/rsocket/rsocket-rpc-js/tree/master/rsocket-rpc-protobuf) would be similar to the below:

```js
const request = new HelloRequest();
request.setName('John Doe');
helloServiceClient.sayHello(request).subscribe({
    onComplete: (response) => {
        console.log(`HelloService.sayHello response received with message: ${response.getMessage()}`);
    },
    onError: (error) => {
        console.log(`HelloService.sayHello response received with error: ${error.name}`);
        console.error(error);
    }
});
```

In the above example a service has been implemented that exposes a method that utilizes the "Request Response" flow (review the list of flows on the [rsocket.io website](https://rsocket.io/docs/Protocol#stream-sequences-and-lifetimes)), and this `helloServiceClient` implementation can be generated by protoc when given a protobuf definition such as:

```protobuf
syntax = "proto3";

service HelloService {
    rpc SayHello (HelloRequest) returns (HelloResponse) {}
}

message HelloRequest {
    string name = 1;
}

message HelloResponse {
    string message = 1;
}
```

### Platforms

In addition to promoting the creation of tooling, a standardized opinionated specification also supports the creation of platform services and solutions, such as [netifi](https://www.netifi.com/), which is a language agnostic broker for RSocket that aims to solve many of the common service mesh orchestration challenges.

### rsocket-flowable

[rsocket-flowable](https://github.com/rsocket/rsocket-js/blob/master/docs/03-flowable-api.md) provides an implementation for Reactive Streams in JavaScript, and is deeply ingrained into both rsocket-js and rsocket-rpc-js. Learning the rsocket-flowable API, and what it means to work with Reactive libraries and paradigms is going to be paramount to successfully working with rsocket-js and rsocket-rpc-js.

If you are familiar with [RxJS](https://github.com/ReactiveX/rxjs), then rsocket-flowable will likely feel familiar to you. If you are not familiar with RxJS, or reactive programming paradigms, then you could experience a somewhat steep learning curve.

## What to Expect Next

While the bulk of ongoing development appears to be centered around the Java and C++ implementations of RSocket, it is my hope that the JavaScript implementation will continue to receive updates and additions to meet the full RSocket spec, but only time will tell.

If you are passionate about RSocket and a seasoned JavaScript developer, now may be a good time to get involved with the project to leave your mark and assist in its continuity. If contributing to the project is not your cup of tea, then sharing this article and RSocket with your friends, coworkers, and on your social networks would be a great way to increase RSocket's visibility in the JavaScript community.

Drop me a comment below, or tweet me on Twitter and let me know if this is the first you are hearing of RSocket, or if you are also closely watching its development, as I am.



Real-time reactive systems are some of my favorite types of systems to build; however, they can be daunting tasks to approach, especially when learning a new programming language or ecosystem along the way. This notion was recently reinforced for me as I set out to build a "simple" real-time client/server application utilizing Javascript/React & Java/Spring,  where I quickly learned of several new messaging protocols (Stomp & RSocket).

Apart from serving as an exercise in [Cunningham's Law](https://meta.wikimedia.org/wiki/Cunningham%27s_Law), this article aims to expedite the start of your real-time distributed system research and save you some of the time investment that I have already spent scouring the internet, and [screaming into the void on Twitter](https://twitter.com/vigs072/status/1169987607008296960).

**What should you expect from this article?**

This article serves as a high-level overview of topics related to building real-time/distributed systems using several common patterns, ultimately providing a high-level overview of the RSocket protocol. The goal of this article is not to provide fully working complete code samples or in-depth deep dive into any one topic. Instead, this article aims to guide the reader with understanding the common terminology and concepts so that they can more easily gather a more in-depth understanding as they continue their research elsewhere.

## What is a distributed system?

Distributed systems possess a core characteristic in that a network boundary separates the individual system components. Additionally, in a real-time distributed system, each component reacts to changes as they happen across the system topology, automatically, without existing a requirement for components of the system to keep tabs on or regularly check with other parts of the systems to see if something has changed. Implementing such a system using primarily stateless communication protocols such as HTTP is generally considered bad practice, as it quickly leads to implementations that rely on wasteful polling techniques. However, there exist numerous patterns to overcome the wasteful characteristics of HTTP polling.

Some common patterns you may see today include Webhooks, Messaging systems (Buses, Brokers, & Queues), and duplex data streams implemented using a persistent connection protocol such as TCP, WebSockets, and less commonly Aeron.

### Webhooks

The implementation of webhooks may be one of the easiest to understand, as well as to implement (as a consumer), as it is no more complicated than adding an HTTP route handler to your application (the consumer) and accepting HTTP requests from event publishers.

At the core of this pattern is the agreement between publishers and consumers that when a specific event occurs, a publisher agrees to execute an HTTP request against an endpoint implemented on the consumer application, and potentially include with each request some data.

Once the consumer has received the request from the publisher, the consumer may now take some follow-up action. This follow-up action could include modifying records in a database, executing an HTTP request back to the publisher to consume some additional data or numerous other actions. The sky is the limit. This relationship is, however, known to become problematic in some scenarios. For instance,  in systems where publishers publish large volumes of messages, or where the throughput of a publisher is higher than that of its consumers, requests to consumers commonly "back up," thus causing strain on both the consumer and the publisher as each struggles to free resources.

![webhooks UML diagram](/images/articles/2019-12-08-reviewing-distributed-system-architectures/webhooks-uml.png)

### Message Buses, Brokers & Queues

Message buses, brokers, and queues are a step up from webhooks in that they avoid some of the scalability issues that can be common when implementing webhooks. With webhooks, there exists a requirement for event publishers to keep track of each consumer who is interested in any given event and to individually notify each consumer when an event occurs. At scale, this characteristic can quickly introduce performance issues as the number of events a publisher publishes and the number of consumers interested in those events increases organically over time.

To overcome these challenges, a Message Bus, Message Broker, or Message Queue implement similar yet different patterns where they interface with an intermediary component that acts as a "go-between." This go-between is responsible for ingesting (receiving) events from event publishers and re-publishing those events to each concerned consumer/subscriber, or for receiving requests from consumers asking for any new messages. However, the latter strategy (polling by consumers) is less common and more resource exhaustive. As such, consumers more often are "pushed" messages by the broker as fast as the broker can publish them. This relationship between broker and consumer places the same burden on the consumer as with webhooks, in that they must consume messages at the rate which they are published, or risk the system re-queueing or even dropping messages.

To accomplish the re-publishing of events and overcome the above issues at scale, systems often involve a high-performance component that supports a persistent connection between publisher and go-between, such as [Redis](https://redis.io/), [Apache Kafka](https://kafka.apache.org/), [AMQP](https://www.amqp.org/)/[Rabbit MQ](https://www.rabbitmq.com/), or a cloud offering such as [AWS SQS](https://aws.amazon.com/sqs/). By employing this go-between, each cog in the wheel (publisher, go-between, and consumer) can all scale independently, as well as reap other benefits, such as avoiding the overhead of establishing a TCP connection for each published message.

Additional to the benefit of independent scaling of system components, and avoiding the overhead of ephemeral TCP connections, the usage of a Message Bus allows event publishers to be ignorant of the number of, and even who the consumers of events are, and to only concern themselves with communicating with the go-between. Furthermore, consumers/subscribers benefit from this pattern by being ignorant of which applications are publishing events and only need concern themselves with communicating with the bus/broker/queue.

![message bus UML diagram](/images/articles/2019-12-08-reviewing-distributed-system-architectures/message-bus-uml.png)

### WebSockets &  TCP

If you have made it this far into the article, you hopefully recall that in the previous section, we noted that maintaining a persistent connection between a go-between is a common strategy to avoid ephemeral TCP connection overhead. This characteristic of a well designed distributed system also holds for directly establishing persistent connections between publishers and consumers using network protocols such as TCP (for clients/languages which support it directly) or, where TCP is not directly supported, WebSockets.

However, establishing a direct persistent connection between client/server (either of which can be a consumer or a publisher in this scenario) re-introduces some of the complications overcome by leveraging a Message Bus as a go-between. Most namely, the requirement of the party establishing the connection to have prior knowledge of other parties' domain names or direct IP addresses, as well as added complexity to scale the overall system as more clients and servers are added to the topology. Regardless, there are many added benefits to leveraging WebSockets or raw TCP as the communication protocol between your system components, with one of the most significant being that data can "flow both ways," which is unique when compared to Webhooks and Message Buses.

![websocket UML diagram](/images/articles/2019-12-08-reviewing-distributed-system-architectures/websocket-uml.png)

**Duplex Communication**

A characteristic of Webhooks and Message Bus system implementations is generally that data only flows in one direction, from publisher to consumer, which is contrary to a Duplex Communication system that leverages raw TCP or WebSockets, where data flows between components in both directions seamlessly. This bi-directional transfer of data between two connected components over a network boundary, known as "Duplex Communication," allows for both sides of a connection to react to changes and events occurring in the other with minimal overhead, but with added implementation complexity surrounding the format (protocol) of the messages.

**Message Format**

Similar to how natural spoken languages use a defined list of rules and expectations that both sides of a conversation agree to, so do all network communications. This message format commonly referred to as the "protocol," dictates to all involved parties how to construct a message, which messages to expect, and possibly in what order.

A simple implementation of a messaging protocol might define the following parameters and constraints:
- Every 1000 milliseconds, each participant will transmit to the other a "keep-alive" message; otherwise, either participant is free to terminate the connection as they see fit.
- Each message in the exchange must consist solely of UTF8 encoded strings serializable to valid JSON.
- Each message is to include an `id` and a `data` property.

An example message in this protocol may look something like the below:

```JSON
{
  "id": "user_created",
  "data": {
    "user": {
      "id": 1,
      "username": "user_1"
    }
  }
}
```

Given such a protocol, a participant in communications that leverage it would have everything needed to handle incoming messages in an expected and understood manner. In a real-world application, this could manifest as the application invoking the `userCreated` method on a `UserEventHandler` class, in which the application has implemented to invoke anytime it receives a message with an `id` of `user_created`.

For the most simple of systems and even some moderately active/complex systems, such a protocol would contain sufficient features to allow for the successful development of the system, with examples of relatively simple (yet still sufficiently complex and featured) protocols such as this including [Stomp](https://stomp.github.io/), and [XMPP](https://xmpp.org/). Extensive usage of protocols similar to these exists in today's modern applications. Yet, they do not offer mechanisms for overcoming the common pitfall of overloaded consumers; some more recent development in protocols, however, do consider this, such as [RSocket](http://rsocket.io/), through the notion of [back-pressure](https://www.reactivemanifesto.org/glossary#Back-Pressure).

### The highest of RSocket overviews

RSocket is a relatively new addition to the distributed systems landscape, with search term popularity only just starting to increase in the United States since the latter half of 2018 (according to [Google Trends](https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=RSocket)). Yet, it offers some exciting concepts which I am personally excited to dig into more.

**Back-pressure**

RSocket supports the concept of [back-pressure](https://www.reactivemanifesto.org/glossary#Back-Pressure), which at its core means that any given component of a system is empowered to control the rate at which messages transmissions occur between itself and other components of the system. This concept allows for a client to communicate to a server that it is prepared to handle N number of messages and that the server should send it no more than N messages without the client requesting additional messages.

**Message types**

The RSocket protocol supports multiple message types, each of which satisfies common request/response patterns utilized in various numbers of systems.

__Fire and Forget__ - The client sends a message to the server and does not expect the server to send a response. The client sends the message immediately.

__Request Response__ - The client is to send a message to the server and expected the server to respond with precisely one response message. The client does not send the request message until it has also prepared to handle the response.

__Request Stream__ - The client sent a request to the server and expected the server to respond with N number of responses. The client does not send the request until it is prepared to handle the responses, and the server does not send the responses until the client has executed subsequent request(s) for N responses.

__Request Channel__ - The client and server each prepare themselves to transmit and to receive N number of messages from the other. Neither client nor server transmits messages to the other until the other has signaled they are prepared to handle messages by first requesting N number of messages from the other.

**Multiple Language Implementations**

Although RSocket is a relatively new technology, the protocol can be implemented in any language or runtime, which can utilize either TCP directly or WebSockets. This characteristic of the protocol has allowed for client/server implementations in Java, Javascript, Go, .NET, C++, and Kotlin. The full list of language implementations and the full set of protocol features can be reviewed on the official [RSocket website](http://rsocket.io/).

### Webhooks, Message Buses, and WebSockets, oh my!

We covered much ground reviewing Webhooks, Message Buses, WebSockets, and finally, an overview of RSocket. With any luck, this article has assisted you with learning something new and assisted you with deciding which architecture(s) best fit your application's use cases and requirements.

If you found this article helpful, let me know in the comments below, drop me a note on Twitter (link in the bio below), or if you would be so kind, share it on your favorite social media network. If you found this article to be widely inaccurate, or take other issue with any of the content, feel free to send me your favorite disdainful meme on Twitter. Might I suggest a [current trend](https://knowyourmeme.com/memes/woman-yelling-at-a-cat)?



*Note: this article references laravel 5.1, or later. Behavior may differ depending on your application's framework version.*

As I've been working on updating an older Laravel application from the dark ages of 4.2 to the newer 5.x (with the help of [Shift](https://laravelshift.com/)), the lack of unit tests in the application has drastically decreased confidence in the migration(s) and slowed me down tremendously. So with that, I've been making an effort to increase the unit test coverage of the application, along with making refactors to make the application more testable, and thus hopefully more reliable. Though, it hasn't been smoothing sailing all of the ways.

One of the issues I recently ran into was how to deal with captcha validation when working with Laravels built in [Application Testing](https://laravel.com/docs/5.1/testing#application-testing) to write tests for the applications account registration flow.

Most applications that offer an account registration flow employ some type of bot prevention techniques such as the use of a captcha, Google's [reCAPTCHA](https://www.google.com/recaptcha/intro/v3beta.html) being a popular solution. However, for this particular application, we are currently using the [mewebstudio/captcha](https://github.com/mewebstudio/captcha) package.

I was initially stumped on how I could either simulate the user interacting with the captcha field, or disable the captcha functionality during testing, but luckily after looking into [Mockery](https://github.com/mockery/mockery), I was able to come up up with a few solutions that seemed reasonable for bypassing captcha validation in testing scenarios.

## Bind a test double in createApplication

The method I eventually went with was to simply bind a test double to the applications container during tests which would always pass validation.

```php
class TestCase extends Illuminate\Foundation\Testing\TestCase
{
    /**
     * Creates the application.
     *
     * @return \Illuminate\Foundation\Application
     */
    public function createApplication()
    {
        $app = require __DIR__.'/../bootstrap/app.php';

        $app->make('Illuminate\Contracts\Console\Kernel')->bootstrap();

        /**
         * Override captcha binding with a double that always passes.
         */
        $app->bind('captcha', function($app)
        {
            $mockCaptcha = Mockery::mock(Mews\Captcha\Captcha::class);

            $mockCaptcha
                ->allows()
                ->img()
                ->andReturn('');

            $mockCaptcha
                ->allows()
                ->check('12345')
                ->andReturn(true);

            return $mockCaptcha;
        });

        return $app;
    }
}
```

Based on the internal workings of the captcha library we are using here, this requires that the double we bind to the container expect that the `img()` method and the `check()` method will be invoked, and for the `check()` method to return true when provided with some expected input.

Once we have our double registered with the applications container, we can now write a test that visits our applications registration page, fills in the form inputs, and then submits the form request.

```php
use Illuminate\Foundation\Testing\DatabaseMigrations;
use Illuminate\Foundation\Testing\DatabaseTransactions;

class UserRegistrationTest extends TestCase
{

    use DatabaseMigrations;
    use DatabaseTransactions;

    /**
     * @test
     * @return void
     */
    public function redirected_to_login_page_after_good_registration()
    {
        $this->visit('/register')
            ->type('TestUserName', 'username')
            ->type('testemail@localhost.com', 'email')
            ->type('testemail@localhost.com', 'email_confirmation')
            ->type('testpassword', 'password')
            ->type('testpassword', 'password_confirmation')
            ->type('12345', 'captcha')
            ->press('Submit')
            ->seePageIs('/login')
            ->see('Thank you for registering.');
    }
}
```

## Overriding the captcha validator

As an alternative to registering a test double with the applications container, we could simply override the applications captcha validation rule to always return true, thus passing validation.

```php
class TestCase extends Illuminate\Foundation\Testing\TestCase
{
    /**
     * Creates the application.
     *
     * @return \Illuminate\Foundation\Application
     */
    public function createApplication()
    {
        $app = require __DIR__.'/../bootstrap/app.php';

        $app->make('Illuminate\Contracts\Console\Kernel')->bootstrap();

        /**
         * Override the validation captcha extension to always return true.
         */
        $app['validator']->extend('captcha', function()
        {
            fwrite(STDOUT, print_r('Bypassing captcha validation and returning explicit true!', TRUE));
            return true;
        });

        return $app;
    }
}
```

Although this method works just as well as the previous, I believe that I prefer the original method over the later. The reason is that although we are not making use of any of Mockery's API to assert on the state of the test double during my tests, we lose the ability to inspect what happened to the object during the test, should you wish to do so.

## Closing thoughts

Neither of these solutions are entirely clean as they both require essentially disabling functionality which would ideally be confirmed to be rock solid, and a crucial part of your applications feature set, however, if we agree that would rather have these tests with some imperfections, rather than no tests for the user registration flow at all, they will suffice for now.

Do you have a better way to mock/bypass captcha during testing scenarios? Let me know in the comments below!



For Node.js developers, there aren't many options if you are looking for a full featured MVC framework, but there is one that i've been expirementing with lately, [AdonisJs](https://adonisjs.com/).

AdonisJs offers a wide range of functionality and a whole family/suite of official and complimenting packages, such as its command line tool [Ace](https://adonisjs.com/docs/4.1/ace), and its test framework [Japa](https://github.com/thetutlage/japa).

With its growing ecosystem of official and user written packages, AdonisJS is looking to be one of the most fully featured, but opinionated frameworks available to Node.js developers today. Along with AdonisJS's being a full feature framework, the framework should feel very familiar to developers coming from PHP and Laravel, as AdonisJS appears to be heavily inspired by Laravel.

However, for Node.js developers who have been working in more minimal frameworks such as Express or Koa, myself included, I have found that there can be some friction with breaking away from our preconcived notions for how the framework should work, or how our existing tools and processes should work with the framework.

The most recent pain point I experienced with this has been trying to use vscode to debug my application and its test suite (written using Japa), as I have historically used [Mocha](https://mochajs.org/), which has great support in rich development environments and IDEs such as Intellij Idea and Microsoft Virtual Studio Code.

So what needs to be done to get debugging of your AdonisJS application working with vscode? Not a whole lot actually, its just more than you would have to do if you were using Mocha.

## Debugging your application

I have found that debugging a AdonisJS application with vscode to be relatively easy to setup, and a much better development experience compared to examining log output.

### 1) Register a dev script with npm

In your projects package.json, add a new entry to the scripts object with the value of `adonis serve --dev --debug`. This script can be named however you'd like, but I typically name my development script `dev`.

```json
{
  "scripts": {
    "dev": "adonis serve --dev --debug",
  }
}
```

The above script when executed with `npm run dev` will invoke adonis and by passing the `--dev` and `--debug` flags, instruct Ace to begin watching files for changes, as well as start the node debugger on the default port.

### 2) Create a new launch configuration

Now that you have your npm script registered, you will need to create a new [launch configuration](https://code.visualstudio.com/docs/editor/debugging#_launch-configurations) in vscode that will allow you to invoke your `npm run dev` script, as well as attach to the node inspector.

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Application",
      "runtimeExecutable": "npm",
      "runtimeArgs": [
        "run",
        "dev"
      ],
      "port": 9229,
      "internalConsoleOptions": "openOnSessionStart"
    }
  ]
}
```

You can now launch your application, set breakpoints, and more, as described in the [vscode documentation](https://code.visualstudio.com/docs/editor/debugging).

## Debugging Tests

### 1) Create a ace.js file

Every AdonisJS project comes with an `ace` file by default. The `ace` file is essentially a configuration file that the AdonisJS command line uses to bootstrap itself and execute commands in a manner that is contextual to your specific application.

A bare bones `ace` file will look something like below:

```js
'use strict'

/*
|--------------------------------------------------------------------------
| Ace Commands
|--------------------------------------------------------------------------
|
| The ace file is just a regular Javascript file but with no extension. You
| can call `node ace` followed by the command name and it just works.
|
| Also you can use `adonis` followed by the command name, since the adonis
| global proxy all the ace commands.
|
*/

const { Ignitor } = require('@adonisjs/ignitor')

new Ignitor(require('@adonisjs/fold'))
  .appRoot(__dirname)
  .fireAce()
  .catch(console.error)
```

For reasons that are unknown to me, this file is named `ace` rather than `ace.js` by default, and this unfortunately causes vscode to complain that it cannot find the file when trying to invoke it with node in a launch configuration.

![vscode error dialog](/images/articles/2018-04-14-debugging-adonis-with-vscode/vscode-ace-error-dialog.jpg)

To overcome this, we can rely on the ability for Node.js to require a file and attempt to interpret it as Javascript, regardless of its extension. This allows us to create a `ace.js` file in our project that simply requires the `ace` file.

```js
require('./ace');
```

### 2) Create a new launch configuration

Once you have your `ace.js` file created, you can go ahead want to create a new vscode [launch configuration](https://code.visualstudio.com/docs/editor/debugging#_launch-configurations) entry. If you created a launch configuration entry from the above section, you can simple add your new entry to the configrations array.


```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Application",
      "runtimeExecutable": "npm",
      "runtimeArgs": [
        "run",
        "dev"
      ],
      "port": 9229,
      "internalConsoleOptions": "openOnSessionStart"
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Tests",
      "program": "${workspaceFolder}/ace.js",
      "args": [
        "test"
      ],
      "internalConsoleOptions": "openOnSessionStart"
    }
  ]
}
```

With your `ace.js` file created, and your new Debug Tests launch configuration, you can now use the full power of the vscode debugger to step through your code and your tests in your AdonisJS applcation.

If you found this information useful, or if you've found a better way debug your AddonisJS application and tests, drop me a line over on [Twitter](https://twitter.com/vigs072), or leave a comment below.

Cheers!



![require('framework-x')](/images/articles/2018-04-02-framework-x/HDtNnOg.png)

So, I've decided to write my own HTTP 'framework', if you can even call it that.

For the time being I am going to call this project framework-x, until I think of something better... or not, IDK.

What qualifies as a framework? I'm not exactly sure, but that is what I am going to call this project until I am educated otherwise.

Either way... what I am attempting to do is write a series of tested, proven, and exercised patterns that will assist with working with the popular Node framework [Express](https://expressjs.com/).

The main goals of the project will be to implement a series of objects that will assist a consumer in managing their application in three high-level areas:

* Application lifecycle (startup, teardown, etc)
* Object lifecycle (object creation, IOC, dependency injection, etc)
* Routing (extend the express router to add additional features, route groups, etc)

The framework may expand past these concepts in the future, however, with the goal to simply extend Express and hopefully promote some 'best practices', I want to keep it pretty simple to promote the concept of migrating an existing Express powered application to the framework with minimal fuss.

If you'd like to check out the project you can dig through the source code over on [GitHub](https://github.com/viglucci/framework-x).

You can also take a look at a simple application I will be building with the framework in this [repository](https://github.com/viglucci/framework-x-example).

Check out the below screenshot for an example of how I think the API might look down the road.

![](/images/articles/2018-04-02-framework-x/hLj4wFM.png)

Thanks for reading 😊.



When I sat down this evening to write this post I really didn't think it would be special in any kind of way, or really be anything more than a skeleton of a post so I could once again see Jenkins do all of the automated things I've seen it do hundreds of times before, however, it quickly gained a significance that I never intended.

That trait being that this post marks almost exactly the two year anniversary from when I first checked in to git the first lines of code for this repository.

Anyways...

I sat down to write this because I noticed that my "Hello World" post mentions that this blog exists as an exercise for deploying to GitHub pages, but that has changed and this blog is no longer hosted on GitHub pages, and this post shall hereby serve the purpose of briefly outlining the changes it has undergone.

### Changelog

* No longer deployed to GitHub pages
* Now deployed to AWS Elastic Beanstalk
* Dockerized using Multistage Builds
* Automated builds using Jenkins
* Automated deployments using Jenkins
* Updated deployment/build scripts to use ES6 syntax
* Replaced promise usage with async/await in deployment/build scripts

### Final Thoughts

If you are reading this post, then that means that Jenkins once again has done its job of building a new Docker image after being triggered by GitHub, pushed the new Docker image to AWS ECR., created a new AWS EBS application version, and deployed that application version to an existing EBS environment, and that I am patting myself on the back for putting all of the legos together in the correct places to make it happen.

Finally, if any of the above made you scratch your head and wonder how that is done, I recommend you check out the wonderful series by [Chris Fidao](https://twitter.com/fideloper), [Shipping Docker](https://shippingdocker.com), which is where I learned most of it, and which I still refer to as a resource on Docker or AWS services.

Cheers



While I do not have much to say at this time, I wanted to deploy this blog to GitHub pages as an exercise. If you are reading this, then hopefully by this point I will have added more posts.


Thoughts on Software Development, Gamedev, and other topics.

osyrs: A Binary Patcher

Check for an Object in AWS S3 with Ansible

How to build oclif for windows with GitHub Actions

An Introduction to System Design Interviews

How to polyfill Buffer with Webpack 5

Testing Redis Clients in Node.js with Testcontainers Node

Reactive Streams in JavaScript with RSocket Flowable

The State of RSocket in JavaScript

Reviewing Distributed System Architectures

Handling captcha in Laravel unit testing

Debugging AdonisJS with vscode

Framework X - Whats in a Framework?

Two Years Later

Hello World