In this blogpost we will explore our best practices regarding requirements engineering & estimations.

Requirements engineering​

This first step is by far the most arduous, as it requires to gather and describe the essential aspects of a feature or a product. Our team of engineers has to take a deep dive into an existing codebase, or imagine how a new one could look like.

From business needs to specifications​

In this process, it can be useful to help the client formulate business needs, as the specifications (whether in design or development) should derive from them - not the other way around. In other words, what is the use case behind a feature, what problem does it solve? As opposed to: how is the feature implemented, how does the page look?

Consider these two requirements:

• On the accounts index view, the user should be able to see the timestamp of the account's last login
• The user should be able to easily identify stale accounts

The first one already locks the reader into a technical solution. It does not say what the user should make of this timestamp information.

The second one expresses a need. It makes no implication on how the system functions. This gives much more freedom to the engineer on the solution, and they can derive a technical specification from it in way that fits the model and the client. It also provides a simple way to assess, retroactively: are we solving the given problem with our solution?

Data model​

Building a data model, even as a draft, can be a great tool and visual support during the requirements engineering process. It is an excellent way to:

• identify corners of the product that could otherwise easily be missed
• discuss the implementation strategy with the team
• already define the domain and main components of the application
• pin vocabulary words (like a resource name) that everyone in the team can then use, and understand their relationship to the product

Estimations​

Estimating a project just after reading a client's proposal is extremely hard. This exercise only becomes manageable after two (or more) engineers transformed the business needs into specifications, and broke down those requirements into pieces small enough that a developer can tell “this would take an average developer XX days”.

Enter the Mindmap​

Most of the time, text formats are a very poor fit for describing technical specifications. A text document is structured from top to bottom and fails to visually transcribe how our codebases and how our web products are just good old graphs. The web is still essentially file navigation, like our code repositories.

Additionally, text can imply a hierarchy between sections, while in a graph, the order of the nodes at the same level is irrelevant. This can encourage an engineer to design features in isolation, already predicting an implementation roadmap that can be parallelized.

A mindmap is an excellent tool to visually support the task at hand, since it is well-equipped for exploration, re-structuring, moving sub-graphs into another area. Its visual framework gives clear encapsulation boundaries and shows the dependency between a parent node and its children.

The capacity to easily collapse and expand entire sections of the graph also allows to visualize the requirements at different levels of detail.

New codebase? Don't forget non-feature specs​

When estimating a new project, one should not forget to note down in the mindmap some essentials of a new codebase. For example: initial repository setup (and sometimes, running generators), linters, CI/CD pipelines, error tracking etc. And additionally on the application level: i18n, layout styles, navigation and so on.

To the numbers​

Going back to our previous example:

Business need

The user should be able to easily identify stale accounts

Specification

On the accounts index view, the user should be able to see and sort by the timestamp of the account's last login.

Estimation

Here the developer would need to consider what steps are necessary to achieve this task. We need a database migration to store the last login timestamp, we need to record this timestamp on a successful login, and we need to update the accounts index view to show this timestamp, and sort by it. These steps have most likely been done hundreds of time by an average backend developer. They are familiar and graspable. The levels of uncertainty & complexity is quite low.

The effort might be affected by the quality of the codebase, whether it is well structured and well tested, whether the developers are familiar with the codebase, whether reusable components already exist etc. The effort should also account for review cycles and QA.

And then?​

What is important here is that we are not asking the developers to commit to this number. We are asking them to make an estimation based on what is currently known for the project, and on tasks that were broken down enough that they can comprehend them fully. The variability of estimations can be alleviated by involving two or more developers, preferably with different experience levels or domain knowledge.

The mindmap allows us to rollup the days up to the central node and get a number of days. Again, if the result is 102 days, we are not signing off on that exact number, but can then tell if the project will take 2 weeks, 2 months, a year, and where complexity lies. Of course, the smaller the scope, the more precise the estimation can be. We use the same technique for clients coming up with a brand new product idea, as for clients asking for a new feature.

Imagine the following: The dev team receives the envelope with the application specs from a dead letter box in the park at night. Two months later, they deliver the code. Their client deploys their app, and everything works as expected.

The question: What was in that envelope?​

Product managers, engineers, and procurement teams have tried to deliver this perfect blueprint for a software product. The ones that did not fail have mostly succeeded by sheer luck. After decades of engineering, there is still no commonly accepted standard for software requirements that guarantees, when followed, the desired outcome. There is IEEE 29148, but that carefully avoids specifying the how: It tells you a requirement must be unambiguous and verifiable, but not how to express it so that it actually is.

We were faced with this problem while we were working on Surveyor, a tool to extract specifications from running (legacy) systems: What is a good format for the description, once you discovered its features?

Surveyor is currently in early access testing (you can take part to), and we already learned a lot during the first sessions applying it to other peoples' legacy software. A good moment to share what we have found out, and to ask for feedback. We have run analytics on our beta testers’ codebases, and here is what has worked.

The Premise​

Our clandestine team needs freedom of choice. We don't want to force them to use TINYINT if their favorite database does not support that. If an implementation detail is important in a way that does not become clear from the context, we need specify that in the tech choices, or describe the observable outcome in a spec. Otherwise, we should aim to specify independent from technology / solution. Here are our picks to do that.

Architectural overview​

The C4 model is a way to visualize software architecture at four levels of abstraction, created by Simon Brown. The "C4" refers to its four diagram types, each zooming in further than the last:

• System Context shows your system as a single box surrounded by the users and external systems it interacts with. It answers "what does this system do and who uses it" without any internal detail.
• Containers zooms into the system to show its major deployable or runnable pieces, like web apps, mobile apps, databases, APIs, and file systems, along with how they communicate. Here "container" means a separately running unit, not specifically a Docker container.
• Components breaks a single container down into its main building blocks and their responsibilities, showing how the code is organized into logical groupings within that container.
• Code is the most detailed level, but mostly unused or generated from the code.

The core idea is that you start broad and progressively zoom in, so different audiences can engage at the level of detail that suits them, much like zooming in on a map.

Our Beta testers have found these diagrams most helpful. Even engineers who have worked with their codebase for years welcomed this organised visualisation. Surveyor is generating C4 diagrams in the form of Structurizr DSL.

This solves two purposes:

1. It offers a standard for generating charts with a well known systematic
2. It can serve as an input for subsequent steps

C the fourth​

The fourth C is “Code”, but it’s not often used as code changes are frequent. Re-aligning UML diagrams each time the code changes adds friction we don’t want. Instead, we leave the diagram level and move on to describing the behavior of the system.

Describing Behavior​

In 2006, a movement called Behaviour-Driven Development (BDD) emerged. The idea was to write specifications in natural language that could be parsed and executed as tests simultaneously. This enabled product teams to create specifications that engineers could use directly as automated tests. The engineers would start by creating the bindings between feature language and test code. The implementation would begin with a failing test (red), which the actual feature code would then make pass (green).

Gherkin is a plain-text language for writing executable specifications in a Given-When-Then format, Cucumber is the framework that maps these specifications to code (step definitions) so they run as automated tests.

A feature like the following:

Feature: Monster battle

  Scenario: Battle

    Given there is a monster

    When I attack it

    Then it should die

Would map to a binding like this:

# test/features/step_definitions/monster_steps.exs
defmodule MonsterSteps do
  use Cucumber.StepDefinition
  import ExUnit.Assertions

  step "there is a monster", context do
    Map.put(context, :monster, Monster.new())
  end

  step "I attack it", context do
    Map.put(context, :monster, Monster.take_hit(context.monster))
  end

  step "it should die", context do
    refute context.monster.alive?
    context
  end
end

BDD’s peak popularity came in the early 2010s, a period when teams worldwide embraced it and equivalents appeared in every major language with implementations like Behave and pytest-bdd for Python, SpecFlow/Reqnroll for .NET, Behat for PHP, and Godog for Go.

Tooling is broad too, including IDE plugins for IntelliJ, Visual Studio, and VS Code offering syntax highlighting and step navigation, plus reporting tools and CI/CD integrations

Interest has cooled down some years after that because developers grew tired of maintaining large chunks of BDD code and bindings. Luckily, we can leave that part to the LLMs now.

The Case for BDD​

We are revisiting BDD today, because AI assisted coding has a problem: Plans and free formats like OpenSpec are not parsable. That means they can not be translated to automated tests directly. You need an agent step to build the testing, with all uncertainty agentic coding brings.

The relatively rigid syntax of Gherkin is an advantage here: It maps mechanically to test code. And it can also be parsed to produce other output formats: Manual QA instructions, OpenSpec or Jira tickets.

But for that, it’s missing the meta information that embeds a feature description into its application context.

The Assay format​

We have used the C4/structurizr DSL to express how our system interfaces with the outside world. We also used it to describe the inner composition of our application. When we use Gherkin alone, this context would be lost. But we can preserve it with syntax-aware comments like this:

# ---
# component: orderLifecycle
# container: api
# schema: schemas/order_lifecycle.ex
# workspace: architecture/workspace.tsp
# definitions:
#   a valid customer:
#     a Customer with status Active, verified email,
#     and a credit limit greater than zero
#   in stock:
#     the product has available quantity greater than
#     the requested amount
# invariants:
#   - total must not exceed customer credit limit
#   - at least one line item required
#   - cannot cancel an order that has shipped
# ---

With this, we can access the information in the structurizr DSL to get context information. This allows us to write parsers and plugins for coding agents that can not only read the features, but also access context information we gathered before. But what's purpose of the schema: definition?

Schema Definitions​

Schema work as a shared contract about the business objects of a component. Having them speeds up onboarding. Developers don't need to scan through all features to detect what fields are needed, and agents are less likely to invent fields. It's a token-efficient way to inject exactly the ground truth into context.

For specifying schemas, three obvious candidates come to mind:

• Protobuf — https://protobuf.dev — Google's schema and wire format. Compact and fast, with code generators across many languages, but its scalar types are physical commitments, so it presumes the most about storage and encoding.

  syntax = "proto3";
  
  message Post {
    string title = 1;
    int32 views = 2;
    bool published = 3;
    repeated string tags = 4;
  }
  

• JSON Schema — https://json-schema.org — A decade-plus standard for describing the shape and constraints of JSON. Stays loose where you want, needs no toolchain for consumers, and fits the premise most directly; the cost is verbosity. It can be authored directly, e.g. as YAML and be compiled to JSON:

  $schema: "https://json-schema.org/draft/2020-12/schema"
  $id: "https://assay.example/post"
  title: Post
  type: object
  required: [title]
  additionalProperties: false
  properties:
    title: { type: string }
    views: { type: integer }
    published: { type: boolean }
    tags:
      type: array
      items: { type: string }
  

• TypeSpec — https://typespec.io — Microsoft's TypeScript-like language that compiles to JSON Schema, OpenAPI, and Protobuf from one source. Compact and readable, at the price of a build step and a single-vendor toolchain.

  model Post {
    title: string;
    views?: integer;
    published?: boolean;
    tags?: string[];
  }
  

Under the premise that the implementing team picks the field types and specifics the three differ in how much they presume. Protobuf fits worst: its scalar types (int32, bytes) are physical commitments that pre-decide the storage details you meant to leave open. JSON Schema is the opposite — it describes shape and constraints without binding to any storage type, and hands the team a directly-usable artifact with no toolchain; its cost is verbosity and $ref-wired files. For our purpose, TypeSpec is the golden spot:

A compact, TypeScript-like authoring that compiles to JSON Schema (or Protobuf later, if needed), at the price of a build step and a younger, single-vendor toolchain. This is not so relevant in this context as we use it for documentation rather than compiling. It adds, however, the cross compilation benefits.

Architectural Decision​

Until this point, we avoided to describe the “how” of our application. Which database, which web framework, what datamodel. This is what Markdown Architectural Decision Records (MADR) cover. Their goal is to document not only the technical decision, but what options were considered and what the problem statement was. Because MADRs contain that background, it allows to effectively reconsider decisions once the Circumstances have changed.

They typically look like this:

## **Use PostgreSQL for primary data store**

* Status: accepted
* Date: 2026-06-29

### **Context**

We need a primary database for a new service with relational data and strong
consistency requirements.

### **Considered Options**

* PostgreSQL
* MongoDB

### **Decision**

Use **PostgreSQL** — the team has operational experience with it, and it
provides ACID transactions plus JSONB for flexible fields. MongoDB was rejected
due to weaker consistency guarantees and less team familiarity.

### **Consequences**

* Good: mature ecosystem, existing team expertise
* Bad: harder to scale horizontally than MongoDB

Non Functional Requirements​

By now we have described our application well in terms of how it is structured, what it does, and what tools it uses (or should use). But we left out the quality of service.

A non-functional requirement (NFR) specifies how well a system must operate, as opposed to what it does (functional requirements). Where a functional requirement says "users can book an appointment," an NFR constrains the qualities of that behavior — performance, scalability, availability, reliability, security, maintainability, usability, and so on.

They are often written in a simple table format:

Here's a fuller set, functional requirement paired with its non-functional counterpart:

Each row shows the same feature twice: the left column is what the system does, the right is the measurable constraint on how well it must do it — the NFR.

The End-Boss: Design​

Specifying UI is a daunting task because it happens at the end of the rendering pipeline where the amount of moving parts is highest. Most tools for web therefore use the web technologies themselves to avoid drifting.

The Image​

The simplest, (and arguably the worst) medium to specify design is the image:

Then the login page should match "fixtures/login.png"

This allows features to specify design alongside the functional feature requirements.

The code​

If the frontend code is using components and maybe even a component library, it might be worth carrying over. A connection to a design system in Figma or similar is the gold standard here, but might be violating our "letter in envelope" constraint. Our goto strategy at bitcrowd is Storybook. If the target system supports it, teams can work very effectively.

Visual Regression Testing​

If you happen to have a legacy system with a decent design, visual regression testing with Playwright or similar might be your tool of choice. It gives teams a quick feedback loop if it comes to single scenarios, and the option to recheck the whole interface. The great thing in combination with Gherkin is that you can switch screenshot processing on and off. The difficulty with visual regression tests is their potential flakiness. However, image processing models have made this a lot more reliable.

So, what is in that envelope?​

So far we have C4/structurizr diagrams for how the system is build and how it interacts, Gherkin/Assay for its behaviour, TypeSpec for schemas, MADR for the technological choices and NFR for its quality of service. We go for storybook for the visuals. Obviously, this is not the final solution for every stack. It would be hard to describe a pacemaker OS with it, or a web application with pristine UI effects. The final version of Surveyor will use plugins to cater for that, but for many of the projects bitcrowd has encountered, this gives a good start.

Fortunately, we have clients who enjoy talking to us, so we haven’t had to resort to espionage technology - yet. Let us know if you have some clandestine system you need to build in the dark, or that legacy application needs a rewrite.

Deploy markers?​

A deploy marker indicates a change in the deployed version of an application.

AppSignal tracks your deploys with deploy markers. They can be seen in the “Deploys” section of their app. They are a great way to document deployments, but also errors reported by AppSignal are then associated to a certain deployment. Additionally our team receives a Slack notification when a new version of our code was released to staging or production.

In short: deploy markers in AppSignal are valuable. And they can be even more valuable when one assigns a meaningful value to the deployʼs revision. Traditionally, such a meaningful value is the short SHA of the last commit in the release. That way, a developer knows exactly what code this release was running on.

AppSignalʼs agent sends the deploy markers via their API when it detects that the environment variable APP_REVISION has changed on the server. This is a first clue to our problem!

APP_REVISION, are you there? 🔮​

A first look at the container environment variables tells us that no, there is no such variable on the server. Hereʼs how to inspect those variables:

kubectl exec -it [pod] -n [namespace] -- env

We need to set this variable somehow, ideally programmatically, to the SHA of the last commit in the release. The release is built with Docker in our Github CI, and pushed to our container registry. The CI would be a natural place to set this environment variable.

Here, with a Github action:

- name: Get short SHA
  id: sha
  run: echo "short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
- name: Build and push Docker image
  uses: docker/build-push-action@v7
  with:
    ...
    build-args: |
       COMMIT=${{ steps.sha.outputs.short }}

The build-args section in this action functions in the same way as the docker flag: docker build --build-arg COMMIT="foo".

Then, in our Dockerfile:

ARG COMMIT
ENV APP_REVISION=$COMMIT
RUN echo "APP_REVISION=$APP_REVISION"

And there we have it, when building the image in the CI and looking at the logs, we see that APP_REVISION is properly assigned the SHA of the last commit!

APP_REVISION pongs 🏓​

A piece of knowledge to have in mind: the environment variables of the container image are “merged” into the container environment variables. So, if APP_REVISION is set in the image, we should see it listed in the container with our env command kubectl exec -it [pod] -n [namespace] -- env.

If itʼs still not set, make sure the ARG and ENV declarations in the Dockerfile are done on the last stage - for multi-stage Docker builds (in other words: after the last FROM clause).

At this stage we do have APP_REVISION set, but we still did not get our deploy marker! Whatʼs going on?

AppSignal agent config​

There is one way to debug this, and it is to look at AppSignalʼs agent log on the server. This can be done with:

kubectl exec -it [pod] -n [namespace] -- cat /tmp/appsignal.log

[2026-06-03T14:46:20 (process) #8][INFO] Starting AppSignal
[2026-06-03T14:46:20 (agent) #16][Info] Started appsignal-agent 0.36.9 (build 68764aa)
(as 0/0) with config Config {
active: true, agent_path: None, agent_standalone: false,
app_name: Some("foo"), app_path: Some("/app"), app_revision: Some("715a5cb"),
...
environment: "staging",
push_api_endpoint: Url { host: Some(Domain("push.appsignal.com")), ... },
push_api_key: "****",  working_dir_path: "/tmp/appsignal",
...
}

From this config, one can verify that the AppSignal agent has picked up the correct app_revision. We can also verify that we have the correct environment configured, and the correct push_api_key.

We found out that the push_api_key was set to our AppSignal organization API Key. Instead, it should be the API Key specific to the given application on AppSignal, for that environment.

We changed the value, rolled out a new release, and there we had it, our first deploy marker! Job done 🪇.

We wanted to contribute our share and decided to work on a performant DataPlane for Bluesky. When we started the project, we expected to work in Go, Rust or even Node. After all, these were the predominant languages in the community. Instead, we landed with Elixir. Here is why and how we came to that decision.

The component nobody knows about​

Everyone who has looked into the Bluesky infrastructure knows about the Personal Data Stores (PDS), the Relay or the AppView. When you set up your own little Bluesky from the ATProto repository, you run the same components as Bluesky, the commercial service. However, there's one notable exception: the DataPlane, a part of the AppView. It exists publicly only as a Node-and-Postgres reference implementation, while the production Bluesky network runs on a dedicated, ScyllaDB-backed, closed-source DataPlane (as documented in Pragmatic Engineer's deep-dive on Bluesky's architecture).

That gap is exactly where things get interesting. The reference implementation tells you what the DataPlane does; it doesn't tell you how to make it survive contact with real traffic.

If you want to run your own, you have to answer the scaling question yourself - and the first step is understanding the workload well enough to stop treating it as one thing.

ScyllaDB is Bluesky's operational choice, not part of the interface. The DataPlane's contract is a gRPC service that answers high-volume, low-complexity queries and returns skeletons - lists of IDs, counts, booleans - which a higher layer later hydrates into full views. What sits behind that contract is entirely up to you: the language, and the datastore. So before picking either, we spent our time on the only thing that actually constrains the choice: the shape of the load.

A tale of two workloads​

The Bluesky's historical data is enormous — terabytes. But when a user opens the app, they read a few dozen recent posts, get distracted, and wander off to a profile or a thread. They almost never scroll back further than a day or two.

A note on specifics: it's been observed that Bluesky's timeline doesn't serve much beyond the last day or two of content, and that deeper cursor positions tend to fill with very recent posts rather than true history. Treat the exact window as illustrative unless you've measured it on your own deployment - the architectural point holds regardless of the precise number.

This leads to an interesting disparity: Data that is more than a few days old is almost never accessed in the timeline. When it is accessed, it is usually through a profile or a thread. Yet the reference implementation compiles every timeline by joining those tables, which are terabyte-scale. This process is slow and becomes slower as the tables grow, and timeline requests account for most of what the DataPlane is asked to do.

So timeline generation deserves a different treatment from record and thread retrieval. The two workloads don't just differ in degree. They have opposite resource profiles, and they want different things from the runtime underneath.

The hot path: timelines served from memory​

Most social platforms use a hybrid strategy to handle timeline fan-out. A post from an account with a few thousand followers is pushed into its followers' timelines as soon as it is published. This is known as fan-out on write in the lingo.

However, a post from an account with millions of followers ("the celebrities") is not immediately pushed to all of its followers' mailboxes; instead, it is merged in when a follower actually requests their timeline: This is called fan-in on read. The former keeps write amplification bounded for ordinary accounts. The second prevents the write storm that a celebrity post would otherwise cause.

Another obvious simplification becomes apparent when you stop thinking of a social feed as an immutable archive. Take into account that users will jump around with their attention and you will realise that timelines are finite. Therefore, a user who follows ten thousand accounts will never see all of their posts anyway. There's reasonable to limit how much content is distributed to any single timeline. The obligation is to provide a good, recent, bounded timeline, not a complete one.

What this boils down to:

• Recent posts - the overwhelming majority of what timelines are made of - can live and be served from memory.
• Fan-out can be deferred: as long as posts land in followers' timelines within minutes, and the backlog of fan-out jobs doesn't outgrow available resources, nobody notices the delay.
• Older content, when it's genuinely needed, is safe to fetch from the database on demand.

The dominant request type, the timeline read, stops being I/O-bound (wait on a giant join) and becomes memory-and-compute-bound (manipulate in-memory structures quickly). That shift changes what "fast" means, and it's what reopened the language question for us.

It also argues for putting the concrete database behind an interface. The hot path barely touches it; the cold path is the only part that leans on it. Keep that boundary clean and you can swap the backing store later without touching the rest of the system.

The follower graph: in memory, but not naively​

There's a catch. Fan-out and timeline assembly both need answers about the follow graph: who follows whom, and the intersections and unions of those sets. Holding hundreds of millions of follow relationships in memory as hash maps would be ruinously wasteful.

Jaz's "GraphD" series documents this exact problem: an in-memory graph store that originally used hash maps and hash sets to track each user's followers and follows. The fix was switching to Roaring Bitmaps, a compressed bitmap structure built for large set operations. The numbers are striking: the entire Bluesky follow graph fits in roughly 6.5 GB of RAM, takes about 1.6 GB on disk, and loads in around 20 seconds.

Jaz also describes two cost modes that map onto our hot/cold split. Paging over all of a user's follows is expensive and belongs in paginated or async fan-out jobs. On-demand set intersection — "which people I follow also follow this person" — has to run at interactive speed. Our split isn't an invention; the access patterns already worked this way in Bluesky's own tooling.

So what do we actually need from a language?​

With the workload pinned down, the requirements are:

1. Fast, compute-bound set operations over a large, long-lived, in-memory graph: bitmap intersections and unions. Per-core compute, byte-level work, cache sensitivity.
2. High-concurrency serving of memory-resident timeline reads, with bounded response sizes and tight tail-latency expectations.
3. A deferrable fan-out queue that absorbs bursts, applies backpressure, delivers within minutes, and degrades gracefully instead of falling over.
4. A clean datastore boundary for the cold path: records, threads, profiles. Ordinary I/O-bound request handling.

The first three are the hot path; the fourth is the cold path. No language obviously wins all four. This is the scorecard we judged each candidate against.

The four candidates​

At bitcrowd we work with Elixir, Go and Rust day to day, with the occasional Node project on the side. So this wasn't a contest between a favourite and a lineup of strangers - we had hands-on experience to weigh on every side of the comparison.

Go​

Go is the incumbent. It's what Bluesky's own production DataPlane is written in, and the fit is strong. Goroutines and channels map cleanly onto "fan out work, gather results, respond." The bitmap work is comfortable in Go (GraphD itself is Go). gRPC support is first-class, deployment is a single static binary, and the tooling is mature. In-process fan-out is doable with worker pools draining buffered channels.

The costs sit at the extremes. Under heavy allocation Go's garbage collector starts eating CPU, and at very high socket counts the network backend can bottleneck on syscalls. Both respond to runtime tuning, but the tuning is ongoing work, not a one-time fix. And the in-process fan-out you build yourself comes with no supervision or isolation layer: a panicking worker takes the process down, and backpressure and lifecycle management would be our's to write.

Rust​

Rust gives the highest ceiling and the tightest control. No garbage collector means no GC pauses in the tail latency. Tokio handles enormous concurrency with low per-task overhead, tonic covers gRPC, and the roaring-bitmap and datastore libraries are excellent. For the compute-bound half of our workload, nothing beats it.

The cost is velocity. As this service is thin on business logic, we would pay the price of the borrow checker and the sharp edges of async Rust on every line while protecting very little. Fan-out with Tokio tasks and channels works very well. However, as with Go, we would need to build lifecycle, backpressure and supervision manually.

Node / TypeScript​

Node deserves real consideration because it's the language of the reference DataPlane and the rest of the atproto stack: PDS, AppView frontend, lexicons. One language across the codebase, shared types from lexicon definitions, the largest hiring pool, the fastest iteration. For the cold path's I/O-bound record fetches it's fine, and for modest-scale self-hosting it's a sensible default.

The hot path is where it breaks down. One event loop per process means in-memory queues and request serving compete for the same loop, and any CPU-bound work blocks both. Using multiple cores means multiple processes with no shared memory, which brings back the cross-process coordination the in-process design was meant to remove, and makes a large shared in-memory graph awkward. It's the right tool for the reference implementation and a strained one for a throughput-oriented production server.

Elixir​

Elixir runs on the BEAM, a runtime built for massive numbers of cheap, isolated, preemptively scheduled processes. Per-process garbage collection means no global stop-the-world pauses, so tail latency stays flat under load. Supervision trees give fault isolation and recovery essentially for free. For high-concurrency request serving and a backpressured in-process queue, it's the most naturally suited runtime of the four.

It has one well-known weakness: raw per-core compute. The BEAM optimises for concurrency and consistent latency, not single-threaded number-crunching. Byte-level work like set operations over a large follower graph is exactly where it's slowest. Taken at face value, that rules it out for a workload with heavy bitmap operations at its core.

Every candidate has a flaw — which ones can you fix?​

Four reasonable options, each with one thing standing between it and a clean fit. Instead of arguing it on paper, we prototyped toward each remedy, far enough to tell whether the flaw could be engineered away or was structural.

Go's GC and syscall overheads respond to tuning, and Bluesky's production DataPlane proves you can push Go a long way. But the tuning never ends. It's a knob you keep turning for the life of the service.

Node's single event loop has exactly one fix, running more processes, and that fix recreates the cross-process coordination and shared-state problems we were designing out. There's no way around it within the language.

Rust has no performance flaw. Its cost showed up as soon as we started building: hand-rolled concurrency, lifecycle and backpressure machinery on every path, for a service with little logic to protect. We realised that that cost would not shrink over time, but stay with us with every change we would need to make.

Elixir's flaw, per-core compute on tight loops, is real, and we hit it where you'd expect: the follower-graph set operations. But it stayed in one place. It didn't smear across the service. It sat in a single, well-defined component we could draw a boundary around, and a component with a sharp boundary can be replaced. That's what sent us back to Elixir for a second look.

Resolving the Elixir paradox​

Can that component actually be lifted out? Yes, and the reason has less to do with Elixir than with how our system is laid out.

Start with the claim itself, because it's often stated too broadly. "The BEAM is slow at CPU work" is true for tight numeric loops on a single core. It is not a claim about how much hardware a real service needs. Most of what a production server spends its time on isn't a tight loop; it's keeping thousands of concurrent requests moving, and the BEAM is efficient at exactly that.

Per-process garbage collection means no global pauses, and tail latency stays flat enough that you can run a node closer to its limit and still hit your latency target.

This is why Elixir services often need fewer machines than the Go or Node version of the same thing, occasionally approaching Rust, even though they lose every microbenchmark. Hardware cost under concurrency and single-core throughput are two different measurements, and only one of them shows up on your bill.

That said, our workload really does contain the thing the BEAM is bad at: the roaring-bitmap intersections and unions over the follow graph. We don't want to talk our way around that, so we move it off the BEAM entirely.

The follower graph lives in a Rust implementation of Roaring Bitmaps, called from Elixir as a NIF (Native Implemented Function). What makes this more than a generic "wrap the slow part in Rust" patch is the shape of the data flow:

• The input crossing the boundary is tiny: a user ID, or a small set of IDs.
• The graph itself, millions of edges, stays on the Rust side in native memory. The BEAM never holds it, never copies it, never garbage-collects it.
• The expensive compute, the intersections and unions, runs entirely inside Rust at native speed.
• Only the result crosses back, and because timelines are length-limited, that result is small by construction.

The usual objection to NIFs is the copy cost at the boundary. But in our case, only small data crosses in both directions while the big structure and the heavy computation stay native, which is the ideal case. That isn't luck. The timeline-length limit we'd already committed to is what bounds the return values.

This subtracts the BEAM's weakness from the hot path and keeps its strengths. The compute-bound half runs in Rust regardless of host language. What's left for Elixir is orchestration: high-concurrency request serving and the fan-out queue, the regime where the BEAM is at its best and where our own production experience says it's most efficient.

Of course, the NIF has its own price. It runs inside the BEAM's memory space, so a crash in the Rust code can take down the VM, and a long-running call can stall a scheduler. So we give up some of the "let it crash" isolation exactly at the native boundary. The mitigations fit this case well: the calls are short (small in, bounded compute, small out), and the Rust surface is small, stable, and the kind of code that rarely changes once written. Two languages is a real maintenance cost, but Rustler keeps the boundary ergonomic, and a small, bounded Rust core seemed the cheapest of the available evils.

The deciding factor: fan-out as code, not infrastructure​

The NIF made Elixir competitive. The fan-out requirement made it the choice.

Recall the spec: a deferrable queue that absorbs bursts, applies backpressure, delivers within minutes, and degrades gracefully when the backlog grows. In Go, Rust or Node, that almost always becomes an external system — Redis, NATS, RabbitMQ, Kafka — because the language doesn't give you primitives to do it safely in-process.

On the BEAM, those primitives are the language. Processes are the workers, mailboxes are the queues, supervisors handle recovery, and GenStage and Broadway add explicit backpressure, all inside one VM with no network hop and nothing extra to deploy. Several costs disappear outright:

• No serialisation across a queue boundary. An external queue serialises every fan-out job on push and deserialises it on pop, putting per-event encoding cost right back on the hot path we'd worked to keep it off. In-process, a job is a message between processes.
• No second system to operate. No separate scaling story, no "is Redis the bottleneck now," no disagreement between the service's view of the backlog and the queue's.
• One failure model. Supervision covers fan-out workers the same way it covers everything else. There's no seam between "the service crashed" and "the queue is in a weird state."
• Backpressure in-band. Producer and consumer share a runtime, so the producer notices the consumer falling behind directly instead of inferring it from queue-depth metrics after the fact.

The trade is durability. In-process means in-memory, and an in-memory backlog dies with the node. For us that's acceptable: fan-out is best-effort timeline population, timelines are bounded and ageing, and the fan-in path on read covers whatever goes missing for a window. We're choosing fast, simple and lossy-on-crash over durable, external and heavier, and the choice only works because the rest of the architecture makes the loss cheap. If you need durability here, much of the in-process advantage narrows. Check this assumption against your own tolerances.

Why Elixir​

We love working with Elixir, but we did not assume it to be the tool of choice for this project. The ecosystem just did not seem to be waiting for it. Choosing it means making the case for it, repeatedly, starting with this post.

And Elixir doesn't win every category. We came to the decision to use it because, for this specific workload, the one axis it loses on — per-core compute for graph operations — is the one we can cleanly offload to a small Rust NIF, with a data flow that makes the offload nearly free.

Everything that remains is concurrency, coordination, predictable tail latency under load, and a burst-absorbing fan-out queue we can build in the program itself instead of bolting on as infrastructure. Go would have been the pragmatic, proven middle.

Rust would have given us the highest ceiling at the cost of velocity and a lot of hand-rolled concurrency machinery. Node was right for the reference implementation and wrong for a memory-bound production server. Elixir plus a thin Rust core covers both halves of a workload that genuinely has two halves.

What next?​

Most of this is architecture-level reasoning informed by the workload's shape and each runtime's known characteristics. We backed this up with head-to-head benchmarks between the Node reference implementation and our Elixir POC.

For that, we needed to build a performance evaluation toolkit for our studies. If you want to find out the limits of your setup, check it out

The pieces we lean on hardest are well-sourced: Jaz's GraphD work for the in-memory graph and Roaring Bitmaps, and the documented existence of a fan-in/fan-out split between expensive paging and interactive-speed set intersection. The pieces we're least certain about - the exact timeline window, the precise per-request CPU breakdown - we've flagged as such.

We have build and tested a proof of concept that serves the skeleton amazingly fast. What is missing now is the boring part of writing the > 90 endpoints of the DataPlane API and the indexer plugins. Currently, it's a side project. Give us a shout if you want to help us take it further, faster.

Let's start by recapping the problem.

The AT Protocol enables us to build an open social ecosystem of interoperable applications. By far the most popular application in this ecosystem today is Bluesky.

Bluesky has open sourced almost all of their software. You can find their app, their Go services and their TypeScript code (backend services and protocol reference implementation).

This is their production code, with one exception: they don't run the open source dataplane. They've replaced it with a closed source implementation that is more performant.

We talked about some of the issues of the open source dataplane in our previous blog post.
The bottom line is: it won't scale if you're trying to serve hundreds of thousands of users.

Why you should care​

Even the Bluesky people think that hard decentralization is necessary. What is the point of an open ecosystem when everything is tied to the existence of a single company?

• Bluesky can be bought and start to enshittify
• Bluesky operates under US law, why should we want our online social life to be ruled by mad people inside a white house
• Bluesky has outages, alternatives could continue to operate

So, we should want alternative providers of Bluesky to exist.

Luckily for us, there is already a number of organizations picking up that work.
Luckily for them, they can build on the open source components that Bluesky provides.
Unluckily for them, they are left with a dataplane implementation that won't work for them when their userbase grows.

As a result, the Blacksky people have to spend their time rewriting components in Rust and our friends from Eurosky started exploring alternative ways to build timelines.

Serving timelines from Elixir​

Elixir provides a number of benefits for a backend service like the dataplane.
You work on a high level of abstraction, getting more done in the same amount of time and end up with very readable code (yes, this is still important, even for your agent!).
At the same time you get parallelism for free and performance that scales with the cores of the system.
Errors are isolated, so they don't bring down the whole service. As a bonus, the VM provides you with good tooling for introspection and observability, all out of the box.

So, when we built our Proof-of-Concept, we naturally chose Elixir. For the full reasoning behind that decision (why Elixir over Go, Rust and Node), see our companion post. We wanted to investigate the performance issues that will come up in the open source dataplane implementation. For this reason, we focused on implementing the critical endpoint which is responsible for serving timelines to users.

From our exploration in the last blog post, we already know that the most important decision to solve the issues of Bluesky's open source database implementation is to change the approach from fan-in to fan-out.

When a user requests their timeline from Bluesky's open source database implementation it will query the posts that should be in the timeline on demand. This is known as fan-in principle and is a major reason for the performance issues of this implementation.

Consequently, the implementation used by Bluesky in production turns this process on the head, known as the fan-out principle. It constructs the timeline for each user whenever a new post arrives in the system. When the user requests their timeline, the data is already waiting for them to be served.

ETS tables​

With this insight, it was clear to us that we should build our dataplane on the fan-out principle. It was less clear how we should store the data we need. The Erlang VM has multiple built-in options for storing data, so choosing one of them, the Erlang Term Storage (ETS), was a good starting point.

Citing the documentation:

Data is organized as a set of dynamic tables, which can store tuples.
...
Tables are divided into four different types, set, ordered_set, bag, and duplicate_bag. A set or ordered_set table can only have one object associated with each key. A bag or duplicate_bag table can have many objects associated with each key.

ETS stores data as tuples in tables in memory but which tables do we need to store all the data we need to serve timelines? And of which type should the tables be?

To store follows in a way that we can retrieve all the information we need efficiently, we actually need two tables: followers_table and following_table.

:ets.new(@followers_table, [
  :duplicate_bag,
  :named_table,
  :public,
  read_concurrency: true,
  write_concurrency: true
])

:ets.new(@following_table, [
  :duplicate_bag,
  :named_table,
  :public,
  read_concurrency: true,
  write_concurrency: true
])

These store the follows information in both directions, holding pairs of user IDs.
We can find all the followers of a user by looking up the entries with their user ID as key in the followers_table. And we can find all the users this user follows by looking up the entries with their user ID as key in the following_table.

Both tables are of type :duplicate_bag.
We need multiple entries per key, so we must use either :bag or :duplicate_bag as type of the table. :bag would have the benefit of preventing duplicates, however we chose :duplicate_bag for its performance benefits.

The posts_table represents the source of truth for all the posts in the system. It's a :set storing post IDs as we don't want duplicates here.

:ets.new(@posts_table, [
  :set,
  :named_table,
  :public,
  read_concurrency: true,
  write_concurrency: true,
  decentralized_counters: true
])

In the feeds_table we store the feeds of all users.

:ets.new(@feeds_table, [
  :duplicate_bag,
  :named_table,
  :public,
  read_concurrency: true,
  write_concurrency: true,
  decentralized_counters: true
])

Whenever a new post arrives in our system, we look up all the followers of the author of the post using the followers_table. Then, for each follower we store a pair of the follower's user ID and the post ID in the feeds_table. We chose the type :duplicate_bag here for the same reasons as we did for the followers_table. We need multiple entries per key (user ID) and we want the performance benefits of :duplicate_bag over :bag.

Celebrities​

With these tables we would have a functioning system. However, there is still the celebrity_posts_table to talk about.

:ets.new(@celebrity_posts_table, [
  :duplicate_bag,
  :named_table,
  :public,
  read_concurrency: true,
  write_concurrency: true,
  decentralized_counters: true
])

This table is actually a first optimization of the system. You might have wondered what would happen if a post arrives in the system authored by a user who has many many followers. If we would stick to our feeds_table, we would have to loop through all of them to insert the corresponding data.

For this reason, we introduced the celebrity_posts_table. The key idea here is that if a user has too many followers (is a celebrity), we give their posts a special treatment to avoid looping through all the followers and the associated performance issues. Instead we will store pairs of the author and post IDs in celebrity_posts_table, which is again of type :duplicate_bag.

def insert_post(post_id, author_id) do
  :ets.insert(@posts_table, {post_id, author_id})

  follower_entries = :ets.lookup(@followers_table, author_id)
  limit = fan_out_limit()

  if limit == :infinity or length(follower_entries) <= limit do
    Enum.each(follower_entries, fn {_subject, follower_id} ->
      :ets.insert(@feeds_table, {follower_id, post_id})
    end)
  else
    :ets.insert(@celebrity_posts_table, {author_id, post_id})
  end
end

When a user request their timeline, we will first look for their feed in the feeds_table, then do another lookup in the celebrity_posts_table to gather posts from celebrities they follow. The user will then see a combination of those posts. So, in fact this dataplane combines fan-out (for regular posts in feeds_table) with fan-in (for celebrity posts in celebrity_posts_table) for performance reasons.

def get_timeline(user_id) do
  fan_out_posts =
    @feeds_table
    |> :ets.lookup(user_id)
    |> Enum.map(&elem(&1, 1))

  celebrity_posts =
    @following_table
    |> :ets.lookup(user_id)
    |> Enum.flat_map(fn {_actor, followed_id} ->
      :ets.lookup(@celebrity_posts_table, followed_id)
    end)
    |> Enum.map(&elem(&1, 1))

  fan_out_posts ++ celebrity_posts
end

Evaluating our dataplane​

To get an idea about how much performance we gain with our dataplane, we evaluated it using our simulator. The dashboard below shows the results of this comparison.

We can see two blocks of data:

The first run was recorded with Bluesky's open source dataplane. We can see a maximum p99 latency of more than 60 ms. Furthermore, we see the throughput of requests per second fluctuating around 600.

In comparison, the block on the right shows recordings of the same simulation with our dataplane. The latencies are tiny, the throughput sits smoothly at 800 requests per second.

Crimeline comparison​

Crimeline is an experimental timeline builder which caught our attention when doing research for this project. Like our proof of concept, Crimeline is not a complete dataplane implementation. Unlike our proof of concept, Crimeline focuses on creating efficient datastructures and conveniently comes with synthetic benchmarks for them.
After having seen the end-to-end evaluation of our dataplane, we were curious how ETS tables would perform in those benchmarks.

The README describes one of Crimeline's datastructures, the UserMap, as follows:

Sharded adjacency map. Each uid is split via bitmask into shard index (low bits) and backbone index (high bits). Each shard holds a Vec<Vec<Uid>> — a dense backbone of sorted adjacency lists.

It's a struct that holds information whether a user ID is included in it (think your followers on Bluesky) with efficient methods to add and remove user IDs as well as performing the lookup of an ID.

An ETS-based equivalent looks like this:

defmodule UserMapEts do
  @user 0

  def new do
    :ets.new(:forward, [:duplicate_bag, :public, {:read_concurrency, true}])
  end

  def populated(n) do
    table = new()
    tuples = for t <- 0..(n - 1), do: {@user, t * 3}
    :ets.insert(table, tuples)
    table
  end

  def add(table, target) do
    :ets.insert(table, {@user, target})
  end

  def add_bulk_list(table, targets) do
    tuples = for t <- targets, do: {@user, t}
    :ets.insert(table, tuples)
  end

  def add_bulk_each(table, targets) do
    Enum.each(targets, fn t -> :ets.insert(table, {@user, t}) end)
  end

  def contains(table, target) do
    :ets.match_object(table, {@user, target}) != []
  end

  def remove(table, target) do
    :ets.delete_object(table, {@user, target})
  end

  def destroy(table) do
    :ets.delete(table)
  end
end

We can expect this to be a bit slower than the Rust implementation of Crimeline due to the overhead of the Erlang VM.
The table below shows the average time for each operation in nanoseconds when performed with 100 users.
From our results, we can see that ETS tables nevertheless perform relatively well.

So overall we were happy but two benchmarks stood out with growing numbers of users: contains_hit and contains_miss.

Note the units in the following table. While the Rust UserMap goes from 8 ns to 14 ns (not even 2x) when increasing the number of users from 100 to 100_000, ETS goes from 1.36 μs to 1.11 ms, almost 1000x.

Why is this so slow? From the ETS docs:

Insert and lookup times in tables of type set are constant, regardless of the table size. For table types bag and duplicate_bag time is proportional to the number of objects with the same key.

We chose a :duplicate_bag for the table to improve insert performance. Now we are paying for that with slow lookups at scale.

Roaring Bitmaps​

We didn't give up here, of course. So let's see what we can improve.

During our research, we also came across Jaz's blog and learned how to shrink the size of your datastructures and go even further with Roaring Bitmaps. The latter seems to be a good fit for optimizing our user map datastructure.

Luckily, rustler lowers the barrier to use Rust crates from Elixir a lot these days. Conveniently for us, Aaron Gunderson already did the work to wrap the roaring crate in an Elixir package and blogged about it. So, it's ready for us to use.

As a very brief summary, bitmaps are a suitable datastructure to compress the follower graph in our dataplane. Instead of storing pairs of {user_id, follower_id}, we store a single array for each user where each bit represents whether the user with the id at the index follows the user.

Say we have 5 users, and our ids start at 0. If the first four users all follow user with id 4, we would store the following pairs in our ETS table:

# user_id, follower_id
{4, 0}
{4, 1}
{4, 2}
{4, 3}

With a bitmap, we would store a single array of bits for the user with id 4. If the bit is 1, the corresponding user follows the user with id 4.

# user_id/index:
# 0, 1, 2, 3, 4
 [1, 1, 1, 1, 0]

You can see how that compresses the required space in memory in this simplified example.
Assuming 1 byte per id, we would have 4 * 2 = 8 bytes of memory in our ETS table variant. The bitmap however is complete with 5 bits which fit in a single byte.

Roaring bitmaps apply further optimizations on top of this basic idea.

With the roaring package we can plug the Roaring Bitmap into our user map module.

defmodule UserMapRoaring do
  def new do
    {:ok, roaring} = RoaringBitmap64.new()
    roaring
  end

  def populated(n) do
    list = for t <- 0..(n - 1), do: t
    {:ok, roaring} = RoaringBitmap64.from_list(list)
    roaring
  end

  def add(roaring, target) do
    RoaringBitmap64.insert(roaring, target)
  end

  def add_bulk_list(roaring, targets) do
    for t <- targets, do: RoaringBitmap64.insert(roaring, t)
  end

  def add_bulk_each(roaring, targets) do
    Enum.each(targets, fn t -> RoaringBitmap64.insert(roaring, t) end)
  end

  def contains(roaring, target) do
    RoaringBitmap64.contains?(roaring, target)
  end

  def remove(roaring, target) do
    RoaringBitmap64.remove(roaring, target)
  end

  def destroy(_roaring) do
    :ok
  end
end

With that we get the following results.

We're still slower than Crimeline's implementation but it's acceptable and more importantly does not degrade with the number of users.

Comparison in a more realistic scenario​

After satisfying our curiosity about Crimeline's synthetic benchmarks, we were wondering how the comparison would look like in a more realistic scenario. Therefore, we added another benchmark feeds that is somewhere in between Crimeline's benchmarks and the work we've previously done with our simulator. Interestingly, in this more realistic scenario the performance difference between the implementations diminishes.

Where to go from here?​

We showed how to build a basic dataplane implementation in Elixir, then added optimizations to squeeze out the issues we encountered.

One essential feature that is missing from our implementation is persistence. So far we store everything in memory, so our data wouldn't survive a restart of the server.

Our goal is to provide a simple to use dataplane implementation. So, the first idea would be to add persistence with Postgres and treat our in-memory data as an index to avoid costly queries and retrieve the data directly by primary key. Another idea is to provide an adapter interface so that everyone can use their preferred database.

We've seen how to combine Rust and Elixir to get the strengths of both while mitigating their weaknesses (Discord has a great write-up on the same idea). We dig into why this pairing fits a dataplane so well in our companion post; here the takeaway is practical. We can optimize the hot spots in Rust, confident the native code won't crash the VM thanks to Rust's memory-safety guarantees, while keeping the readable, high-velocity Elixir everywhere else. Whether it's Roaring Bitmaps or even wrapping the Crimeline datastructures, a whole ecosystem of crates is available to us.

With that, we have all the tools to build the remaining features a complete dataplane implementation needs and we are ready to add more advanced features like the social proof that is available in Bluesky.

Discourse alone wasnʼt enough — we needed a few major customizations to bring Wiblioʼs functionality and design to life. In this post, Iʼll walk through how we approached customizing Discourse so far, and the challenges we ran into along the way.

What is Discourse?​

Discourse is an open source internet forum system built with a Ruby on Rails backend and an Ember frontend. It already comes with a lot of functionality and preinstalled plugins, like chat, calendar, events, and more. Discourse is built around the idea of being customizable. Customizations are possible via multiple layers:

• Admin settings. Discourse exposes a surprising amount of behaviour through admin settings. Some examples: chat as overlay vs. fullscreen, calendars on category pages, fine-grained access rights, and so on.
• Plugins. Plugins can change backend behaviour and the frontend. Examples: Discourse Calendar and Discourse Chat
• Themes and theme components. These customize the frontend with CSS and JavaScript.

Figuring out what we needed to build​

Discourse has a massive codebase and can be customized in many ways through its settings. One important task we worked on together with the ZLB people was to identify which features could already be enabled simply by adjusting these settings. We also researched existing plugins to find out whether maybe a feature we needed was already implemented and available.

This was already quite some work and a question we asked ourselves regularly was: “Could we just solve this using some admin, site, or access right settings instead of implementing something new?” When a setting or existing feature was enough, we saved implementation and maintenance effort, important for the MVP. It’s also less likely that something breaks on core updates to Discourse.

In the end, three areas required custom work:

1. Allowing users to use the platform even without confirming their email address, and the extra security measurements needed to safely enable that.
2. Adding features that make the user feel like they are moving in their chosen community and not a general forum with all users.
3. Building a custom Wiblio theme.

Implementing the changes​

It became clear that the easy-access (1) and community (2) requirements were independent. So we decided to create a new plugin for each of them. As the plugins in the long run should be usable by others, this design choice also had the benefit that the use of the plugins is more flexible: Discourse instances might use only one or the other and should still work fine.

The plugins​

Implementing a plugin that in some way changes the core Discourse behaviour is a balancing act where you are trying to make sure the changes are in place, but in a way that is not too invasive or too fragile under core updates.

To adjust backend behaviour, Discourse provides a range of helpers, like event hooks, modifiers, and more. When those helpers are not enough, the last resort was to override an existing Discourse method.

To add an element to the UI, Discourse provides “Plugin Outlets”. These act as named slots in the DOM where a plugin can render its own content.

In the image below you can see the Plugin Outlets on the category show page. Each green box represents one outlet. So for example we used the above-main-container outlet to render a custom header component on most of the pages.

One thing we learned: building a brand-new page that the plugin fully controls was generally a lot easier than doing complex overrides for an existing page. We could still reuse Discourseʼs components and logic, but have full control under which conditions they are shown and what else we display on the page.

Adding new components in a plugin​

For the community plugin we added a full new “Dashboard” page that required several new components: a header, a card and a card button component.

We wanted these components to be themeable, but to also look good out of the box if a theme didnʼt restyle them.

To achieve this, we built a structured set of components following the atomic design architecture that Discourse uses, using Discourseʼs built-in components as much as possible. Each component was built with sensible CSS defaults, and exposed CSS variables to be overridden in the theme. Additionally, we attached classnames to the components and many of their elements to allow rich customisation.

We used the Styleguide plugin that’s recently been made core to discourse, to provide an isolated environment where we could implement our new components. For the new dashboard components, this meant we added new sections for the header, the card, and the card button component, as well as for the composite components such as the forum card and upcoming events card.

The theme​

The themeʼs job was to realize the visual design on top of everything the plugins and Discourse core provided. We worked with a colleague from the ZLB to extract the design tokens from the designs and implement them in the theme as a two-tiered system of CSS variables.

The Styleguide plugin made this much easier: separated from the functionality and state of the main app, we render the real components in every state and variant that’s needed, where they can be compared directly to designs. This is a great dev environment, leading to quicker and more consistent implementation, and allows the user interface components to be QAd independently of the rest of the work.

Wrapping up​

Wiblio was pre-released at re:publica 2026 🎉. But itʼs not fully live yet — so this post is more “notes from along the way” than a retrospective.

In this project, we focused on reusability: making sure every part we built could be reused on another Discourse instance: splitting features into independent plugins, building on Discourseʼs own components, exposing CSS variables, documenting each component in the Styleguide.

We also learned that “customizing Discourse” can mean a lot of different things — each new feature has been its own little judgement call between “use a setting”, “write a plugin”, or “tweak the theme”. We might revisit some decisions. Whatʼs worked for us so far is staying as close to Discourseʼs built-in extension points as we can, and accepting a new dedicated page over a fragile override whenever thatʼs an option.

What is CDP?​

First up, what is this CDP thing actually? In brief, it's a low-level JSON-API running over a WebSocket-connection to control Chrome's browser engine. It was originally developed for Chrome's DevTools UI, but Google at some point stabilized and documented it, so that it can be used by external tools as well. It's an alternative to the old HTTP-based WebDriver standard.

Outlook: WebDriver BiDi​

While CDP is tied to Chrome alone, there is also a new version of the WebDriver standard currently evolving. It's named WebDriver BiDi - as it's "bi-directional" - and essentially taking all the good parts from WebDriver and CDP and mixing them together. At the time of writing, it's still work in progress though and using it in Selenium requires dropping down to the raw driver.browser-level API.¹

Ruby CDP​

The Ruby CDP project makes an effort to build higher level Ruby APIs on top of the CDP protocol. At the time of writing, they have three projects:

1. Ferrum - a high-level API to control Chrome in Ruby
2. Cuprite - a pure-Ruby driver for Capybara based on Ferrum
3. Vessel - a high-level web crawling framework also based on Ferrum

The idea is to have pure-Ruby tooling based on the Chrome DevTools protocol without requiring too many external dependencies from other ecosystems.

First impressions​

We've been using FerrumPDF, a PDF library built on top of Ferrum to replace Grover in one of our projects. Grover's dependency on the Puppeteer NPM package was a bit of a maintenance burden and it's nicer to have less individual tools involved. Up until now, we're happy with the switch, but more on that in a different post…

Cuprite​

We also switched the test suite of a large Rails app from using Selenium to Cuprite as a driver for our Capybara end-to-end feature tests.

The pitch is:

Cuprite is a pure Ruby driver (read as no Selenium/ WebDriver/ ChromeDriver dependency) for Capybara. It allows you to run Capybara tests on a headless Chrome or Chromium. Under the hood it uses Ferrum which is high-level API to the browser by CDP protocol. The design of the driver is as close to Poltergeist as possible though it's not a goal.

Before​

The app is quite aged already and has a rather big and long running suite of feature tests. The tests used to run in a headless Chrome browser driven through Selenium. Under the hood, Selenium used Chromedriver, which is a standalone server that implements the WebDriver standard to then talk to a Chrome browser to perform actions in there. The setup still works well and we used it in a lot of projects over the years.

But for this particular app, the feature test suite was rather flaky. While being a classic server-side rendered Rails app, the app is quite JavaScript-heavy and contains sections which are pure ReactJS with state management on the client. With all of this in place, the feature tests of this app were unreliable: Elements may have been re-rendered and therefore become stale while Capybara wanted to interact with them, some interactions involved loading screens and animations and were taking longer than what Capybara expected, etc.

One error that appeared often was Selenium having problems with elements being re-rendered:

Selenium::WebDriver::Error::UnknownError:
  unknown error: unhandled inspector error: {"code":-32000,"message":"Node with given id does not belong to the document"}
    (Session info: chrome=144.0.7559.96)

In order to mitigate all the falkiness, there were things in place like this:

config.around :each, :js do |ex|
 ex.run_with_retry retry: 3
end

config.default_max_wait_time = 10

Capybara was set to wait up until 10 seconds - instead of the default two seconds - for asynchronous processes to finish. Test tagged with "js" would be retried up to three times before they would finally be reported as failing. Plus, on CI the failed tests were also retried once more before they would be marked as failing:

bundle exec rspec spec/features --exclude-pattern || bundle exec rspec spec/features --only-failures

With all of this in place, the feedback of CI was super slow. Failing tests would be retried up to six times until CI was finally done and red.

Running the test suite locally was equally painful. Skipping the retry mechanisms in favor of faster feedback meant dealing with falkiness: for any failing test, developers first needed to find out whether it's a failure related to their change or "just" a flaky test. This can be quite a time sink…

Switch​

People on the internet reported their feature tests being more reliable and less flaky when using Cuprite as a driver for Capybara compared to Selenium. So we were curious if this could improve our situation and went on a journey (together with Claude…) to switch the drivers and compare the speed and reliability to the Selenium setup.

We observed, that the tests did not run faster than with Selenium. But there was a huge difference when it came to reliability: we were able to remove all mechanisms in place for mitigating the falkiness with Selenium, from the retry logic up to the high wait time for Capybara.

The tests ran super stable, both locally and on CI 🎉

Migrating to Cuprite​

Migrating our test suite required some small adjustments to the tests here and there where Selenium and Cuprite behave differently.

External URLs​

This may be quite specific, but Cuprite won't load external, non-existing URLs. Instead, it will show a Chrome error page. So one cannot match on the current path for those URLs like it's possible with Selenium.

- expect(page).to have_current_path("https://localhost/non-existing-path", url: true)
+ expect_external_url("https://localhost/non-existing-path")

What we ended up doing instead, was checking the details of the resulting error page using a helper:

def expect_external_url(url)
  expect(page.current_url).to eq('chrome-error://chromewebdata/')
  expect(page).to have_css('h1', text: 'This site can’t be reached')
  expect(page).to have_button('Reload') do |button|
  expect(button['data-url']).to eq(url)
end

Form values​

For some form fields, the values extracted via Capybara where yielding a different type. A non-checked checkbox for instance:

- expect(find('.my-selector')[:checked]).to be_nil
+ expect(find('.my-selector')[:checked]).to be(false)

Matching across elements​

Again a quite specific issue: when matching the text across multiple elements, Cuprite is separating the elements via tabs instead of whitespace:

- expect(page).to have_content("foo bar")
+ expect(page).to have_content("foo\tbar")

Conclusion​

Migrating an existing codebase from Selenium to Cuprite did not require much adaption of the tests itself, but it helped to stabilize the overall suite a lot. Alone for that it's definitely worth considering!

But if you have time, maybe it's also fine to wait for full BiDi support in Selenium.

References​

Before you read any code​

The instinct is to start opening files. Don't. Read the shape of the project first. Four things, in order: the directory tree, the dependency manifest, the CI config, and the git log. Thirty minutes here saves days.

The directory tree​

tree -L 3 -I 'node_modules|.git|dist|build'

Glance at the output - you're noticing absences and asymmetries. A legacy/ directory, a missing tests/ folder, three directories that look like they do the same thing. These are facts about the project's history before you read any logic.

Watch for: a legacy/ or old/ directory (migration started and abandoned?); empty or missing test directories; duplicate-purpose directories (utils/, helpers/, lib/); top-level files suggesting a rename or restructure.

The dependency manifest​

cat package.json
# or pyproject.toml, Gemfile, pom.xml, go.mod

Skim it. The framework version, anything forked, anything that doesn't belong with the rest of the stack - these are usually visible in under a minute and worth more than reading any individual source file.

Watch for: forks pinned to a specific commit (almost always load-bearing - someone fixed something upstream wouldn't accept); a dependency that doesn't fit the stack (a queue library in a project that "doesn't use queues" is often the entry point to a subsystem you didn't know existed).

The CI config​

ls .github/workflows/ .gitlab-ci.yml .circleci/
cat .github/workflows/*.yml

Read it once. You're looking for what this team tests, what they skip, and how often they deploy. The CI config is one of the few places in a codebase where the team's actual habits, rather than their intentions, are recorded.

Watch for: jobs marked continue-on-error: true or conditionally skipped - usually a flaky test the team gave up on. Deploy steps that bypass the test suite. Environment matrices that test against runtime versions that don't match production. The gap between what CI runs and what production runs is often where the surprises live.

The git log​

git log --oneline | wc -l
git log --format="%an" | sort | uniq -c | sort -rn | head -20
git log --since="12 months ago" --format="%ad" --date=format:%Y-%m | sort | uniq -c

Three small aggregations: total scale, who carried this, cadence over the last year. None of these require reading a single commit message.

Watch for: a single author with disproportionate commits who hasn't committed recently - that person was the system, and they're gone. A sharp cadence drop without a corresponding completion usually means the team got pulled to other work, not that they finished. Recent activity from authors who weren't historically involved often means the codebase was handed off to whoever was available.

The pattern across all four: small commands, fast interpretation, a short list of things the interpretation might miss. Thirty minutes and you have a model of the project as an artifact - its shape, its history, its current state - before you've opened any source code.

The four entry points​

Once you know the shape of the project, you start with its entry points. Every system has four kinds, and reading them in this order saves you from getting lost in the middle: the boundary, the data model, the configuration, the tests.

The boundary​

# pick whichever applies to your stack
grep -rE "(get|post|put|patch|delete)\s*['\"]" --include="*.rb" --include="*.ex"
grep -rE "@(Get|Post|Put|Patch|Delete)Mapping" --include="*.java"
rg "router\.(get|post|put|patch|delete)" -t js -t ts

The boundary is where the outside world touches the system: HTTP routes, message queue consumers, scheduled jobs, CLI entry points, webhooks. You want the complete list, not a sample. The shape of the list - how many, how named, how distributed across modules - is the first real fact about what this system is.

Eighty-seven routes with no consistent naming convention tells you the system grew without anyone owning it. Twelve clean REST routes tells you someone cared, at least once. A cluster of routes named legacy_* or v1_* next to a smaller modern cluster tells you a migration that didn't finish. The grouping matters more than any individual route.

The data model​

find . -name "schema.rb" -o -name "*.sql" -path "*/migrations/*"
ls db/migrate/ priv/repo/migrations/ migrations/

The data model is the system's memory of what it thinks is important, and it changes more slowly than code. That makes it the most reliable guide to what the system actually is - code can lie about intent, but a table either exists or it doesn't.

Three tables called user, users, and accounts tells you a migration was abandoned. A column called legacy_id tells you a system this one replaced. Tables with no foreign keys to the rest of the schema are often integrations that were planned and never connected, or subsystems that were retired but not removed. The data model is where the system's archaeological layers are most visible.

The configuration​

find . -name "*.env*" -o -name "config*.{yml,yaml,json,toml}" -o -name "application.{yml,properties}"

Configuration is where you find out what's actually turned on in production, which is often not what the code suggests. A feature flag at 100% for two years is a fact about the system; the disabled branch is a fossil. An environment variable that's set in production but undocumented anywhere else is load-bearing by definition.

Watch for the gap between "config that exists in the repo" and "config that the code reads." Code that reads ENV["LEGACY_PAYMENT_MODE"] with no corresponding entry in any committed config file means someone, somewhere, has that variable set in production, and removing it will break something nobody can currently predict.

The tests​

find . -path "*/test/*" -o -path "*/spec/*" -o -path "*/__tests__/*" | head -50
find . -path "*/test/*" -type f | wc -l

Tests aren't useful here for correctness - legacy tests are often wrong, stale, or skipped. They're useful for intent. A well-named test tells you what someone, at some point, thought the code was supposed to do. The gap between the test names and the current behavior is exactly the gap between the system's designed reality and its lived one.

Heavily tested modules are where the team was nervous. Barely tested modules are either trivial or terrifying - and you usually can't tell which without reading them. Test files for modules that no longer exist in the codebase are common and informative: someone removed the code but kept the test, or vice versa, and either way it's a story.

The pattern across the four: for each entry point, get the complete inventory, then ask for the shape of the inventory rather than its contents. The shape is the information; individual items are usually noise at this stage. You're still not reading code. You're building a map of what the code is for, which is the prerequisite for reading any of it usefully.

Reading code, finally​

By now you have a map: the project's shape, its boundaries, its data model, its configuration, its test surface. You haven't read any logic. That changes here, but not in the way most engineers do it. The mistake is to open the most interesting-looking file and start reading downward. The better move is to pick a thread and follow it, with the agent walking alongside.

A thread starts at a boundary. Pick one HTTP route, one queue consumer, one scheduled job. Just one. The goal isn't to understand the whole system - it's to understand one path through it deeply enough that you can predict what the second path will look like before you trace it.

A few specific patterns worth watching for as you trace:

Conditionals that look unremarkable but exist anyway. A line like if user.created_at < Date.new(2019, 4, 1) is almost never trivial. Run git blame on it. The commit message is often the entire story: "fix INC-1247." That's an incident in 2019 that nobody on the current team remembers, and the conditional exists because removing it broke production once. The agent will explain the conditional accurately - "this checks if the user was created before April 2019" - without flagging that the existence of the check is the interesting part.

Defensive code with no obvious threat. Try/catch around what looks like a pure function. Nil checks on values that are clearly never nil. Retries on calls that shouldn't fail. These are scars. The thing that caused the scar may or may not still exist, but the code is treating it as if it does.

Dead-looking code that isn't dead. The function that looks unused. The module nobody imports. Before you assume it's safe to ignore, check.

A lot of "dead" legacy code is called by reflection, scheduled by a cron config nobody opened, or invoked by a feature flag that's set in production-only config.

How many threads to trace before stopping: three to five. Not the whole boundary. By the third thread, you should be able to predict what the fourth will look like before you trace it - same patterns, same idioms, same kinds of weirdness in roughly the same places. When you can do that, you've absorbed the system's grammar, and continuing to trace is procrastination dressed as preparation.

A heuristic that's worth more than its size suggests: read the bug fixes before the features. A six-line commit that fixes a production issue teaches you more about a system's actual behavior than a six-hundred-line commit that adds a feature. Bug fixes are where the system's observed reality intrudes on its designed reality. Find the last twenty bug-fix commits - git log --grep="fix\|bug\|incident" --oneline | head -20 - and read the diffs. You'll learn more in thirty minutes than you would in a day of reading feature code.

The artifacts you produce while reading​

Reading without writing is forgetting. By the end of your first few days you should have three artifacts on disk, all rough, all evolving. None of them are deliverables in the formal sense - they're tools for your own thinking that happen to be useful to the next person too.

The map.

A C4 model is what you want: a structured way of describing a system at four zoom levels - system context, containers, components, code. For legacy work, the top two levels are where the value is. The system context diagram names what this system talks to in the world; the container diagram names the major deployable parts and how they communicate. Both fit on one screen.

The act of coding the diagram is what builds the model in your head. structurizr is a great DSL for it.

The questions list​

Every "why does this exist," every "I don't understand this conditional," every "this can't be right" - write it down. Don't try to answer in the moment. The list itself is the deliverable for week one.

Half the questions will answer themselves by week two - you'll be reading something else and suddenly understand the thing that confused you on Monday. The half that don't are exactly what you need to ask the remaining humans about, in one focused conversation rather than forty Slack messages spread across two weeks.

Try to formulate your findings in BDD stile syntax. If you are able to write them down in that strict framework, you have a defined mental model of what is going on.

The third category is the one to pay attention to. Questions that started as "I don't understand this" sometimes turn out to be "this might actually be a bug nobody's noticed."

The glossary​

Every domain term, every internal name, every acronym. Legacy codebases are full of words used in non-obvious ways. "Account" and "User" and "Customer" mean three different things and the difference matters. "Order" might mean a purchase, or it might mean a sequence - depends on the module. Write them down as you encounter them, with where you saw them and what you currently think they mean. Update entries as you learn you were wrong.

The output is often surprising. A codebase where Account means three different things in three different modules is a codebase where every conversation between teams has been slightly broken for years, and nobody noticed because everyone was using the same word.

The glossary is the artifact that pays off latest but most. The map fades as you learn the system. The questions list shrinks as you answer them. The glossary keeps mattering forever, because every new person who joins the team needs it, and you're the only person who has just been through the experience of building it from scratch.

When to stop​

The trap at the other end is reading forever. Engineers who enjoy this work can spend weeks "understanding the codebase" and never ship anything. The agent makes this worse, not better - there's always one more module to ask about, one more thread to trace, one more cross-reference to run. Infinite patience on the agent's side meets infinite curiosity on yours and the result is a beautifully understood system you haven't actually done any work on.

You're done when you can answer three questions without going back to the agent or grepping:

• If a user reports a bug in feature X, where would I look first?
• If I had to add a new field to entity Y, where are the four places I'd have to change?
• If this system stopped working at 3am, what are the three things most likely to be wrong?

If you can answer those, you have enough.

Circling back to the 200k loc - Does it scale?​

As we have seen, agents issue far more commands than a human comfortably would, which means they often have more raw data to work from. They also don't tire when documenting their findings. But they have to manage their context window, and they often need humans to point out peculiarities or directions to pursue.

What works at scale:​

The inventory work scales fine. grep -r for routes returns more results but you still get a complete list. git log aggregations don't care how big the repo is. Glossary work scales — agents can chunk through the codebase incrementally. The C4 map at the top two levels is explicitly designed to compress; a system context diagram fits on one screen whether the codebase is 20k or 200k LOC.

Thread-tracing is where agents quietly fail at scale.​

In a 200k LOC system a single thread can touch fifteen to twenty files. After six or seven file reads at typical legacy file sizes, context starts crowding. Agents will follow a call graph six or seven levels deep without complaining — and then start compressing its earlier observations to make room, losing exactly the cross-file consistency that made the trace useful.

How to make that scale?​

Trace in segments. Three files at a time, summarize, clear context, continue.

You're now the one holding the through-line, not the agent.

The narration gets produced in pieces and you stitch them together. Less elegant than an end-to-end trace but it's what actually works at 200k.

Conclusion - What's the best strategy?​

Reading legacy code well is one of the most undertaught skills in this field. Engineers spend years learning to write code and approximately zero hours learning to read it.

It might have become clear that there is some aspects where agents have the advantage over humans. It's the stupelligent parts of the work where they are most powerful. To facilitate their strengths, it makes sense to codify the routines and standardize their output to a meta level humans and machines understand alike.

This is why we started to work on Surveyor, which extracts specifications from legacy systems and turns them into runnable BDD style tests. That's a different conversation if you're curious. There is an early access program if you would like to get engaged.

Three approaches to the same problem: by hand, with an agent, or with a deterministic tool built for exactly this task. The reading skill matters either way.

Sometimes that's the fifteen-year-old monolith. Sometimes it's the service that grew complicated faster than the team could keep up with. Sometimes it's that project that started as a shortcut and turned into a roadblock. Legacy code has been exposed to real world demands long enough to gather fixes for edge cases we are not aware of anymore.

And — increasingly — sometimes it's the repo your CEO vibe-coded last weekend and that they proudly present on Monday. It runs. Now you have to own it.

Legacy software is the code we lost confidence in​

That last category is genuinely new and growing fast. Code written once, never reviewed, tested a bit, deployed one day. Or, worse, released as a library and never maintained after that.

A new kind of technical dept has evolved: "The code we should review thoroughly one day."

"Can we rewrite this?"​

The actual problem is not the writing of the new code, but the discovery of what the old code does, and the certainty that the new one does the same thing.

We've been building two small Elixir tools to help with both halves of that problem.

• Surveyor scans a codebase in any language and produces an architectural model.
• Assay runs the same plain-text behavioral specs against both the legacy and the rewrite, so you can prove they behave the same.

Together they cover the workflow: Figure out the structure, capture the behavior, rewrite, prove the rewrite works.

How we got here​

Before going into either tool, it's worth walking through the shape of the problem. The picture builds up step by step, and the final diagram only makes sense once you've seen the holes in the earlier ones.

What we don't want​

Point an agent at a legacy codebase, ask it to produce a new one, hope for the best. Call it "the Claude zombie rewrite." You end up with a target codebase that nobody understands either, plus a generation gap between the old assumptions and the new. What has the agent overlooked, misunderstood, assumed? A black box gets replaced with a different black box. That is not a rewrite, that is a transcoding.

What we want instead​

The valuable artifact in the middle is not new code — it's a readable specification of what the old code actually does. Plain text, human-reviewable, version-controllable. The kind of thing you need when you want to ask the product owner if a feature is really wanted that way. - Or if it is just a legacy quirk.

If you have it, you can use it for estimates. It can be your map to plan the implementation and might even give you a hint about that stakeholder that you might have forgotten about otherwise.

Organised and illustrated​

A single big text file describing a whole legacy system is only useful for very small projects. You want the spec broken down by module — by bounded context — and the modules themselves arranged on a map of the architecture. Once each module has its own spec associated to it, two things become possible: you can divide the work, and you can reason about coverage.

Job done! Or is it?​

We now have and organised map of the system and a set of specifications. We have an overview, and the details. We could now talk to the stakeholders, the development team, and set to work.

However, we have not yet verified our findings.

What we are looking at is not yet a specification of the legacy system — it is a speculation of the legacy system. A document that claims to describe what the code does, written by reading the code (or by asking an LLM to read the code). Plausible, well-organised, reviewable. But unverified.

1. Verifiability​

This is speculation as specification, and it's the trap most "let's document the legacy system" projects fall into. A Word document of "what the system does" can be worse than no document at all, because people start trusting it. But even if the specification is done as a group effort by all stakeholders involved, and is thoroughly verified, it still lacks one critical property:

2. Reproducibility​

What we actually want is a verifiable specification: Not "verified once," but verifiable on demand, any time someone asks the question. You could verify a paper spec by hand, of course — sit down, read it, run the system, tick off the scenarios — but manual verification is expensive enough that it gets done once at sign-off and never again.

A verification you don't actually do is verification that doesn't exist. And unless the legacy codebase is a museum exhibit, it's a moving target:

The team is still shipping fixes, the system is still drifting, and a paper spec written on Monday is no longer accurate by Friday.

Runnable specs​

The fix is to make the specs executable — to wire them into something that can run them against the actual legacy system, on demand, and tell you whether they still hold.

That is the move from speculation to verifiable specification: not a one-shot audit, but a button you can press on every commit, every nightly, every time anyone asks "is this still true?"

A green run proves the spec is still true. A red run is a useful signal: either the legacy drifted, or your spec was wrong. Either way, you find out before the rewrite, not during it.

That magic is Assay​

Assay is the runner. It parses the .assay specs Surveyor produced, matches each step against bindings you write, and exercises the real legacy system through whatever interface it actually exposes — HTTP, CLI, message queue, file drop, whatever. The framework is small.

The runtime contract is "your bindings + your assertions + a deterministic runner."

The systemic approach​

Open up that "parts you fill in" box and you find the actual surface area of work: pattern recognition for step matching, a binding per target system, and Given/When/Then automations for setup, action, and verification.

The end state has a property that nothing in the earlier pictures had: the problem space has been dissected into manageable pieces. Architecture, behavior, target adapters, assertions — each is small, each can be reviewed independently, and each can be picked up by a human or an agent without having to hold the whole system in their head at once.

That's the goal of the toolkit. Everything below is the detail of how each piece is built.

Surveyor — discovering the architecture you inherited​

Surveyor is a Mix project that takes a path to a codebase and produces a Structurizr DSL workspace file. It runs in three phases mapping to the C4 model: C1 (system context), C2 (containers), C3 (components per container).

Crucially, Surveyor does not parse code. There are no language-specific parsers, no AST walkers, no fragile grammar files for thirty different ecosystems.

It scans the filesystem to identify what's there — languages, frameworks, project files, entry points, deployment manifests — and then feeds meaningful chunks to an LLM. That sounds trivial, but is the same mechanism that commercial coding agents use when they analyse your code.

Surveyor's intelligence is in the prompts and the chunking, not in a stack of half-broken language adapters.

The CLI is interactive at every phase.

$ surveyor ./legacy-monolith --phase c1

Phase 1 — System Context
  Scanning codebase...
  Querying LLM...

  System: "Order Management System"
  Description: Manages the full order lifecycle

  Actors:
    ✓ Customer — places and tracks orders via Web UI
    ✓ Warehouse Staff — manages fulfillment via Back Office
    ? Admin — found references in auth config (confidence: low)

  External Systems:
    ✓ Stripe — payment processing (REST API)
    ? SendGrid — found API key in config (confidence: low)

  [a]ccept  [e]dit  [r]etry with more context  [q]uit
  >

The LLM is asked to flag uncertainty. Anything tagged confidence: low shows up with a ? and a short reasoning string for human review. That's not just nice ergonomics — it's the bit that makes the tool trustworthy. An LLM that confidently invents a PaymentReconciliation container that doesn't exist is worse than no model at all. An LLM that says "I see a SENDGRID_API_KEY in .env.example but no Sendgrid client in the source, please confirm" is doing the right kind of work.

Each phase is resumable. Results are saved as JSON in ./surveyor/, so a thirty-container system doesn't have to finish in one sitting. You can hop in, accept C1, take a break, come back tomorrow, and resume at C2.

The end product is a workspace.dsl you can render in Structurizr. But more importantly, it's a workspace.dsl whose components are decorated with assay.specs and assay.schema properties, pointing at the behavioral specs and the domain schemas for each bounded context.

orderLifecycle = component "Order Lifecycle" "Orders" "Bounded Context" {
    properties {
        "assay.specs" "specs/order-lifecycle"
        "assay.schema" "schemas/order_lifecycle.ex"
    }
}

DDD pattern annotations from the LLM (customer-supplier, anticorruption-layer, …) ride along as properties on the relationship. Every component gets assay.specs and assay.schema properties — the contract the next tool reads from.

Examples​

The System Context of a Medical Application​

Surveyor identified the actors and the external systems of the application. Without knowing the code, we would already know which actor types there are

An example of an assay spec created in the second phase:​

Assay — proving the rewrite behaves the same​

Assay is a minimal behavioral spec runner. Specs are plain text and look like this:

Component: [orderLifecycle] Order Lifecycle
Context: Order Placement

Definitions:
  - "a valid customer" means:
      a Customer with status Active, verified email,
      and a credit limit greater than zero

Invariants:
  - total must not exceed customer credit limit
  - at least one line item required

Rule: Customers can place orders for in-stock items

  @critical @phase-1
  Scenario: [OL-001] Place a simple order
    Given a valid customer "Alice" with credit limit €10,000
    And product "Widget" is in stock with 50 units available
    When Alice places an order for 3 units of "Widget"
    Then the order status becomes "Placed"
    And stock for "Widget" is reduced to 47 units

The bracketed [orderLifecycle] is a workspace.dsl identifier — the same identifier Surveyor wrote into the architecture model. The bracketed [OL-001] on the scenario is an optional free-form id that survives across rewrites of the spec text and lets you cross-reference from tickets or audit trails.

If that looks like Cucumber or Gherkin, it does — with one large difference. There is no glue layer, no abstraction over what kind of system you are testing, no framework opinions about HTTP, databases, or browsers. The bindings are just Elixir.

defmodule Targets.Legacy.OrderLifecycle do
  use Assay.Binding, component: "orderLifecycle"

  @base System.get_env("LEGACY_API_URL")

  step :given, ~r/a valid customer "(?P<name>.+)" with credit limit €(?P<limit>[\d,.]+)/ do
    {:ok, resp} = Req.post("#{@base}/test/customers", json: %{
      name: params().name,
      credit_limit: parse_money(params().limit),
      status: "Active"
    })
    assign(customer_id: resp.body["id"])
  end

  step :action, ~r/.+ places an order for (?P<qty>\d+) units? of "(?P<product>.+)"/ do
    {:ok, resp} = Req.post("#{@base}/orders", json: %{
      customer_id: var(:customer_id),
      lines: [%{product_id: var(:product_id), quantity: to_int(params().qty)}]
    })
    assign(order_id: resp.body["id"], http_status: resp.status)
  end

  step :expect, ~r/the order status becomes "(?P<status>.+)"/ do
    {:ok, resp} = Req.get("#{@base}/orders/#{var(:order_id)}")
    assert resp.body["status"] == params().status
  end
end

A binding is just a module that brings whatever it needs — Req for an HTTP API, AMQP for a message broker, System.cmd for a batch processor, File for a directory-watching pipeline. Assay itself does not ship an HTTP client or a database adapter.

The framework provides parsing, step matching, scenario lifecycle, and assertions; the rest is your code. That's what makes the same spec runnable against an HTTP API, a CLI tool, a batch job, or a message queue, depending on what the legacy system happens to be.

The crucial property is target swappability:

assay run specs/order-lifecycle/ --target legacy
assay run specs/order-lifecycle/ --target new

Same specs. Different bindings. Different systems. The legacy binding hits the old SOAP API; the new binding hits the new Phoenix endpoint. If both go green, the rewrite is behavior-equivalent for the things the spec covers — which, after a few months of writing specs, is most of the things that matter.

There is no separate .exs file generated from the specs. The runner parses .assay files, matches each step against a regex on a binding function, executes them, and prints pass/fail. That's it.

How Assay actually works​

The runtime is around six hundred lines of Elixir. Five design choices do most of the work.

A two-pass parser​

The first pass lifts triple-quoted doc strings out of the file (so the line tokenizer doesn't have to reason about them). The second pass dispatches on the first keyword on each line.

def parse_string(content) do
  {filtered, doc_strings} =
    content
    |> String.split("\n")
    |> Enum.with_index(1)
    |> extract_doc_strings()

  Process.put(:assay_doc_strings, doc_strings)

  filtered
  |> Enum.reject(fn {line, _} ->
    trimmed = String.trim(line)
    trimmed == "" or String.starts_with?(trimmed, "#")
  end)
  |> parse_lines(%Spec{})
end

Output is a plain %Spec{} struct. No AST nodes, no string interpolation.

The step macro generates real module functions​

Each step :given, ~r/.../ do ... end call expands into a uniquely-named function plus a %StepBinding{} record on an accumulating attribute.

defmacro step(type, regex, do: block)
    when type in [:given, :action, :expect] do
  fn_name = :"__step_#{:erlang.unique_integer([:positive])}__"

  quote do
    @assay_steps {unquote(type), unquote(regex), &__MODULE__.unquote(fn_name)/0}

    def unquote(fn_name)() do
      unquote(block)
    end
  end
end

At compile time, __before_compile__ exposes __steps__/0 and __component__/0 for the runner to read. There is no string codegen, no eval — just normal Elixir function definitions.

Component-scoped step matching​

Anyone who has used Cucumber has hit the global step-definition problem: the same step text means different things in different contexts, but Gherkin treats step definitions as one shared namespace. Assay scopes bindings by component:

defp run_spec(spec, bindings, target, tags, exclude_tags) do
  component_bindings =
    Enum.filter(bindings, fn b -> b.component == spec.component end)

  # ... run each scenario against component_bindings only
end

This is the direct reason Surveyor and Assay use the same identifier convention. The architecture model and the spec runner share a vocabulary, and step text never collides across bounded contexts.

Pre-check, then execute​

Before running any step, the runner walks the whole scenario and finds a binding for every step. If anything is unbound, the scenario fails before any side effects.

def run_scenario(scenario, bindings) do
  matched_steps =
    Enum.map(scenario.steps, fn step ->
      case find_binding(step, bindings) do
        nil -> {:unbound, step}
        binding -> {:bound, step, binding}
      end
    end)

  if Enum.any?(matched_steps, &match?({:unbound, _}, &1)) do
    %ScenarioResult{status: :error, error: "Unbound steps found"}
  else
    execute_matched_steps(scenario, matched_steps, bindings)
  end
end

This is what stops a scenario from running half its Givens, hitting an unbound When, and leaving a dangling test customer in the database.

Per-scenario state in the process dictionary​

Each scenario gets a fresh context — a tiny module that stores variables, params, doc strings, and data tables in the process dictionary.

def init do
  Process.put(@vars_key, %{})
  Process.put(@params_key, %{})
end

def get_var(name) do
  Process.get(@vars_key, %{}) |> Map.get(name)
end

def set_vars(keyword) do
  vars = Process.get(@vars_key, %{})
  Process.put(@vars_key, Enum.into(keyword, vars))
end

The runner can save/0 and restore/1 the snapshot, which is how Assay tests itself by running its own .assay specs through itself in the same process.

There is no plugin system, no dependency injection, no scenario hooks beyond cleanup. Adding any of those would push complexity into the framework and out of the binding, which is the wrong direction. The framework exists to be small enough that you can read it on a Friday afternoon and trust it on Monday.

How they fit together​

The flow is straightforward and it maps onto the five phases laid out in the Assay handbook.

1. Discovery (Surveyor). Run Surveyor against the legacy codebase. Walk through the C1/C2/C3 output interactively. Edit, retry, accept. Produces workspace.dsl.
2. Behavioral extraction. For each component in the model, write .assay specs and an Elixir schema. The assay.specs and assay.schema properties on every Structurizr component tell you exactly where they go: specs/<context>/ and schemas/<context>.ex.
3. Validation. Write bindings against the legacy system in targets/legacy/. Run assay run specs/ --target legacy. Iterate until green. You now have a verified, executable specification of the legacy system's behavior.
4. Stabilization. Review what's covered, what's missing, what's flagged as ambiguous. Get sign-off on scope.
5. Rewrite. Build the new system. Write bindings against it in targets/new/. Run assay run specs/ --target new. When that's green, the rewrite is done — at least for the surface area the specs cover.

The reason the two tools sit next to each other is that the architecture model is what gives the spec work structure. Without a model, "write specs for the legacy system" is an open-ended task with no obvious stopping point. With the model, every component has a specs/ directory and the question becomes "is each context's behavior covered?" That's tractable.

The cross-target story is what makes the rewrite verifiable. The same parsed .assay scenarios are dispatched, step by step, through two different binding modules at two different points in the project's life:

Same spec, two bindings, two systems. The runner doesn't know or care which target it's dispatching to — that's the property that makes the rewrite a measurable thing rather than a leap of faith.

Four audiences, one artifact​

Four very different parties end up reading — or executing — the same .assay file: the product owner, the developer, the coding agent, and the test runner. The interesting observation is that they don't conflict on content — they conflict on form.

Product owners want to know what the system does in business terms, what would break if you changed X, and where the risk lives. They don't want UML. They want to read scenarios in their domain vocabulary, in plain language, and trust them.

Developers want to know how the system is structured, where the seams are, and what invariants they mustn't break. And — crucially — they want the documentation to be wrong less often than the code is. The moment docs lie, developers stop reading them.

Coding agents want machine-parseable, unambiguous, addressable artifacts with stable identifiers. They want to be able to say "the assertion at assay://orderLifecycle/order-placement/OL-001/step-3 failed" and have that mean something durable. They want types, schemas, and graphs they can traverse.

Automated tests want executable bindings — the layer Assay provides — and they want failures that point back to the spec, not just to a line of code.

Same facts, four presentations.

The .assay file gives the product owner readable scenarios in domain language. The workspace.dsl plus the schema modules give the developer the structural truth. The bracketed identifiers (Component: [orderLifecycle], Scenario: [OL-001]) give the agent its stable addresses. The runner turns the whole thing into a regression suite. None of these audiences is asked to read the others' representation; they all read the same spec, surfaced at the level of detail they need.

Where the LLM lives, and where it doesn't​

A reasonable question at this point: with all the recent enthusiasm for LLMs, why isn't more of this LLM-driven?

The split is deliberate.

Surveyor uses an LLM because architectural discovery is genuinely a language task — you're reading config files, route definitions, deployment manifests, dependency lock files, and inferring "this is the API, that's the worker, that's the read model." That's exactly the kind of fuzzy, evidence-weighing job an LLM does well, especially when you tell it to flag uncertainty rather than guess.

Assay does not use an LLM at runtime. The runner is deterministic: parse the spec, match the regex, execute the binding, assert. A behavioral spec that sometimes passes and sometimes does not, depending on which model variant happened to answer, is not a spec.

LLMs are very welcome on the authoring side — drafting .assay files from legacy source is a great agent task, and the handbook has explicit guidance for coding agents about how to do that without inventing behavior. But the green/red signal at the end has to come from a deterministic runner, or it is not really a signal.

How you use them on a project​

In practice, the first day of a legacy rewrite looks something like this:

1. Clone the repo. Run Surveyor with --phase c1 to get the system in context. Discuss the actors and external systems with the client — this alone surfaces "wait, who is the Admin role?" conversations that would otherwise happen three months in.
2. Run --phase c2 to map containers. This is where you discover the data plane that nobody documented, the cron job running on a forgotten server, the queue that two services share. Surveyor flags confidence: low items; the human resolves them.
3. Run --phase c3 per container. By the end you have a workspace.dsl that is, for the first time in the project's history, an accurate picture of what is deployed.
4. Pick the highest-risk bounded context. Read the legacy source. Draft .assay specs for its behavior. Write a legacy binding. Run it. Iterate until green.
5. Repeat per context, in priority order, until coverage is good enough to start the rewrite.
6. Build the new system, one bounded context at a time, with targets/new/ bindings going green as each context comes online.

The thing both tools are optimized for is the same: making the work legible. A legacy rewrite is a long project with shifting personnel and a nervous client. Both Surveyor and Assay produce artifacts — a workspace.dsl and a directory of .assay files — that survive turnover, show progress, and that the client can actually read.

Friendly User Trial​

Our next step is to test Surveyor on real world projects. We want to find out if the artefacts it produces are already helpful or need to be extended. This is where we need your help:

If you have a legacy system staring you down and a rewrite on the roadmap, get in touch. We would love to hear from you!

At bitcrowd we recently did a couple of those redesign projects for our clients. Let's explore how we demystify them, what strategies work for us and where the challenges are.

🌊 Problems​

Things on the internet age quickly. Some age well, some don't. A web application may very likely look dated and just not state of the art after some years out in the wild. This process is heavily influenced by the fast turning tech standards and the iterative nature of development projects, which can lead to a loss of an overall UX & design vision for the product.

The need for a visual overhaul of an application can also become apparent when the design stands in the way of new features. For example, an horizontal topbar for an app menu can be the right choice when the navigation is concise, but this choice does not scale as new primary features requiring an entry in the main navigation menu are added.

Finally, disorganized styles, templates, and generally a lack of componentization, can cause an increase of complexity for adding or updating views. Let's look at a classic example: buttons being styled by how they look, instead of what they are.

<!-- Looks -->
<button class="button button--blue"></button>

<!-- Roles -->
<button class="button button--secondary"></button>

When a developer is asked to change the looks of a button variant in the app, having a componentized button means there is only one file to change, as opposed to an endless amount of template files to update.

Redesign: how hard can it be?​

🤿 It's not just the looks​

In most cases, the term "redesign" goes way beyond changing a few colors, fonts or images. It is most likely a complete change of the look and feel of the app, including a new navigation, restructuring of content, etc. A redesign has to be carried on the entire view layer, not just the styles.

🎰 It's not just legacy​

Plus, the new design may also include new features. Some might be hiding in plain sight: a new redirection logic that is invisible in the new design files, or a text change that requires to wire the data from the controller differently. These can easily be missed when estimating the effort. There can also be real new features, which can be hard to estimate as they assume that the application is in a futuristic state.

🚧 It's blocking everything else​

Last but not least: it is practically impossible to release a redesign in small cycles. That would lead to Frankenstein-y UIs and user flows. In other words, the majority of the visual work has to be finalized before the launch: it has to be a big-bang production release. This can be problematic as redesign projects typically take months to reach a first release.

Strategy and timeline​

When clients approach us to implement a redesign for them, they usually have large parts of the new design already laid out by their (or external) designer(s). Starting from there, we mostly go through the same series of steps with them:

Design access and feedback​

Over the years, Figma ended up being our go-to tool for anything design related, being it design systems, prototyping, wireframing or UX design. There are alternative tools which do the same. We for instance used Zeplin for a while, too. Regardless of the tool, the important part is to have one source of truth for all involved parties.

In the ramp-up phase of the redesign, we review the designs with the client, check for consistency, verify what's doable within the app and check for completeness of the new design. The most important task here is to identify when the design includes new features which aren't present in the current state of the app.

Ideally, the designs should be "final" by the time the actual implementation starts.

Estimate the effort​

Estimating the effort of a redesign can be quite hard. When putting together the work packages and labelling them with their effort, separating pure redesign work from new features and backend-only work turned out to be useful for us. The key point is again to identify when the new design accidentally or deliberately adds or changes functionality and which additional effort this brings.

Another thing to take into account for estimating the effort is the potential overhead of switching over to a different UI abstraction or library. Let's say we're moving a Rails app from using partials to ViewComponents, an old Phoenix app from partials to PhoenixComponents or even from controllers and dead views to Phoenix LiveView. Doing this with consistency is essentially a rewrite of the whole view layer.

At best, one has thorough knowledge of the app and the codebase, its patterns, its degree of legacy and where complexity lies. It can be interesting to factor these in the effort estimation: how comfortable is the team with the codebase, what is the test coverage, are the views already componentized, etc.

UI Abstraction & Integration strategy​

Once the page designs are set up properly in Figma, they can be used as a basis for extracting design tokens like colors or spacing and shared components. It may seem tempting to be sloppy on this step, since it does not change anything within the app itself yet. But being concise about having the whole new design set up properly in Figma turned out to be absolutely worth the effort in 100% cases for us.

On our redesign projects we identify core components and build a design system and a component library. Those core elements, like typography, buttons, cards, etc, are built in isolation using the given frameworks capabilities to test and showcase them with Storybook (React), Lookbook (Rails) or PhoenixStorybook (Phoenix). Being able to develop our core components in isolation, showing all of their different states without interfering with the existing application is a huge boost for productivity as it allows building the components (styles & logic) independent from using them.

For the CSS, we often start completely from scratch. Either moving the old rules to a "legacy" folder of having the new ones in a separate redesign folder. One of them should be empty and gone at the end. It's always super satisfying to delete old code...

Implementation​

Any changes necessary from the backend are ideally done before the frontend ticket is worked on. So that when it comes to implementing the new design, all the data is already present. It can for instance also make sense to already render out plain data that is needed for a new design, but keep it hidden in the DOM. This way, the person taking on the design ticket has an easier time: with unit & feature tests already in place to make sure they did not break anything, and the changes in the PR being only on the view layer.

Without a tool like Storybook, depending on the size and complexity of the application, developers need to spend time to get the application into the particular state where a component behaves like xyz. Same when reviewing the work on the components. That can be an absolute time sink. Here again, componentized views and the associated tooling help the team!

Once a good portion of the redesign is done, it can make sense to re-estimate the effort. Maybe it turns out, that a technical decision had to be done that either simplifies a lot of the rest of the work, or - unfortunately easier the case - there was hidden complexity in some area of the application which slowed down the progress a lot. Re-estimating can also be a chance to cut down on feature changes and re-align the scope of the redesign.

Conclusion​

We're rolling quite well with our approach to redesigns! The right tooling is key: for us, Figma, some kind of Storybook and a framework to build UI components just make sense. Another core part is to have as much discussion about the designs and their consistency done before any actual development takes place. Tackled the right way, a redesign can be loads of fun for all parties involved. From a developer's perspective it just feels great to be able to iron out bad decisions from the past, refactor and update things left and right and just leave every part of the codebase nicer as it was before. For the clients, it's absolutely satisfying to see their ideas about the new design come to life bit by bit like blossoms in spring. Everything is - at best - fresh, more thought through and generally better than before!

The open atproto system provides a social graph of more than 40 million Bluesky users.
Building on this protocol, you can make your project everyoneʼs new favorite social network.

Interoperable alternatives to Bluesky, such as Northsky, Eurosky and Blacksky, managed to attract user groups by addressing their specific needs.

They all are built on the software components that Bluesky provides.

We will get an overview of those components and find out which problems arise when your userbase grows.

The Bluesky AppView​

The diagram below shows an architectural overview of Bluesky with the three main infrastructure components involved.

• Personal Data Server (PDS): every user has their own, this is where all their data is stored.
• Relay: polls every existing PDS on the network to collect changes and emit them in a single event stream called the Firehose.
• AppView: essentially is a regular app backend which receives events from the Firehose, processes them and makes data available via an API. The actual data storage is abstracted in a dataplane component.

For a write event, such as Create Post, a record is stored in the Content Archive inside the PDS of the user.
It is then emitted via the Relay into the Firehose.
The dataplane of the AppView subscribes to the Firehose to receive the event and indexes it into a permanent database.

To read their Timeline, a user performs a request which is proxied via their PDS to the AppView, which in turn retrieves the data from the dataplane and returns the timeline to the user.

All of the components displayed are available as open source software. Bluesky itself runs almost entirely on these open source components.

There is one notable exception: the dataplane, the part of the stack that processes the event stream of the atproto firehose and does the most heavy lifting. While it is available as open source, this dataplane implementation is not very performant. Bluesky uses a closed source dataplane implementation in production for this reason.

The open source dataplane functions on a fan-in principle:

1. events are streamed into the system (e.g. “new post”)
2. the dataplane stores and indexes the events in ordinary Postgres tables
3. on user request, data is queried and presented to the user (fan-in)

The Problems of the Dataplane​

The fan-in approach works well as long as the number of users is small enough. At some point, often with less than 100_000 users, the user experience degrades and the server runs into issues.

<--- Last few GCs --->

[55267:0x7bd40c000] 12985916 ms: Mark-Compact 4016.0 (4165.4) -> 3962.5 (4175.4) MB, pooled: 0.0 MB, 1065.65 / 0.00 ms (average mu = 0.156, current mu = 0.074) allocation failure; scavenge might not succeed
[55267:0x7bd40c000] 12987076 ms: Mark-Compact 4026.5 (4207.4) -> 3973.1 (4217.9) MB, pooled: 0.0 MB, 1078.12 / 0.00 ms (average mu = 0.116, current mu = 0.071) allocation failure; scavenge might not succeed

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

This episode of the pragmatic engineer newsletter lists a number of issues with Blueskyʼs Postgres-based dataplane.

• Connection pool issues and lock contention. The Postgres connection pool backup and Nodeʼs event loop got into a bad feedback loop. The team observed Postgres lock contention issues. This refers to multiple processes trying to access the same data simultaneously, but the data is locked to all except one process. For example, when multiple processes attempt to update the same row.
• Small Postgres outages. Postgres doesnʼt give the developer much control over which query plan it will take. Bluesky had a few smaller outages due to a query plan randomly flipping to something that ran about 1,000x times slower.
• The need for horizontal scaling. Horizontal scaling is adding more machines to a service, so that the throughput of this system improves linearly with each new machine. But Postgres does not support horizontal scaling because it runs as a single database with transactional guarantees, meaning it becomes a bottleneck – if a necessary one – for the entire network.

The bottom line is: the open source dataplane implementation is not optimized at all.

It exists solely to be able to say thereʼs at least one open source implementation available.

A performant open source dataplane​

The real trick to solve these problems is to change the approach. Instead of querying the data on user request, we prepare data in a dedicated place for each user on write (fan-out). When the user makes a request, the data is already sitting there waiting to be served to the user.

This is the also the approach followed by the Bluesky closed source dataplane which builds on ScyllaDB. From all publicly available information, that closed source dataplane seems very optimized to their hardware, and pretty expensive to run.

We followed the same approach for our proof of concept of a dataplane, which supports indexing post and follow events, as well as serving timelines to users.
As we wrote this dataplane in Elixir (our language of choice), we used ETS, the built-in storage of the Erlang VM, instead of ScyllaDB. ETS provides us with tables that can store tuples of data.

In the snippet below, you can see the insert_post function which indexes posts that arrive in the dataplane. First, we record the post in the global @posts_table. Then we lookup all the followers of the user in the @followers_table and finally store the post in the feed of each each follower, in the @feeds_table.

def insert_post(post_id, author_id) do
  :ets.insert(@posts_table, {post_id, author_id})

  follower_entries = :ets.lookup(@followers_table, author_id)

  Enum.each(follower_entries, fn {_subject, follower_id} ->
    :ets.insert(@feeds_table, {follower_id, post_id})
  end)
end

The read operation to get a timeline for a user is then a single lookup in the @feeds_table.

def get_timeline(user_id) do
  @feeds_table
  |> :ets.lookup(user_id)
  |> Enum.map(&elem(&1, 1))
end

Our ETS dataplane not only shows a significant performance improvement over the available open source implementation, it also has no operational overhead — ETS is built into the Erlang VM. So we run a single node with our Elixir application and thatʼs it.

As a bonus, we get all the benefits of running Elixir:

• good resource utilization and error isolation just come naturally with the parallelism provided by the Erlang VM
• tooling for introspection and observability is available out of the box
• in case of traffic spikes, the runtime will bend instead of breaking, and continue to serve the users
• no big refactor needed if we ever want to distribute the load over multiple nodes or start sharding the data

Our big goal here is that we build a dataplane that is low-effort to operate and runs on commodity hardware, but scales way better than Blueskyʼs open source dataplane.

A peek into performance​

As we want to focus solely on the characteristics of the storage technology and basic approach (fan-in vs. fan-out), we created two versions of the dataplane in Elixir:

1. the proof of concept we already talked about which follows the fan-out approach and stores data in ETS
2. a copy which differs only in following the fan-in approach and storing data in Postgres (and is therefore representative of Bluesky's open source dataplane)

With that, we could evaluate the proof of concept by measuring direct reads and writes to the dataplanes, bypassing all other factors that play a role in a real-world scenario.

We simulated write traffic and let a number of users make requests to load their timeline (get_timeline). In the dashboards below, you can see the number of simultaneously active users (“Active Sessions”), the write traffic (“Posts created”), and the latencies and throughput of get_timeline requests.

Under heavy load, the Postgres backed fan-in implementation showed significantly higher latencies and lower throughput in comparison to the ETS backed fan-out implementation.

Advanced simulations​

We extended this proof of concept into a performance toolkit you can use to simulate traffic via atproto events and user requests.

As a fundamental principle, we want to be able to repeatedly simulate scenarios. Therefore, you provide the configuration for a scenario as a file that you can load into the system.

Userbase parameters​

It makes a difference for your appʼs performance whether you already have millions of users or posts in your database or none. With the performance toolkit, you can quickly create base data to prepare your application instance for the simulated scenario.

Generate users in by passing in parameters in JSON format:

{
  "name": "my userbase",
  "num_users": 100000,
  "follower_density": 5.0
}

A power-law follower graph​

We assign the generated users sequential IDs, and they follow each other according to a power law.

All users follow the very first user account. With increasing user ID, the number of followers gets smaller quickly.

We do this for two reasons:

1. this distribution represents a realistic userbase where you have few accounts with many followers (looking at you @bsky.app) and many accounts with very few followers.
2. a predefined distribution makes it trivial to pick suitable accounts for the traffic you want to simulate, as you can determine the number of followers from the user ID. You donʼt have to first search a suitable account — to simulate a post from a user with many followers, pick a small user ID.

Simulating traffic​

To simulate traffic, we introduce two concepts: scenario and simulation plan. With these concepts we can represent repeatable and realistic simulations.

Scenario​

A scenario captures a series of user actions that will happen at defined time offsets during simulation.

At the moment three actions can be simulated:

• user creates a post
• user follows another user
• user requests their timeline

A single scenario can contain series for all three actions, but all of them are optional.

As you can see, these actions are associated with specific users and an offset. For instance, “user 1 posts at offset 1000ms” or “user 2 follows user 3 at offset 1500ms”. This series of events gives us a repeatable scenario.

You donʼt have to write the full scenario yourself, the toolkit lets you generate a scenario from parameters, similar to what we saw for the userbase.

Here is an example for posts: we have 100_000 users in the system but only the first 10_000 will post. We classify the users into tiers to make their behavior more realistic. Users with many followers will post more.

{
  "n": 100000,
  "max_active_user_id": 10000,
  "seed": 43,
  "time_units": 1,
  "tiers": [
    { "max_followers": 10, "posts_per_day": 0.2 },
    { "max_followers": 100, "posts_per_day": 1.0 },
    { "max_followers": 1000, "posts_per_day": 3.0 }
  ]
}

This will result in a list of post events like the following.

"posts":
  [
    {"user_id":124, "offset_ms":719},
    {"user_id":5037, "offset_ms":4160},
    {"user_id":6504, "offset_ms":21584},
    {"user_id":7422, "offset_ms":23714},
    {"user_id":1611, "offset_ms":27590},
    ...
  ]

The simulation will then emit a realistic corresponding atproto event at the given offset into the firehose.

For usersʼ requests, such as the get_timeline request in our simulator, you can use a similar set of parameters to define the user sessions.

{
  "n": 100000,
  "max_active_user_id": 10000,
  "seed": 42,
  "time_units": 1,
  "tiers": [
    { "max_followers": 10, "session_minutes": 10 },
    { "max_followers": 100, "session_minutes": 30 },
    { "max_followers": 1000, "session_minutes": 60 }
  ]
}

With the following resulting sessions.

"sessions":
  [
    {"user_id":4684, "duration_ms":1800000, "offset_ms":9638},
    {"user_id":8021, "duration_ms":1800000, "offset_ms":17306},
    {"user_id":567, "duration_ms":3600000, "offset_ms":19438},
    ...
  ]

Each user will start a session at the corresponding offset and will perform requests to get their timeline in fixed intervals for the whole session duration.

Scenarios are intended to represent typical real world traffic, for instance a regular “saturday morning”, or you know, a “shitstorm”.

Simulation Plan​

A simulation plan contains multiple scenarios that start at defined time offsets.

Example:

simulation_plan = [
  %{scenario_name: "saturday morning", path: "saturday_morning.json", offset_ms: 0},
  %{scenario_name: "shitstorm", path: "shitstorm.json", offset_ms: 100_000}
]

In this example simulation plan, the simulator starts with traffic as defined in the “saturday morning” scenario. At the offset of 100_000 ms, it will additionally start simulating the traffic as defined in the “shitstorm” scenario.

Every simulation run follows a single simulation plan.

Repeatable Simulations​

As stated before, providing repeatable simulations is a primary goal of our simulator, but we still want to enable interactive use for experimentation. If you add scenarios interactively to the running simulation, they will be included in the current simulation plan at the offset corresponding to the current runtime. That simulation plan will then automatically be exported.

So after noticing issues during experimentation, you will always have a simulation plan that you can run again to confirm the issues you noticed are real.

Measuring performance​

Based on the simulated traffic, we can measure the performance users would experience.

The performance toolkit includes a Docker Compose setup for Prometheus and Grafana, with a preconfigured Grafana dashboard.

The key question weʼre interested in is how long it would take under the simulated load for a user to receive their timeline. The dashboard shows the p99, p95, p50 latencies so we can assess this.
To gain insights into the running simulation, we also show how many user requests are performed per second and how many user sessions are active. Furthermore, we show how many post and follow events are created, dispatched, and dropped.

The outcomes​

With our performance toolkit, we ran simulations with up to 40 million users, the actual scale of Bluesky.

The dashboard below shows a comparison between the open source dataplane and our ETS-based proof of concept. In this case, we ran a simulation with 4 million users, which caused around 800 get_timeline requests per second, as well as some traffic for created posts and follows.

In the screenshot of the dashboard we can see two blocks of data.

The first run was recorded with the open source dataplane implementation.
We can see a maximum p99 latency of more than 60 ms. Furthermore, we see the throughput of requests per second fluctuating around 600.

In comparison, the block on the right shows recordings of the same simulation with our ETS-based dataplane. The latencies are tiny, and the throughput sits smoothly at 800 requests per second.

Here a zoomed in view on the results for the ETS-based dataplane.

Next steps​

Our big goal is an open source dataplane implementation that requires little effort to operate and scales better than the existing one. For now, our proof of concept only indexes posts and follows, but there are many more relevant event types. For comparison, Blueskyʼs dataplane implementation has 14 indexer plugins to cover all of them.

Furthermore, our proof of concept only supports a single endpoint for users to request their timeline. To complete the implementation, many more endpoints must be implemented — Blueskyʼs implementation has 96 endpoints, supporting all the features youʼd expect, including moderation and search. However, most of the endpoints are straightforward.

Lastly, we can still optimize the ETS-based implementation in many ways. And if we ever hit a performance issue, ETS as storage technology can be supported by or even completely replaced with other natively implemented datastructures, while we can retain the Elixir implementation of the dataplane server.

The problem​

One of our client provides generated, dynamic, potentially large, watermarked PDFs to their users. The PDF content is generated based on rich text content stored in the database, that is assembled into HTML. A small JavaScript snippet runs on the generated HTML to add a watermark to each page. The rendered data is then given to Ferrum PDF to generate the actual PDF file.

Letʼs look at the weaknesses of this flow:

1. First of all, the requests are processed synchronously and the PDF is generated in memory. A malicious user, or - truly - just your regular impatient user, can bring down the server by nervously clicking on the download button X times in a row 💥.

2. Additionally, large PDF generation (around 100 pages) can take between 2 to 5 minutes. In the age of doom-scrolling, 5 minutes are not acceptable for your average user. If they close their browser, we have successfully used server resources to generate a PDF that no one will receive.

Embracing asynchronicity​

To solve the first problem, we need to track that a user requested a PDF. This way, if they are late for their train and hit the button 10 times, we do not generate the PDF 10 times, just once. We introduce a new model in our database, that links a user with an Active Storage attachment.

class PDFRequest < ApplicationRecord
  belongs_to :user
  belongs_to :resource # The resource is the model containing the rich-text content for the PDF
  has_one_attached :document, dependent: :purge
end

The user is redirected to a page showing the status of the PDFRequest that was created, and a small JavaScript snippet refreshes the page regularly. Once an attachment is present on the PDFRequest record, the download starts. This naturally calls for an asynchronous processing of the request. Sidekiq jobs to the rescue!

This already has some benefits:

1. We prevent crashing the server as the PDF rendering is done asynchronously (we control the amount of parallel jobs in the queue).
2. If a user closes their browser, then reopens it later and asks for the PDF again, we redirect them to the existing PDFRequest. No waiting time, the PDF was already generated and is ready to be downloaded.

But the actual rendering of the PDF can still take some minutes. If many users are asking for a PDF, their waiting time can add up as they could be last in the job queue!

Hash it! Cache it!​

One thing we did not mention so far is a characteristic of the PDF files that we generate. This is specific to our client: they can look identical for two different users. If the upstream resource changes, the PDF should change as well. But as long as this resource does not change, meaning the rich-text content does not change, we could save some time re-using an existing PDF. As long as we know that it did not change!

A classic, standard way to do this is by hashing the file. But in our case, we do not take the risk of hashing the PDF itself, since we cannot be sure if that gives a deterministic output. Instead, we hash the HTML output which should be deterministic: we are pretty confident that the HTML generated from the same content should look identical.

Our Sidekiq job can also be updated to follow this flow:

Hereʼs more gold: no need to store the HTML checksum in the database! The generated PDF can be stored in a cache, with the hash checksum as its key in the store. We can then rewrite the rendering code like so:

def render_pdf
  html_checksum = Digest::MD5.hexdigest(html)
  Cache.file_cache.fetch("pdfs/#{resource.id}/#{html_checksum}", expires_in: 1.day) do
    FerrumPdf.render_pdf(
      html: html,
      pdf_options: {
        format: :A4,
        scale: 1.0,
        ...
      }
    )
  end
end

This has great benefits:

1. The PDF does not need to be generated for each (user, resource) pair, but only once. The first user to request a PDF for the resource will wait, but everyone else after them will not (until something changes on the resource).
2. Using the hash approach preserves us from two other approaches:

• Hook on any change made to the resource. While this could be a viable option, in our case the “resource” is actually a very complex entity relationship model and tracking changes made to associated models would be hell.
• We are not fans of database triggers for this use case because they are rather invisible from the codebase perspective.

3. Moreover: while these two other approaches operate on the content, the hash approach operates on the generated HTML. This gives us something extra: if we change the CSS, JavaScript, or the HTML layout (literally anything listed in the HTML output) used to render the PDF, the computed hash would change. In other words, we get a neat and clean cache invalidation: in case the content changes or in case the looks of the PDF changes, the cache would not be hit, and we would re-render a fresh PDF.

Note that for this to work, you need to make sure the HTML output is deterministic. This means for example that you cannot have random IDs in the DOM.

Wanna see me run to that mountain and back?¹​

At this stage, the PDF rendering is already much better. It runs asynchronously and leverages the redundancy of identical HTML outputs. But we can do more. The first user to request a PDF for a resource that is not cached yet still needs to wait 2-5 minutes to get the PDF. In our case, the same sections can be rendered for different users across resources, so we could actually cache parts of the HTML.

Rails Fragment Caching to the rescue!

- resource.sections.each do |section|
  - cache section do
    = react_component 'RichContentViewer', props: { content: section.content }, id: "rich-content-#{section.id}"

This is just given by Rails, and we can use it to shorten the rendering time when the PDF is re-generated.

Conclusion​

In this article we saw an iterative approach to optimize the rendering of large PDF files in Rails. First by adopting a queue system, then by caching the entire PDF files, and finally by using Rails fragment caching. The rendering time at our client has gone from 2 - 5 minutes down to some seconds! 🎉

The Challenge​

Steady Media needed to migrate 600,000 users from Chargebee to a new in-house billing system. The complexity was brutal:

• Chargebeeʼs billing logic is opaque - we had to reverse-engineer behavior through observation
• Historical data had to work in the new system (canʼt rewrite the past, especially not payment transactions!)
• Multiple payment gateways (Braintree, GoCardless, PayPal) with different quirks
• Complex billing workflows: trial subscriptions, gift subscriptions, mid-term subscription upgrades, subscription cancellations (with or without refunds)
• Complex support workflows: issuing refunds, invoice & credit notes document versioning
• Legal compliance (invoicing and taxes)
• Migration had to happen without downtime or data loss

Steadyʼs development team asked us to lead the Charger project, so that they could focus on their core application. The tech stack had to be compatible with their team skills: Elixir & Phoenix!

The Journey​

bitcrowd built Charger across three iterations:

2021: Core System​

The first iteration aimed at building the data model and implementing the external communication layer of Charger:

• API layer with authentication between Charger and the Media Makers app
• Webhook notifications between Charger and the Media Makers app
• Payment providers: integrated Braintree and GoCardless services (API & webhooks)
• Multiplexed calls from the Media Makers app to hit Chargebee OR Charger depending on where the user lived

Once these various components were finalized, we could start the implementation of the core flows: create new subscriptions, bill users and trigger payments, upgrade subscription plans with prorated billing, settle payments, and so on.

2023: Feature Parity​

After the first development cycle ended, new features were added to the Media Makers app that needed to be back-ported to Charger. We added new models and new flows to support trial and gift subscriptions. We took the opportunity to add Paypal as a third payment provider. Finally, we also tackled the generation and design of legal billing PDF documents.

2025: Admin UI, Support Tools, and Migration​

A missing piece in Charger was a simple, lightweight, workable admin UI, that could be useful for Steadyʼs developers, QA engineers, support staff and financial experts. The goal was to minimize the implementation effort, while preserving a clean UX. We opted for DaisyUI since we knew the admin UI would use standard components like index tables, description lists, navigation elements etc.

Powerful search​

One strong requirement was to make sure that the index pages had a powerful and performant search as well as sorting and filtering mechanisms, as one of the pain points of Chargebee had been searches that time out once enough data was in the system. We solved this issue by paying attention to the queries' efficiency. Additionally, for cross-table filtering, we implemented a search-typeahead dropdown in order to avoid expensive joins.

Activity logs​

For such a billing product, it was crucial to allow admin users to see a clear changeset history on any resource. Charger comes with an auditing layer, that tracks changes and versions each relevant resource (like payments, invoices...). We built an abstraction on top of the auditing, that allows to plug any resource in it and view its version history with a UI resembling a Git diff.

Support staff tooling​

At that stage, we had a working product for developers, QA engineers and financial experts. But Steadyʼs support team still needed tailored features to enable them to solve specific situations. For example, when a userʼs credit card expired, and the system could not withdraw money from their account, the user might send the missing amount via a bank transfer. On Steadyʼs bank account, the balance is correct, but Charger does not know about this transaction, and the accounting balance is affected. Similarly, the support team might refund a user via a manual bank transfer. To keep the accounting in check, we implemented a solution to record offline refund & payment, as well as various manual actions & flows to fix any situation that would not auto-heal.

Outcome​

We migrated all 600,000 users to Charger. The lights stayed green. Steady now controls their billing infrastructure and its support team can handle edge cases and requests through the admin UI. The developer team can also leverage the activity logs and powerful search capabilities to debug or resolve errors.

bitcrowd led on self-contained packages: we took topics Steadyʼs team didnʼt have capacity for and delivered them finished. We were able to coordinate with stakeholders (support team, finance, legal) to turn “we need X” into actionable tickets. We applied our standards of excellent documentation and test coverage, high code quality and review, which ensured a smooth handover with Steadyʼs team.

This blog post explores the strategy that was used to achieve this challenging task.

The backstory​

Steady supports Media Makers in building an audience and driving revenue via subscriptions to newsletters, blogs etc. While most of the tools and features for Media Makers live in their main application, they outsourced the subscriptions & billing management to a third-party service (Chargebee). The Media Makers app was very tightly coupled to Chargebee as it relied on it for core business logic within the app, as well as for accounting.

Introducing: Charger, the fraternal twin​

Although Chargebee was useful initially, it became increasingly painful to work around their standard processes over time. This ultimately led to the development of a custom billing tool.

For this we had to reverse-engineer all of what Chargebee was doing. We started by building the required schemas, and exposing the API endpoints and webhook notifications that were needed by the Media Makers app. Any changes in the Media Makers app usage of the API/Webhooks, or to Chargebee itself, needed to be propagated to Charger. Both billing systems had to behave identical from the outside.

The Media Makers app now needed to forward calls to Charger and/or Chargebee. A “multiplexer“ abstraction close to the API & Webhook layer was added, so that for developers & users of the app, the fact that Charger or Chargebee was used for the calls would be transparent.

Charger needed to process payments and refunds, with all payment providers supported by the Media Makers app. The first flow of the application was created: subscribe to a publication, generate an invoice, create a payment, and settle that payment 🎉!

A bird watching experience 🔭​

At this stage, no real users were migrated to Charger. No real users were even created on Charger! The system just worked™... Subscription statuses, invoices, payments processing being crucial to the system meant that we had to come up with a transition plan, that allowed the developers to observe and watch the behaviour of the system for just a handful of users. If any problem arose, it would need to affect only such a small amount of users that their support team could resolve it manually. We handpicked 10 users with rather simple data: a single recent subscription, no cancellations, and various payment providers / currencies to cover as much ground as possible.

Using our beloved Oban for job processing, we built a migration job, that would, for a given user:

1. Fetch all their data from Chargebee
2. Format it to please Chargerʼs data model
3. Send it to a dedicated endpoint in Charger
4. Charger creates all the given data for the user (subscriptions, invoices, payments, etc.)
5. Cancel the user subscriptions on Chargebee (so that they donʼt get billed twice!)

Of course, all of this must happen in a transaction on both systems, fail gracefully and report meaningful errors to the devs. We leveraged Oban configuration to optimise the parallelisation and retry mechanism of the jobs, as we could be rate-limited by Chargebee. At some point, we would want to migrate users in batches of 100-10k users at a time, so the migration mechanism had to be robust and auto-healing.

Silent bugs​

A main challenge in this process was to validate the integrity of the migrated data. The Media Makers app and Charger run on different databases, and have their own representation of what is an invoice, a subscription, a user, etc. Forgetting to migrate a field, or mapping the wrong timestamp from one system to the other, would happen silently because a part of the migration code lives in the Media Makers app, and the other part lives in Charger. The data could be silently missing or corrupted, and we would only realise down the road that all users were migrated with errors. Luckily, we had the chance of having access to a Chargebee staging instance with users in all kinds of configuration. This helped us a lot in finding migration issues by comparing the user data on both systems before and after the migration, and strengthening our tests to cover all the edge cases we could find. The developers at Steady also added ERPC, which enabled us to run integration tests evaluating the effects of Charger on the Media Makers app.

Conclusion​

Once we were confident that our 10 users were rolling, we migrated 10 more users with more complex data. And so on, for weeks, increasing the batch size, and deciding to create new users on Charger with a rand() choice. This allowed Steady to steer the future of their billing strategy for their users, integrating with other payment providers and simplifying some of the user flows for refunds etc. While it looks simple on paper, this project required a lot of patience, collaboration and planning ahead as the stakes were really high and we had to build up confidence in the new system and in the migration approach.

Scenario​

On this project, we were implementing the new design for a particular page. At it's core, it shows two lists, each with a "Load More" button to asynchronously fetch and append more items to it. The first list is just a stack of items, the second one also includes a grouping of the items. Imagine it roughly like this:

In "List A", clicking "Load More" should append another batch of 5 items to the bottom of the list until the bitter end. For "List B", clicking "Load More" should also append another batch of 5 items, where some may still belong to "Group C" and others will form new groups, each with their respective group header. The list items are heterogeneous and contain complex markup with graphics, etc.

Pagination​

We decided to implement the pagination logic using Turbo Streams, as they seem to be a good fit for our use case. They also integrate nicely into the Rails ecosystem without being too tightly coupled (e.g. to a specific version of Rails). From Rails 7 on, Turbo also replaced Rails' UJS (Unobtrusive JavaScript) functionality and is the recommended tool for sprinkling asynchrony and SPA (Single Page Application)-feelings. From the Turbo Stream docs:

Turbo Streams deliver page changes as fragments of HTML wrapped in <turbo-stream> elements. Each stream element specifies an action together with a target ID to declare what should happen to the HTML inside it. These elements can be delivered to the browser synchronously as a classic HTTP response, or asynchronously […]

And further on:

They can be used to surgically update the DOM after a user action such as removing an element from a list without reloading the whole page […]

This sounds like a match for what we are doing!

By having our server respond with turbo_stream templates for the pagination requests, we are able to use the same View Components and partials we already have in place from the initial server-rendered page. This is important to us because we're already using our component library throughout the whole project. Reimplementing the same components in a different framework or language just for this single page would be unnecessary duplication, not to mention the constant risk of the two implementations diverging.

Example​

For the sake of simplicity, let's focus on the first list, "List A". The initially rendered page would have a markup like this for the list:

<h2>List A</h2>
<ul id="list-a">
  <li><abc-item/></li>
  <li><abc-item/></li>
  <li><abc-item/></li>
  <li><abc-item/></li>
  <li><abc-item/></li>
</ul>
<turbo-frame id="load-more-a">
  <load-more-button/>
</turbo-frame>
<!-- ... -->

When the user clicks the "Load More" button, we want to retrieve the next batch of items, append them to the list and replace the button with one to request the next batch. If there is no next batch of items, we remove the "Load More button.

In the controller taking care of the pagination request, we can render a response template of the format turbo_stream. A simplified version could look like this:

<%= turbo_stream.append 'list-a' do %> <%= render
AbcItem.with_collection(@items, as: :item) %> <% end %> <% if @pagy.next %> <%=
turbo_stream.replace 'load-more-a', method: :morph do %>
<turbo-frame id="load-more-a">
  <%= render LoadMoreButton.new(path: '/', params: { page: @pagy.next }) %>
</turbo-frame>
<% end %> <% else %> <%= turbo_stream.remove 'load-more-a' %> <% end %>

Note that the example is View Components and the pagy pagination library. In the template, we're appending items to "List A" and replace the "Load More" button with a new one. If there aren't any items left the "Load More" button gets removed.

The response from the server would then look roughly like this:

<turbo-stream action="append" target="list-a">
  <template>
    <li><abc-item/></li>
  </template>
</turbo-stream>
<turbo-stream action="remove" target="load-more-a">
</turbo-stream>

It appends one item to the list and removes the button to load more.

Problem​

Our <abc-item/> element contains a React component, MyComponent in its markup, rendered via React on Rails' react_component helper. In the final HTML those React components appear as three elements each. Two script tags with context and props and one wrapper element for the rendered HTML of the component:

<script type="application/json" id="js-react-on-rails-context">
  …
</script>
<div id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410">
  <!-- MyComponent markup -->
</div>
<script
  type="application/json"
  class="js-react-on-rails-component"
  data-component-name="MyComponent"
  data-trace="true"
  data-dom-id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410"
>
  …
</script>

For any <abc-item/> element added to the page via Turbo Streams, those three elements are also added to the DOM, but the wrapping <div> stayed empty: the component wasn't actually mounted! 🙀

<script type="application/json" id="js-react-on-rails-context">
  …
</script>
<div
  id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410"
></div>
<script
  type="application/json"
  class="js-react-on-rails-component"
  data-component-name="MyComponent"
  data-trace="true"
  data-dom-id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410"
>
  …
</script>

Turbo is injecting markup into the already fully rendered page and none of the DOM events React on Rails is hooking into to execute the JavaScript to mount React components on the page are being fired.

Options​

React on Rails already has support for Turbo Streams, but only as part of their paid pro plan. Going "pro" wasn't an option for the project as they are on a tight budget and only using small parts of React on Rails in isolated areas of their app. So we're not really qualifying for "pro" here…

Server-side rendering the React components would also have been an option to tackle the problem. But this again was out of scope for our project and we did not want to run JavaScript on the server.

So what now?

JS to the rescue​

With the help of some AI coding agents, we explored the internals of the react-on-rails and @hotwired/turbo-rails NPM package to see what was actually happening. The rough idea was that, if any events were emitted by Turbo when Stream elements were injected and rendered, we could hook into them and then manually call React on Rails to mount all the newly inserted components on the page.

In Turbo, we found the StreamElement. It has a render function and a target pointing to the ID of the DOM-element it's rendered to. Combining these two things, the plan was to extend the render function to also take care of mounting any React elements within the updated target element. This means, when the StreamElement contained any React components, we want to make sure they are correctly mounted after the rendering of the StreamElement itself was done.

/**
 * Gets the current target (an element ID) to which the result will
 * be rendered.
 */
get target() {
  return this.getAttribute("target")
}

So we wrote our own tiny JS module to patch Turbo with some additional logic for our React on Rails integration:

// First we get hold of the original render function of Turbo's StreamElement
const originalRender = Turbo.StreamElement.prototype.render;

// Then we override it to extend it with our own additional logic:
Turbo.StreamElement.prototype.render = function () {
  // First we call the original render function
  originalRender.call(this);

  // Then we get hold of the target element
  const target = document.getElementById(this.getAttribute("target"));

  // Then we wait for the next frame, to make sure that the DOM is fully loaded
  // and we call a function to have React on Rails mount all components within
  // the newly inserted DOM snippet.
  if (target) {
    requestAnimationFrame(() => mountReactOnRailsComponents(target));
  }
};

The custom mountReactOnRailsComponents function is pretty much a manual re-implementation of what's happening when React on Rails is client-side rendering components:

function mountReactOnRailsComponents(domNode) {
  domNode
    // First we get all React on Rails components within our node
    .querySelectorAll("script.js-react-on-rails-component")
    .forEach((script) => {
      // Next we get the name of the component and the DOM ID of the placeholder
      const name = script.dataset.componentName;
      const { domId } = script.dataset;
      const shouldHydrate = script.dataset.hydrate === "true";

      // Then we put together the component props
      let props = {};
      try {
        props = JSON.parse(script.textContent || "{}");
      } catch (error) {
        console.error("ReactOnRails props JSON parse error for", name, error);
        return;
      }

      // Now we need the placeholder element
      const placeholder = document.getElementById(domId);
      if (!placeholder) return;

      // We check whether the component was already mounted
      // in this case we don't want to re-render it.
      const hasMounted = placeholder.dataset.reactOnRailsMounted === "true";
      if (hasMounted) return;

      // Then we call React on Rails' render function to mount the component
      ReactOnRails.render(name, props, domId, shouldHydrate);

      // Finally, we mark the placeholder as mounted, so that we don't accidentally
      // re-render it.
      placeholder.dataset.reactOnRailsMounted = "true";
    });
}

With this in place, our Turbo Streams based pagination works like charm. Any React component newly added to the DOM is mounted as normal 🎉.

Conclusion​

Drilling around like this in the internals of Turbo and React on Rails of course yields its risks. Now we're maintaining additional code which may break any time we update our Turbo or React on Rails dependences. But as we're only using the module in a small isolated area of the application, it's a risk we can accept for now. Our test coverage of the feature will inform us early enough.

Using open source technologies, we have the chance to peak under the hood of the tools and can if necessary adapt them to our needs. And working with legacy codebases and legacy Rails apps, their may be situations with unconventional needs…

We hope this post was helpful and inspires you to explore the details of the tools you're using the next time you hit a wall with them. Migrating to an entirely different tool is often a way bigger operation, especially in large monoliths.

Background​

Tickety-Tick is on the nichier side of tech within our fleet of open source projects: it's a browser extension for establishing conventions on naming branches and writing commit messages in Git. And while we're slightly leaning towards team Firefox among us developers, Tickety-Tick is of course also supporting Google Chrome (and others…). For this, we as developers have to be able to run a local build of the extension in Chrome during development.

The old way​

For local development, we had a yarn open:chrome command which would open Chrome with the current local build of the extension already installed, ready to click around. Under the hood, we utilized Google's chrome-launcher package to start a fresh, unpersonalized instance of Chrome for us. The code looked about like this:

const chromeFlags = ChromeLauncher.Launcher.defaultFlags()
  .filter((flag) => flag !== "--disable-extensions")
  .concat(["--no-default-browser-check", "--load-extension=/dist/chrome"]);

ChromeLauncher.launch({chromeFlags}).then((chrome) => console.log("Chrome running")

The important part is the --load-extension flag which points to the local Chrome build of the extension.

Some day, this just stopped working. Chrome would start as usual, but without the browser extension loaded.

What happened?​

Some debugging and internet research brought us to this post by the Chrome team from March 2025:

RFC: Removing the --load-extension flag in branded Chrome builds

Turns out the --load-extension flag was commonly abused by malicious parties and the Chrome team was looking into ways to further restrict this. It lead them to removing this feature from all branded Chrome builds (so Chromium would still have it…) and instead promote other means for loading extensions for testing and development.

What now?​

To figure out the new™ way to load an unpacked, locally build extension in Chrome, we just naively searched for the --load-extension flag on GitHub, hoping to find hints how other people updated their code. A promising hit was in the chrome-launcher package itself. If somebody knew how to do it, they should be the ones. The searching pointed us to a really helpful code comment in their tests:

// Note: --load-extension in chromeFlags used to be the primary method of
// loading extensions, but this is removed from official stable Chrome builds
// starting from Chrome 137. This shows the officially supported way to load
// extensions, with --remote-debugging-pipe.
// See: "Removing the `--load-extension` flag in branded Chrome builds"
// https://groups.google.com/a/chromium.org/g/chromium-extensions/c/aEHdhDZ-V0E/m/UWP4-k32AgAJ

As this was in a load-extension-test.ts file, we figured we'd just try to adopt what they are doing in their test setup for our open:chrome script. And as the comment indicates, we need to utilize the --remote-debugging-pipe flag, something we already knew from ChromicPDF our PDF rendering library for Elixir based on headless Chrome.

Following pretty much exactly what the loadExtension via remote-debugging-pipe test was doing, we ended up with an updated open-in-chrome script which would:

1. Start Chrome with the respective flags

   const chromeFlags = launcher.Launcher.defaultFlags()
     .filter((flag) => flag !== "--disable-extensions")
     .concat([
       "--remote-debugging-pipe",
       "--enable-unsafe-extension-debugging",
       "--no-first-run",
       "--no-default-browser-check",
     ]);
   
   const options = {
     chromeFlags,
     ignoreDefaultFlags: true,
     startingUrl: url,
   };
   
   const chrome = await launcher.launch(options);
   

2. Get hold of the debugging pipes

   const pipes = chrome.remoteDebuggingPipes;
   if (!pipes) {
     throw new Error("Chrome did not expose remoteDebuggingPipes");
   }
   

3. Put together the command to load the extension

   const requestId = Math.floor(Math.random() * 1e6);
   const request = {
     id: requestId,
     method: "Extensions.loadUnpacked",
     params: { path: dir },
   };
   

4. Send the request and listen on the pipes

   const firstResponse = new Promise((resolve, reject) => {
     let buffer = "";
   
     pipes.incoming.on("error", reject);
     pipes.incoming.on("close", () =>
       reject(new Error("Pipe closed before response")),
     );
   
     pipes.incoming.on("data", (chunk) => {
       buffer += chunk;
       let end;
       while ((end = buffer.indexOf("\x00")) !== -1) {
         const message = buffer.slice(0, end);
         buffer = buffer.slice(end + 1);
         try {
           const parsed = JSON.parse(message);
           if (parsed.id === requestId) {
             resolve(parsed);
           }
         } catch {
           // ignore non-JSON noise
         }
       }
     });
   });
   
   pipes.outgoing.write(JSON.stringify(request) + "\x00");
   

5. Check that the request was successful

   const response = await firstResponse;
   if (response.error) {
     throw new Error(`Failed to load extension: ${response.error.message}`);
   }