Whenever I talk about great software tooling with colleagues, Golang always comes up. It is a fantastic language for building tools, has a great ecosystem, but above all, it has the best developer experience I have ever seen.
Go's compiler is lightning fast, produces a single production ready binary and has out of the box support for cross-compilation. This means that you can build a binary for any platform, even if you are developing on a different one. For example, you can build a Windows executable on a Linux machine without needing to set up a Windows environment.

## But, it's not _all_ sunshine and rainbows.

Go has a feature called CGO, which allows you to incorporate raw C code into your Go projects. This is great for performance critical code or to use existing C libraries (for UI, rendering, or whatever).
```go
package main
 
/*
int add(int a, int b) {
    return a + b;
}
*/
import "C" // any comments above this line will be treated as C code
import "fmt"
 
func main() {
	result := C.add(3, 7)
	fmt.Println("3 + 7 =", result)
}
```

Though CGO is powerful, it does start to break the cross-compilation story.
This is because the Go toolchain isn't capable of compiling C on its own, it relies on the system's C compiler (like gcc or clang) to do the heavy lifting.
The exact compiler it calls under the hood also depends on the platform you're building for, because then you also need to think about header files, libraries, and other dependencies that might be different across platforms.

The biggest culprit for this is macOS, which uses the clang compiler and has a very different set of libraries and headers compared to Linux or Windows. Getting these macOS headers is also a bit of a pain, because they can only be obtained by installing Xcode, which only runs on Mac hardware. This means that if you want to build a CGO project for macOS, you need access to a Mac, which can be a significant barrier for developers who primarily work on other platforms.

![Toolchain comparison](/blog/articles/goomba/toolchain.svg?fl&w=50)

And because of all that, the fantasy of "write once, build anywhere" that Go promises starts to crumble. You either need to set up a complex cross-compilation environment, or you need access to the target platform's hardware and toolchain. Neither are feasible for small teams or individual developers who want to build tools for multiple platforms.

## ZIG building what GO could not

To solve this problem, I turned to Zig, a newer programming language that has been gaining popularity for its focus on simplicity, performance, and cross-compilation.
Zig has a built-in C compiler that can compile C code for any platform without needing to rely on the system's C compiler, and it has first class support for cross-compilation.

What's neat is that they expose their C compiler directly, without actually needing to write any Zig code. It's mostly a drop-in replacement for gcc/clang.
And best of all, _it's fully portable_ and doesn't require any platform specific setup.

### How `zig cc` actually works

One of the lesser-known features of Zig is that it ships its own fully self-contained C and C++ compiler, and you can use it without writing a single line of Zig. Just point `CC=zig cc` and `CXX=zig c++` at it, and it slots in wherever `gcc` or `clang` would normally go, including as the C compiler that CGO delegates to.

What makes this actually work rather than just being a cute trick is what's bundled inside the Zig distribution: a full copy of LLVM, plus pre-packaged `libc` implementations for every major target (musl for Linux, MinGW for Windows, and Apple's libSystem stubs for macOS). It doesn't reach for the host system's compiler or headers at all. The entire toolchain is self-contained and reproducible, which means the same `zig cc` invocation on a GitHub Actions Ubuntu runner produces the exact same object files as on your MacBook.

For CGO specifically, this means you can set `GOOS=windows GOARCH=amd64 CC="zig cc -target x86_64-windows-gnu"` and Go will happily hand off the C compilation to Zig, which already knows what it's doing for that target without you installing MinGW or anything else. That combination, Go's cross-compilation for the Go side and Zig's for the C side, is the core of what Goomba is built on.

It's still a bit of work to set up, and doesn't fully solve the macOS header problem on its own, but it's a very solid start.

## Solving the macOS header problem

The macOS header problem is the last real blocker once Zig handles the compiler side. Apple only ships the SDK through Xcode, which means officially you need Mac hardware to get the headers. Goomba solves this by pre-packaging the SDK once, on a Mac, and embedding the result directly into the toolchain binary using Go's `embed` package.

The packaging step uses `xcrun` to locate the active SDK, then extracts only what's actually needed for building: `.h` header files and `.tbd` text-based stub files, which describe framework symbols without shipping actual binaries. Private headers, module maps, and documentation are excluded as to not step on Apple's toes. The result gets zipped, where Go's `//go:embed` directive picks it up at compile time. None of the SDK ever reaches the actual Goomba repository, it's just a blob of data that gets unpacked at build time.

At build time, Goomba detects when a macOS target is requested, extracts only the headers and stubs relevant to that specific build into a temp directory, creates the symlinks that Clang and Zig expect (the SDK has a particular directory structure it's picky about), and sets the appropriate sysroot flags so the compiler knows where to look. When the build finishes, the temp directory is cleaned up.

It's worth being transparent that this approach lives in a legal gray area. The macOS SDK EULA restricts redistribution, so Goomba is better understood as a workflow for teams who already have legitimate Xcode access on at least one machine, not a way to bypass that requirement entirely. The Mac is still in the loop, it's just only needed once to generate the embedded SDK rather than for every developer and every CI run.

In practice, this is exactly why Goomba's own release pipeline runs its first job on a macOS runner to generate the embedded SDK once, so that every subsequent build job, and every user of Goomba downstream, doesn't have to.

## Architecting our new toolchain

I'm currently working on another project that I'm planning to write about in the future, which needs specific Go libraries embedded through [The Java Native Interface](https://en.wikipedia.org/wiki/Java_Native_Interface). Because some of our developers run newer MacBooks with Apple Silicon, and others run Windows or Linux machines, we need a reliable way for all of our builds to embed the same libraries without needing everyone to spin up their own binaries every time.
So the need for a clean solution was pretty urgent.

The original plan was to either:
- Build a massive Docker image with compilers for Windows and Linux, and a dedicated macOS runner from a Mac mini in the office
- Use a cloud CI service that supports macOS builds, and have it build the macOS binaries for us
- Use a cross-compilation toolchain like osxcross to build the macOS binaries on Linux (which may have worked, but I was unable to get it running with Go, Zig, and XCC together)
- Eat crap and die

I had some time to think about this during my 7 hour flight from the Netherlands to the US, and I came up with a much better fitting solution. Instead of trying to shoehorn Docker or cloud CI into our workflow, I decided to build a custom toolchain that uses Zig to compile the C code for all platforms, and then use Go's cross-compilation capabilities to build the final binaries. Missing headers get sourced during build time and embedded temporarily using Go's `embed` package, which is a neat feature that allows you to include files in your Go binary at compile time.

So, ready to commit various war crimes against software architecture, I set out to build this new toolchain, which I lovingly named "Goomba" (because it's a Go toolchain that stomps on the problems of CGO).

![Toolchain comparison](/blog/articles/goomba/goomba-flow.svg?fr&w=45)

## Goomba in action

Goomba is still a work in progress, but it's already meaningfully transforming my JNI project into a much more manageable and portable codebase.

The simplified CI now only has to pull in the latest version of Goomba, and it will make reproducible builds for all platforms without needing to install Go, Zig, any other compilers, or platform frameworks.

This is a huge win, and fills a gap in the Go ecosystem that hasn't been properly addressed for years. While Goomba does live in a bit of a gray area (the way it resolves needed dependencies is a bit hacky, and goes through some gnarly workarounds to get macOS builds working), it is a huge step forward for our project and has already saved us a ton of time and headaches.

![Goomba build example, targeting all platforms/architectures from one command](/blog/articles/goomba/cli.png)

## Side tangent: testing

This project also gave me the opportunity to play with cross-platform testing, using [Blacksmith](https://blacksmith.sh/) as a test runner for all platforms. Our testing story consists of unit tests using Go's built-in testing framework, but our project also tests a fresh build of Goomba from the current commit and tries to build an example project with CGO. After that, it spins up 3 virtual machines (one for each platform) and runs the resulting binaries to make sure they work as expected. This is a great way to catch any platform specific issues early on, and ensures that our toolchain is working correctly across all platforms.

![Test output showing successful builds and test runs on all platforms](/blog/articles/goomba/testing.png)

## Conclusion and release

As of writing this, Goomba is still in its infancy, but the fully working 1.0 release is public and fully open source on GitHub. I plan to continue improving it and adding features as needed, but I'm already pretty happy with how it's turned out so far. If you're interested in using Goomba for your own projects, or if you have any feedback or suggestions, please feel free to check out the repository and open an issue or a pull request. I'm always happy to collaborate and improve the toolchain for everyone.

If you like this kind of public work, or this project helped you out, please consider leaving a star on the [GitHub repo](https://github.com/pixelib/Goomba), or sharing this article with your friends and colleagues. It really helps to get the word out and support open source projects like this.




Goomba: Portable hermetic CGO build orchestrations for PC, Mac and Linux

Predicting the future is hard, if not impossible.
Yet, users expect it constantly.

Whether they are waiting for a package delivery, their turn on a rollercoaster, or their food order at a busy restaurant, they want to know *when*. We all have better things to do than to wait.

So I decided to build one. And along the way, I learned why it is so damn hard.

# The setup: a chaotic queue
Tubbo (a popular Minecraft streamer) recently launched his own server called realmSMP. With millions of followers, the launch attracted way more players than any server could handle.

When I tried to join, I was greeted with a queue system with hundreds of players waiting their turn, each with nothing but a number: *your current spot in line* and *the total queue size*.

That is it. No estimated wait time. No progress bar. Just raw numbers.

And that is when I went: "Surely, with just those two numbers over time, I can predict how long this will take?"

# Why the obvious solution does not work
At first glance, queue prediction seems trivial. Just measure how fast people are leaving, and divide your position by that speed:

```latex
\[
\text{Wait Time} = \frac{\text{Your Position}}{\text{Spots per Minute}}
\]
```

If you are at position 100 and people leave at 5 per minute, you will wait 20 minutes. Easy, right?

This works great when the queue moves at a constant speed, like a theme park ride where guests board every 2 minutes like clockwork.

But the realmSMP queue had a problem: **chaos**.

# The chaos factors
Two things make this queue unpredictable:

**1. Priority players can cut the line**
Some players paid for priority access, which means they skip ahead. When that happens, your position can actually *increase*. You are 50th in line, then suddenly 53rd, even though you did not move.

**2. There is no forced session limit**
Unlike theme parks where rides have fixed durations, Minecraft players can stay as long as they want. If a content creator is streaming, players stick around for hours. If it is dinnertime in Europe, suddenly half the server logs off at once.

# The solution: be pessimistic
After a lot of trial and error, I settled on what I call **the pessimist's approach**.

Instead of trying to predict the "correct" speed, I calculate *multiple* speed estimates and always use the slowest one.

| Speed Type | What it measures |
|------------|------------------|
| **Recent speed** | How fast the queue moved in the last 5 minutes |
| **Session speed** | Your average since joining the queue |
| **Historical speed** | Average from previous complete runs |

Then I pick whichever gives the *longest* estimate.

Why? Because **under-promising beats over-promising**. If I say "20 minutes" and you get in after 15, you are pleasantly surprised. If I say "10 minutes" and it takes 25, you are frustrated.

See how the algorithm picks between different speed measurements:

```react-demo
VelocityTrackingDemo
```

# Finding the trend, not just the average
Here is where it gets interesting. A simple average tells you "the queue moved at 3 spots per minute", but it cannot tell you if things are *speeding up* or *slowing down*.

That matters! If the queue is accelerating (people leaving faster and faster), you will probably get in sooner. If it is decelerating, brace yourself for a longer wait.

To detect trends, we use something called **linear regression**, which basically finds the best straight line through a bunch of messy data points.

The demo below tries to visualize this, by showing the anticipated time (green line) powered by linear regression, which turns out to be a pretty solid prediction of where the average line would end up.

```react-demo
LinearRegressionDemo
```

# Real data: what does chaos look like?
Here is what actual queue data from realmSMP looked like over a typical day:

![Queue position over time from production data](/blog/articles/queue-prediction/queue_data.png)

You can see the patterns: periods of steady movement, sudden jumps when priority players cut in, and moments where everything stalls completely.

The velocity graph is even more telling:

![Velocity patterns showing queue speed variations](/blog/articles/queue-prediction/velocity_data.png)

Those spikes? Moments when a bunch of players suddenly logged off (probably dinner in Europe). The flat sections are harder to explain. This could show priority players overwhelming the queue, or a "stable" period where the queue growth is equal to the throughput.

# The feedback loop
The beautiful part of this system is that it learns from itself.

Every time we complete a full queue run (joining to reaching the front), we record exactly how long it took. That becomes "historical data" that helps future predictions.

Early estimates are rough guesses. But after a few full runs through the queue, the system has *real* data about how this queue actually behaves, and the predictions get remarkably accurate.

# What I learned
Building this taught me a few things about prediction in general:

1. **Users prefer pessimistic estimates.** A comfortable overestimate beats a stressful underestimate, every time.

2. **Trends matter more than averages.** Knowing if things are getting better or worse is more useful than knowing the current speed.

3. **Real-world data is messy.** Textbook examples use clean data. Reality has priority players, dinner time, and streamers.

4. **Self-improving systems are powerful.** Feeding real outcomes back into the model makes predictions better over time, without any fancy machine learning.

The fundamental insight is that queue prediction is not about being *right*. It is about being *useful*. A prediction that is consistently slightly pessimistic is more valuable than one that oscillates wildly.

# Theme parks do this too
If you have ever wondered how theme parks calculate wait times, it is the same principle with different inputs.

Disney asks guests to carry tracking cards through the queue. They measure *actual* wait times and update the signs accordingly. Some parks use weight sensors on queue barriers, or cameras counting people.


![Disney card](/blog/articles/queue-prediction/disney-card.png)

The key insight: they are not predicting the future. They are measuring the recent past and assuming it continues.

That is really all we can do. The future is unknowable. The best we can offer is an educated guess based on what just happened, and enough padding to make sure we do not disappoint.

# The result: when-can-i-join-the-realmsmp.net

All of this math and logic culminated in a public tool: [when-can-i-join-the-realmsmp.net](https://when-can-i-join-the-realmsmp.net/).

![Screenshot of the queue prediction website](/blog/articles/queue-prediction/final_tool.png)

It is a free website where players can see real-time estimates and historical data to find the best time to join. Since its release, it has been used by over 2,000 players to plan their sessions.

It even got a shoutout from Tubbo himself on stream!

[Watch the clip here](https://www.twitch.tv/tubbo/clip/ConfidentGloriousChickenDancingBaby-0cwpMdC-r3W59bLA)

If you are stuck in a queue right now, give it a try. At least you will know exactly how long you have to wait.


The magic of time prediction


# Introduction
Recently I launched a personal project, [the Mojang blacklist check](/blog/mojang-blocklist-deepdive), expecting that a few people might use it. But to my surprise, it quickly gained significant traction, with over 1,000 daily users within the first week.

However, this sudden popularity brought an unexpected challenge: the costs associated with Cloudflare D1 started to skyrocket.

# The Cost with Cloudflare D1
Cloudflare D1's pricing is *extremely* generous, with the paid plan including 25 billion monthly rows (read) and 50 million monthly writes for just $5. When I first started the blacklist tool project, I looked at these numbers and thought, "There's no way I'll ever hit those."

But, as usual, the law of exponential numbers once again bit me in the rear.

Because every time the tool "refreshes", it needs to do a full DIFF against the Mojang blacklist, which is about 2,500 rows. Do this every 15 minutes, and you're already up to 7.2 million reads per month.

This doesn't even include the actual usage of the tool, which has the same terrible scaling characteristics. Every time the page loads, it compiles a full history to seed the user with context. And on top of that, every time a new test is done, a handful of variants are generated (to check for subdomains, evasion, DNS info, etc.), and each of these variants needs to be hashed and then checked against the database.

And before you know it, you're hit with this:
![CF bill](/blog/articles/hyperdrive/bill.png)

Sure, it's not a massive amount of money, but for a student like me, it's a lot to throw at a free project.

# Cloudflare Comes to the Rescue... Again
Who's better at solving Cloudflare problems than Cloudflare themselves? Of course, they would love for you to stay in their ecosystem (and I'm happy to oblige because their stuff is great).

You can already open a manual TCP connection from a worker, letting you implement any protocol you want, and thanks to some amazing OSS work by Brianc, we have [node-postgres](https://github.com/brianc/node-postgres), which has native support for Cloudflare's TCP sockets.

There's not really much more you need than this, because from here on out, it's just regular Postgres. But you also lose a lot of the "edge" benefits of D1. The thing that makes D1 so special is that it's a distributed database, meaning that your data is replicated across all of Cloudflare's data centers. This means that no matter where your user is located, they will always have a low-latency connection to the database.

Taking control of your own database means that the latency between the worker and database will be a lot higher, which becomes more noticeable for every query (because of the cumulative round-trip time).

Thankfully, Cloudflare lets us have the best of both worlds. Enter [Cloudflare Hyperdrive](https://developers.cloudflare.com/hyperdrive/).

# What is Hyperdrive?
Hyperdrive looks like a regular tunnel/proxy service on the surface, but it's actually a **lot** cooler than that. It will man-in-the-middle your Postgres connection, and not just forward the commands, but understand the queries being done and cache them for you at the edge. This means that if you have a query that is run often, it will be cached at the edge, and subsequent requests will be served from the edge cache instead of going all the way to your database.

![Hyperdrive diagram from https://developers.cloudflare.com/hyperdrive/configuration/how-hyperdrive-works/](/blog/articles/hyperdrive/hyperdrive.png)

This means that you can have a single database in a single region (in my case, on a cheap Hetzner VPS) and still have low-latency access to it from all over the world. And should Hyperdrive get too expensive, you can always fall back to your database directly, albeit with higher latency. Not the best user experience, but a worthy trade-off for cost savings.

# Migrating from D1 to Hyperdrive (SQLite to Postgres)
Migrating from D1 to Postgres + Hyperdrive is pretty straightforward, but does come with a major gotcha.

Though Cloudflare will happily let you export your data as a SQLite script [through Wrangler](https://developers.cloudflare.com/d1/best-practices/import-export-data/), there are some architectural problems you might run into.

The most prominent one is that SQLite doesn't give a single hoot about capitalization in table and column names, while Postgres does. This, combined with `pgloader` being a bit finicky with non-standard table names, makes for a great recipe to waste your entire afternoon trying to figure out why your queries don't work.

To avoid these issues, I recommend the following steps:

1. Export your D1 database as a SQLite script using Wrangler.
    - `npx wrangler d1 export <database_name> --remote --output=./database.sql`
2. Port the SQLite script to a "fat" SQLite database using the [sqlite3 CLI](https://www.sqlite.org/cli.html).
    - `sqlite3 fat_database.db < database.sql`
3. Make a pgloader script to port the SQLite database to Postgres, *including* casing for table and column names.
    - Example pgloader script:
```lisp
LOAD DATABASE
FROM sqlite://fat_database.db
INTO postgresql://USER:PASSWORD@SERVER/DB

WITH include no drop, create tables, quote identifiers
;
```
4. Run pgloader with the script.
    - `pgloader migrate.pgloader`
5. Manually go through your queries to address some inconsistencies between SQLite and Postgres.
    - Make sure to quote table and column names with double quotes if they contain uppercase letters or special characters. Otherwise, Postgres will silently lowercase them, leading to "table not found" errors.
    - If you're using Prisma, make sure to update your `number` fields to `BigInt`, as Postgres `integer` fields are 4 bytes, while SQLite `integer` fields are 8 bytes.

# Conclusion
The migration from D1 to Hyperdrive is now complete, and I'm cautiously optimistic about the results. While the upfront complexity of managing my own Postgres instance and dealing with SQLite-to-Postgres quirks was definitely a time investment, the potential cost savings are significant (expecting continued growth in usage).

Of course, the real test will come over the next month as I collect actual usage data and see how Hyperdrive performs under real-world conditions. Will the edge caching live up to the promise? Will the latency remain acceptable? And most importantly, will my wallet thank me?

I'll be monitoring the costs closely and will follow up with a detailed comparison once I have meaningful data. For now, if you're running into similar scaling issues with D1, Hyperdrive is definitely worth considering; just be prepared to spend an afternoon wrestling with case-sensitive table names.

Stay tuned for the results, and should my projects go down in flames, at least I'll have a good story to tell!


Migrating from Cloudflare D1 to Hyperdrive


# Introduction

> This article goes hand-in-hand with the [Mojang blacklist check tool](/projects/mojang-blacklist-checker). If you want to skip the deep dive and just use the tool, go ahead!

Minecraft brings millions of players together every day in its thriving multiplayer scene. Anyone can spin up a server and build their own virtual world, complete with custom content if they have the coding skills.

But running a popular server isn't free. There are hosting costs, freelance developers and artists to hire, and the time investment of managing a team. Most server owners solve this by selling in-game perks for real money, turning their hobby into a business.

This usually creates a win-win situation. Server owners can expand their projects beyond what they could do alone, while players enjoy richer gaming experiences and support communities they love.

Unfortunately, not everyone plays fair. The law of large numbers guarantees that some communities will have shady or outright bad actors.

# The shady side of Minecraft

Just because you can bring your wildest imagination to life in a game doesn't mean you should. But some people genuinely think otherwise.

The ability to monetize creative content attracts those who aren't interested in maintaining that win-win balance. There will always be parties trying to tip the scales in their favor. Mojang introduced mechanisms to fight back as early as 2016.

Minecraft has rules outlined in their EULA (End User License Agreement) that cover how players and server owners can legally use, distribute, and monetize the game. The main goal isn't to suppress creativity but to protect it, ensuring Minecraft remains fair and safe for everyone. Mojang doesn't want servers becoming pay-to-win factories or shady marketplaces where players buy their way to power. You can sell access, cosmetics, or perks, but selling gameplay advantages or anything that distorts the core player experience is strictly forbidden.

Beyond just gameplay fairness, Mojang has a responsibility to protect their intellectual property and maintain Minecraft's reputation. When servers exploit the game in harmful ways, whether through predatory monetization or inappropriate content, it reflects poorly on the Minecraft brand as a whole. Parents need to trust that when their kids join a Minecraft server, they're entering a safe environment.

To enforce this, Mojang introduced systems that can block servers violating these terms. If your server breaks the EULA, say by selling overpowered weapons or locking key features behind paywalls, it can get blacklisted. Players can't join unless they disable certain checks, which most won't bother doing.

# EULA blocks and enforcement

The blacklist system has one characteristic that frustrates many: limited public transparency.

When a server gets blacklisted, players aren't told what happened. They just see a generic connection error, leaving them confused about what went wrong. The criteria Mojang uses to determine whether a server is a risk remain largely undisclosed, and nobody quite knows where the line is drawn.

The reporting mechanism adds another layer of complexity. Anyone can report servers directly through Mojang's form, opening the door to conflicts of interest. Competitors, disgruntled players, or bad actors can report any server for any reason, real or fabricated. There have been recent cases where this system may have been abused through faked evidence.

On the other hand, Mojang sometimes proactively reaches out to server owners based on their own findings, perhaps from spotting issues on public webstores. In these cases, they generally provide feedback and warnings in advance, giving owners a chance to correct problematic behavior.

Here's the thing though: Mojang's enforcement team isn't a PR department. Their job is to protect the Minecraft brand and ensure player safety, not to manage public opinion or provide detailed explanations for every decision. They don't need community approval to do their job effectively. Being transparent about every enforcement decision could actually undermine their efforts, giving bad actors a roadmap to avoid detection or turning blocklists into unintended advertising.

From a purely practical standpoint, they're doing what they need to do. The system works, even if it's not perfect. But the lack of visibility does create an information vacuum, and when people don't understand what's happening or why, speculation and misinformation rush in to fill that space.

# Keeping track of enforcement actions

The community picks up where publishers drop the ball. This has been the case forever for Minecraft. Don't have an official modding API? Create your own (Bukkit). Not happy with game performance? Install performance mods. And frankly, I could keep going for ages.

Minecraft needs to fetch a list of blocked servers from somewhere to know what to allow. The [@BlockedMCServers](https://x.com/blockedservers?lang=en) Twitter/X account (by [Jackson](https://x.com/minecoder) and [Marvin](https://x.com/xBytezz)) had been the place to keep track of Mojang's enforcement actions since the feature was added. It would post hashes (because Mojang doesn't publish blocked servers in human readable form) and known hostnames within minutes, making it both a spicy drama source and a good way to stay informed. However, this project shut down in early 2025 with no clear successor.

![The old X/Twitter account](/blog/articles/mojang-blacklist/old-twitter.png)

# The challenge of mapping servers

Mojang doesn't keep a public list of blocked servers, only a list of hashes.

When you try to join a server, Minecraft calculates a few hashes for the connecting hostname and checks if any match Mojang's blocked list. Only then does it cause a vague client error.

Think of a hash as a one-way mathematical function that turns any text into a fixed length string of seemingly random characters. Mojang uses SHA-1, which always produces a 40 character hexadecimal output regardless of input length. For example, the server address "mc.example.com" might become "5303a76c19617a55ae2c2102319038f225fcc328."

Here's the catch: you can easily go from "mc.example.com" to its hash, but you cannot reverse the process. There's no mathematical way to take that hash and figure out the original server address. It's like putting something through a meat grinder. You can't un-grind it back into its original form.

This is exactly why Mojang chose this approach. If they published a readable list of blocked servers, it would be free advertising for the very communities they're trying to discourage. With hashes, the blocked list remains public (so the game can check it), but the actual server addresses stay hidden.

The only way to discover what a hash represents is through brute force: generating hashes for thousands of possible server addresses and hoping one matches. This makes maintaining a comprehensive mapping incredibly challenging. You need either a massive database of known servers to check against, or you need people to actively search for specific addresses, revealing matches one by one.

(the old Twitter account probably used a database where people would add their own translations from hashes to hostnames through crowdsourcing)

# Building a new tool

This project came back onto my radar after some developments in the Dutch Minecraft server space. I wanted a place to keep track of what was happening, when, and how it developed over time. This eventually turned into my creatively named "Mojang blacklist check tool."

It works by periodically checking the list of hashes from Mojang and tracking additions and removals. This builds a version history timeline where we can not only check if a server is (or has been) blacklisted, but also see when it was unbanned, banned again, or changed to a broader address.

![Activity timeline screenshot](/blog/articles/mojang-blacklist/activity.png)

This is fun, but showing random strings isn't very useful. That's where the mapper comes in.

Every time someone searches for a hostname, I check if it's already a known hash and gather its recent history if so. If it's unknown, I generate a dozen varieties and check their hashes. If any match, we've uncovered a new entry that can be saved and shown to everyone. The list gets pieced together one search at a time.

![Timeline from the app](/blog/articles/mojang-blacklist/timeline.png)

# Community response

I didn't expect much activity at first, but the tool has seen steady traffic with over 2,000 unique visitors in the first month alone. People are genuinely interested in tracking these enforcement actions, and it's rewarding to see the community engaging with it.

Not everyone is thrilled about the increased transparency though. One server owner reached out rather aggressively, demanding I take down a post about the project where their server was called out as well. While I understand the frustration of being blacklisted, I believe there's value in the community having access to this information.

What really validated the project was when Mojang's enforcement team reached out to me out of the blue to commend it. They personally love it and even provided feedback and offered to add test entries to help verify the tool's accuracy. It's reassuring to know that the people doing the actual enforcement work see value in what the community is building.

![Email from Mojang enforcement team - I originally had the thread here, but they asked me to like, not do that.](/blog/articles/mojang-blacklist/email.png)

> I really do want to thank the Mojang enforcement team for their feedback, and special help testing this tool (they added and removed special test entries to verify accuracy, how cool is that!).

# Conclusion

This tool exists because the community wanted visibility into something that affects them, even though Mojang has no obligation to provide it. Their enforcement team has a job to do: protecting the Minecraft brand and keeping players safe. They're not there to explain every decision or manage public relations, and frankly, they shouldn't have to be.

But that doesn't mean the community can't benefit from better information. When the official channels don't provide the level of detail people want, the community builds its own solutions. That's what makes Minecraft special. It's always been this way, from mods to performance tools to enforcement tracking.

The positive response from both users and Mojang's enforcement team shows that there's room for both approaches. Mojang can focus on doing their job effectively without the burden of constant public explanation, while community tools like this one can help fill the information gap for those who want to understand what's happening.

At the end of the day, everyone wants the same thing: a safer, fairer Minecraft ecosystem. Mojang enforces that through their systems, and tools like this help the community understand and navigate those systems. It's not about demanding transparency from those who don't owe it. It's about the community taking initiative to inform itself, as it always has.


Keeping Track of Mojang's Enforcement Actions

Hacking the housing market

**NS**.
There are two possible reactions to reading that.
Someone who hasn't spent a lot of time in the Netherlands might say *bless you*, but a fellow Dutchie will get shivers down their spine.

For those who don't know, 'Nederlandse Spoorwegen' (also known as NS—kudos if you already figured that out) is the main railway operator in the Netherlands. While they operate train services and manage timetables, they share responsibilities with other entities: ProRail manages the actual rail infrastructure and stations, while regional operators handle some local lines.

What's far more interesting (to me) is that they also own and operate extensive IT infrastructure. While they do have some official public APIs, these require permission requests and expose only a limited subset of their data. Meanwhile, their internal tooling often ends up accessible on the public internet—sometimes without authentication requirements.

You can see where this is going. We're going to have some fun with data exfiltration!


# Solving my personal pain points
Every project needs a goal. And the mission I set out for myself is to make my frequent and long commutes more efficient, or at least less cumbersome.

My main gripe with long journeys during peak hours is the inability to find a seat, or being forced to miss my transfer by some vacationers who (respectfully) don't have a fucking clue what they're doing.

Previously, I already made a private service for Efteling waiting times, which has collected two years of ride queue time history and weather conditions, and can predict waiting times decently well (thanks to a hotchpotch of Python, Pandas, Influx 2, and Grafana). Something like that, but for NS, would be amazing.

Think Buienradar-like prediction, but only slightly less terrible.
> Now that's a joke that won't land at all for international readers. You're welcome.

 
# The Hunt for Data
Before we can generate anything, we need input data. And what better way to get that than their own tooling?
Unlikely to break (they haven't been updated in 5+ years), runs on their production service, and provides instant data.

> **Disclaimer**: I know I'm not being a model web citizen here by poking around internal tools, but I'm keeping things ethical—no aggressive scraping, no overloading their servers, and I won't be sharing specific endpoints or methods to keep things clean.

My favorite entry point for NS data and APIs is their internal sign builder tool, which generates the digital displays you see in stations. While I won't share specifics about how to access it, I can tell you that it provides a fascinating window into their real-time data systems. The interface lets you configure new (local only) station displays, choose content layouts, and much more. The interesting part begins when you examine how it works under the hood—everything runs client-side, including their API integrations, all in readable, unminified JavaScript that essentially serves as live documentation!

![A very real sign for Delft, running on my iPad](/blog/articles/ns/signbuilder.jpg)

The data it's processing looks a lot like this:
```json
{
  "departures": [
    {
      "direction": "Schiphol Airport ✈",
      "name": "NS  3589",
      "plannedDateTime": "2025-07-17T23:07:00+0200",
      "plannedTimeZoneOffset": 120,
      "actualDateTime": "2025-07-17T23:07:00+0200",
      "actualTimeZoneOffset": 120,
      "plannedTrack": "1",
      "actualTrack": "1",
      "product": {
        "number": "3589",
        "categoryCode": "IC",
        "shortCategoryName": "NS Intercity",
        "longCategoryName": "Intercity",
        "operatorName": "NS",
        "operatorCode": "NS",
        "type": "TRAIN"
      },
      "trainCategory": "IC",
      "cancelled": false,
      "routeStations": [
        {
          "uicCode": "8400280",
          "mediumName": "Den Haag HS"
        },
        {
          "uicCode": "8400380",
          "mediumName": "Laan v NOI"
        },
        {
          "uicCode": "8400390",
          "mediumName": "Leiden C."
        }
      ],
      "messages": [],
      "departureStatus": "ON_STATION"
    }
  ]
}
```


This gives us a treasure trove of information, which will be useful for future projects. But what we're most interested in right now is the product number, which is a unique ID for every currently in-use train.

If we do these requests for a handful of hub stations, process and remove duplicates, we end up with a reliable list of currently deployed hardware.

We can then take these IDs and use another API to ask for more specific details, like an estimated occupancy metric.

> Sidenote: The way they do their occupancy estimation is actually really interesting. Though it's true that this data is crowdsourced to some extent, [they primarily get the data from the train's weight](https://tweakers.net/nieuws/137243/ns-app-toont-hoe-druk-het-per-wagon-is-in-trein-op-traject-arnhem-den-bosch.html#:~:text=De%20NS%20brengt%20de%20drukte,het%20is%20in%20bepaalde%20rijtuigen.), which is measured at various points.

 
# Processing Data
Now, we've got a reliable method of reading real-time information. What's next?

Well, dear reader, it's once again time to spin the **🎰🎲✨WHEEL OF SUFFERING✨🎲🎰**, where we'll be choosing our tech stack.

For this project, we'll need:
- a server-side environment to run the lovely crawling scripts
- a persistent database of some sort
- web hosting

And to run it RELIABLY, FAST, and for CHEAP (yeahhhh).

This naturally leads me to the Cloudflare stack, specifically:
- [Workers](https://workers.cloudflare.com/) for a cron-based compute environment, and to host the public REST API to serve data back to the web UI
- [D1](https://developers.cloudflare.com/d1/), an affordable SQL offering with generous free tiers
- [Pages](https://pages.cloudflare.com/), their static site hosting that's competitive with Vercel

The worker is written in TypeScript and is responsible for calling all the external APIs and ingesting the data into D1. Workers are actually the perfect fit for this, because unlike competitors (like AWS Lambda), they only bill you for CPU duration, not request wall-clock time. This is great in our case, because our script will spend most of its time waiting on external API requests (time for which I will not be billed).

After our requests, we end up with four key metrics:
- the number of trains currently in service
- and how many of them are empty, well-filled, or busy

I want to have a single score/percentage which indicates the busyness across the network, which needs to adapt well to the number of trains in service.
Such a formula is pretty easy to come up with, and I ended up with this:
```typescript
function calculateBusynessPercentage(low: number, medium: number, high: number): number {
  const totalTrains = low + medium + high;
  if (totalTrains === 0) {
    return 0; // No trains, no business
  }

  // Calculate weighted score directly
  const weightedScore = (low * 1 + medium * 2 + high * 3);
  const maxPossibleScore = totalTrains * 3; // All trains at highest utilization

  // Normalize to 0-100 scale
  return (weightedScore / maxPossibleScore) * 100;
}
```

where `low`, `medium`, and `high` are all carriages sorted by their occupancy, totaling to the total service. Each value is weighted, and then normalized by the total amount.

This isn't perfect and leads to some weird temporary spikes when a lot of trains are taken into or out of service, but it's good enough for now.

I then store this data in D1 through a cron job and call it a day.
> Yes, a time series database would be more appropriate for this use case, but D1's generous free tier and SQL familiarity make it a practical choice for this project's scale.


# Those Who Hoard Data... Have a Lot to Sort Through
Those 288 data points per day are enough to do some fun stuff with.
The web page I built for this project (in NextJS, though this part is rendered client-side) makes a request to my worker for the last 7 days' worth of data, and then visualizes it.

Then, for the real kicker, it tries to look at historical metrics from the same time frame in prior days and comes up with an educated guess of what the next hour will bring.

And thus I can finally make an educated guess if it's worth running for my train to ride before the wave of peak hours, or to ride it out and wait for things to cool down.


# BONUS: Fun with Napkin Math
Since we have all this real-time data about train occupancy and network utilization, I thought it would be interesting to do some back-of-the-envelope calculations about revenue streams. This is purely for entertainment and educational purposes—think of it as a data science exercise rather than any serious financial analysis.

As a student who's acutely aware of travel costs (especially during summer months when the free travel doesn't apply), I was curious: what does all this passenger movement actually translate to in terms of revenue? The numbers are fascinating, even if they're built on assumptions and estimates.

We have real-time metrics, we know how big trains are, and we can make an educated guess about how many students and gate-hoppers there are at a given moment.
And with that, we can also guess their real-time worth!

```javascript

const avgSeatsPerTrain = 450; // Average seats per train (source: 600 Sprinter, and we'll deduct an arbitrary 25% for people who take up 2 seats)
const avgStandingCapacity = 300; // Standing room (source: my own estimate)
const totalCapacityPerTrain = avgSeatsPerTrain + avgStandingCapacity;

// Calculate current snapshot metrics
const totalCapacity = totalTrains * totalCapacityPerTrain;
const currentPassengers = Math.round((occupancyPercent / 100) * totalCapacity);

// Revenue calculations with more nuanced pricing
const avgTicketPrice = 7.80; // Based on my personal average
const studentDiscountRate = 0.4; // 40% discount for students
const nonPayingRate = 0.12; // (source: once again, freshly pulled from my own head)
const studentRate = 0.25; // 25% students (source: you'd never guess, my own estimate)

const payingFullPrice = Math.round(currentPassengers * (1 - nonPayingRate - studentRate));
const payingStudents = Math.round(currentPassengers * studentRate);
const nonPayingPassengers = Math.round(currentPassengers * nonPayingRate);

const fullPriceRevenue = payingFullPrice * avgTicketPrice;
const studentRevenue = payingStudents * avgTicketPrice * (1 - studentDiscountRate);
const totalRevenue = fullPriceRevenue + studentRevenue;
```

Not enough to make the costs hurt less, but a funny reminder that things are indeed messed up, and well on track to get even worse next year (hope you enjoy this excellent railway pun).

# [You can explore this data yourself on this page](/projects/train-rush)
![The final result](/blog/articles/ns/screenshot.png)


Dutch Rail Data Heist: Real-Time Metrics, APIs, and Commuter Chaos

Saying goodbye to Java rebuilds, hello to JavaScript (but still in the JVM)


# The Challenge of Inter-Service Communication

In distributed systems, particularly stateful applications like large-scale game networks, robust inter-service communication is a fundamental requirement. Services, often running in separate JVMs or on different machines, must exchange data and trigger behavior on one another. For example, a lobby server may need to query a game server for player statistics, or a minigame instance might need to update a player's inventory on a central data service.

The conventional solutions to this problem involve creating and maintaining explicit communication protocols, such as REST APIs or custom message-based systems over a queue. These approaches, while effective, introduce significant overhead in both development and maintenance. Developers must write and manage API specifications, handle network logic, and often write client libraries. This boilerplate can obscure the core business logic and create a disconnect between local development environments and deployed production systems.

The primary motivation for Meteor was to abstract this complexity away. The goal was to enable developers to write code against standard Java interfaces, with the underlying communication mechanism becoming a transparent detail of the runtime environment. This allows for identical application code to run in a monolithic local environment for testing or in a distributed production environment, simply by changing a configuration setting.

## Design: Abstracting Communication Behind Interfaces

Meteor is a Java RPC (Remote Procedure Call) framework designed to make remote method invocations appear as local method calls. It achieves this by allowing developers to work with their own Java interfaces, decoupling the application logic from the networking layer.

Consider a simple `Scoreboard` interface:

```java
public interface Scoreboard {
    int getScoreForPlayer(String player);
    void setScoreForPlayer(String player, int score);
    Map<String, Integer> getAllScores();
}
```

On a service that owns the data (the "server"), a concrete implementation is registered with a Meteor instance:

```java
// The service with the authoritative state registers its implementation.
Scoreboard implementation = new ScoreboardImplementation();
meteor.registerImplementation(implementation, "parkour-leaderboard");
```

Any other service (a "client") can then obtain a proxy instance that conforms to the `Scoreboard` interface, without needing access to the implementation class:

```java
// A client service obtains a remote-controlled proxy.
Scoreboard parkourScoreboard = meteor.registerProcedure(Scoreboard.class, "parkour-leaderboard");

// This method call is transparently sent to the remote implementation.
int score = parkourScoreboard.getScoreForPlayer("Notch");
```

The `parkourScoreboard` object is a dynamically generated proxy. When its methods are invoked, Meteor intercepts the call, serializes it, sends it over a configured transport, and awaits a response from the remote service where the actual implementation is running.

## Core Architecture

The flexibility of Meteor stems from its pluggable architecture. The core library is unopinionated about the underlying mechanisms for transport and serialization, allowing them to be swapped out to fit different use cases.

-   **`RpcTransport`**: This interface defines the contract for sending and receiving invocation and result packets. The `meteor-jedis` module provides an implementation that uses Redis pub/sub for broadcasting packets, a suitable strategy for fan-out communication patterns. For local development and testing, a `LoopbackTransport` is provided, which performs all communication in-memory within a single JVM, eliminating network overhead and dependencies.
-   **`RpcSerializer`**: This interface handles the serialization of method arguments and return values. The default implementation uses Gson for JSON serialization, but custom implementations can be provided for other formats (e.g., Protocol Buffers, Kryo) to meet specific performance or compatibility requirements.
-   **Dynamic Proxies**: At its core, Meteor leverages Java's `java.lang.reflect.Proxy` to generate proxy classes at runtime. When `registerProcedure` is called, Meteor creates a new class that implements the target interface. The invocation handler for this proxy is responsible for coordinating with the `RpcSerializer` and `RpcTransport` to execute the remote call.

This design allows for a clean separation of concerns, where application code is concerned only with business logic defined in interfaces, while the framework handles the underlying communication mechanics.

## Runtime Instrumentation and Dynamic Tooling

The design philosophy of Meteor is that the framework should adapt to the application's code, not the other way around. Instead of requiring developers to extend framework-specific classes or use annotations, Meteor uses runtime instrumentation to generate the necessary components. This approach can be thought of as creating adapters from "playdough"—the framework molds itself around the existing code structure.

Here is a breakdown of the process:

1.  **Interface as Contract**: The developer provides a standard Java interface. This interface is the sole contract and remains free of any framework-specific code.
2.  **Runtime Proxy Generation**: When `meteor.registerProcedure(YourInterface.class, ...)` is invoked, Meteor's runtime dynamically creates a proxy class that implements `YourInterface`. The methods of this generated class contain the necessary instrumentation to delegate calls to the Meteor Core.
3.  **Invocation Interception**: When a method is called on the proxy object, the invocation is intercepted. The method name, arguments, and other metadata are packaged into an invocation object, which is then handed off to the serialization and transport layers.

This runtime-centric approach means the developer is not burdened with writing boilerplate networking or serialization code. The framework handles it transparently.

### Call Flow Visualization

The sequence of events for a remote procedure call can be visualized as follows:

![Meteor sequence diagram](/blog/articles/meteor/dia1.png)

### System Architecture Overview

Internally, the components of Meteor work together to provide this transparent RPC mechanism.

![Meteor architecture](/blog/articles/meteor/dia2.png)

This pluggable and dynamic nature allows Meteor to handle the complex, error-prone networking tasks, freeing developers to focus on application logic.

## Conclusion

Meteor was developed by Duco and Mats in 2023 over a period of two months. It provides a practical solution to the challenges of inter-service communication in Java applications. By leveraging runtime proxy generation and a pluggable component model, it successfully abstracts network communication behind standard Java interfaces. This approach simplifies development, improves testability by allowing local execution without code changes, and decouples business logic from the underlying transport mechanism. The result is a more maintainable and scalable system where developers can focus on their core domain without being burdened by the intricacies of distributed systems programming.

The project was also a significant milestone for us as it was our first to be accepted into the public OSS Sonatype repository. The source code is available on [GitHub](https://github.com/pixelib/meteor).


Meteor: Interfaces as networking contract

Lights are cool. That's just a fact.
You have pretty colored static ones, you have wobbly moving ones, you have ones that can do both.
You know what else has wobbly moving lights and effects? Minecraft. [ImagineFun, the Minecraft disney experience](https://imaginefun.net/) I work for, has a lot of them.

And in the past few years, we've made a ton of improvements to our workflows and pipelines to make it easier for our creative folks to make cool stuff.
But, shows are still made by people, and people are still people. And people are lazy. And people are forgetful. And people are bad at math. And people are bad at random numbers.
This doesn't mean we're bad at making shows, but it does mean that we can make even bigger ones by tapping into the decades of experience from the real Theatre and Live Event industry.
And after months of work, we've crafted a full pipeline to bring real show technology into Minecraft.

## Managing expectations
Before we dive into the technical details, let's set some expectations.
I won't be able to share any of our specific implementations, code, or secret sauce. I will, however, share the general concepts and ideas behind our work.

## What is DMX and Art-Net?
![A DMX splitter from my local student pub, taking one 3pin input, and splitting it into 6 lines for different facades](/blog/articles/ma_minecraft/img_6.png)
DMX is a protocol used in the lighting industry to control fixtures. It's a simple serial protocol that sends a stream of bytes [(RS-485)](https://en.wikipedia.org/wiki/RS-485). It's been around for a long time, and it's still widely used today. You can imagine each "command" as an array, a huge list of numbers (DMX512, has 512 usable addresses, with remaining bytes being used for the sync and frame protocols) that represent the intensity of a light, the color of a light, or any other parameter you can think of.

Each fixture has an address, which is the Nth slot in this list, from where it will read a few numbers (depending on it's parameters) and parse them as it's state.
For example, let's say that we have a simple RGB light, that has 3 channels (1: red, 2: green, 3: blue) If we send the command `[255, 0, 0]` to the light, it will turn on at full brightness and be red. If we add a second light, then it would start listening at the 4th slot, at which point our *universe* would look like this: `[255, 0, 0, 255, 0, 0]`.

You can imagine that this becomes a huge list pretty fast, mostly because most lights need far more than 3 parameters (moving lights can have 20+ channels, for all its gobos, movements, colors, etc). This is why DMX is usually split into *universes*, which are separate lists of 512 slots. This way, you can have a lot of lights, but you can also have a lot of parameters for each light. If one universe gets full, you can just overflow into the next one. It's not uncommon for this to get out of hand. [Eurovision (2024), used a whopping 663 DMX universes.](https://www.malighting.com/news/article/grandma3-for-eurovision-2024-1375/#:~:text=All%20the%20non%2Dkey%20light%20fixtures%20with%20RGB%20or%20a%20fast%20dimmer%20were%20pixel%2Dmapped%20in%20the%20media%20servers%20to%20keep%20the%20creative%20options%20as%20open%20as%20possible%2C%20so%20most%20of%20the%20outputting%20DMX%20universes%20were%20merged%20with%20the%20663%20DMX%20universes%20of%20Art%2DNet%20input.%C2%A0)

DMX is normally transferred over a 3 (or 5) pin XLR cable, but it can also be sent over Ethernet, which is where Art-Net comes in.
Art-Net is a protocol that sends DMX over IP, which is a lot more flexible than physical cables. It's also a lot easier to work with, as you can just use a normal network cable to send DMX around. And that's also where we start to differ from the real world.

Because it's a network signal, we can setup a socket to listen for Art-Net commands, and do with them whatever we want. The lighting desk/software doesn't know if it's talking to a real light or a Minecraft block, it just sends the commands and hopes that something will listen.

[It's an open standard](https://art-net.org.uk/downloads/art-net.pdf) provided by [Artistic License](https://artisticlicence.com/), so you can just implement it in your own software. And that's what we did.

## The pipeline
We have two separate pipelines for working with Art-Net, the primary one is for the creative folks, and the secondary one is for playback.

- Primary pipeline (live)

![A cobalt debug view that shows when someone is connected to the Art-Net node](/blog/articles/ma_minecraft/img_2.png)
When building a show, you want to see what you are programming in your lighting software, preferably in real-time. This is where the primary pipeline comes in. This runs an embedded Art-Net node in-game, which we can target through unicast and then gets injected into our rendering engine. This way, we can see the lights in-game, and we can see the lights in our lighting software. This is a huge improvement over the old way of doing things, where we had to manually convert the show into Minecraft commands or scriptable actions.

- Secondary pipeline (playback)

![The self made Art-Net recorder with timecode support](/blog/articles/ma_minecraft/img_1.png)
And while this works for building shows, it doesn't work for running them. We don't want to have someone operating every show, because they run on a set schedule and we want to be able to run them without us having to get up at 3 AM. This is where the secondary pipeline comes in. We have a custom standalone application (written in Go), that also listens for Art-Net commands, but instead of rendering them in-game, it processes them, minifies every frame (only using keyframes every few seconds), and then pumps them into a file (shared protocolbuffer) that we can upload to our server.

Then, when it's time for our night spectacles, the show can be loaded into our server, and the server will play the show back, parsing the protocolbuffer and sending the raw timecoded DMX changes to our rendering engine, the same way our primary pipeline does. 

## Mapping to in-game actions
![Snippet from Cobalt's abstract fixture class](/blog/articles/ma_minecraft/img_3.png)
We have a predefined set of virtual fixtures (like lasers, moving spots, command triggers, and even pixel-mapped LED blocks), which are all 'patched' to listen to a specific universe and address. That way, our rendering engine can take the raw DMX universe states, and pass them on to our fixtures. And this is where the magic happens. We can now control our in-game lights with a real lighting desk, and we can control our real lights with in-game actions.

| Universe | Type | Start Channel | End Channel | Amount | Description                       |
|----------|------|---------------|-------------|---------|-----------------------------------|
| 5 | End Beam | 0 | 127 | 32 | Castle End Beams                  |
| 5 | Moving Head | 128 | 131 | 1 | Castle Moving Head                |
| 5 | Pixel Block | 132 | 320 | 63 | Castle Matrix                     |
| 5 | Moving Head | 321 | 328 | 2 | Ballroom Lasers                   |
| 6 | Pixel Block | 0 | 62 | 20 | Decorative Pixel (Christmas Tree) |

![Main castle end beams](/blog/articles/ma_minecraft/endbeams.png)
![Main castle pixel mapping screen](/blog/articles/ma_minecraft/pixels.png)

## Conclusion
This has been a huge project for us, and it's been a lot of fun to work on. A few of us work in the real life event industry as well, and are incredibly passionate about bringing the work we love so much into the game we love even more. We've been able to make some really cool shows with this technology, and are still working on improving it every day and seeing how far we can push it to build shows on scales we've never seen before.

![End product - show by SlinxMC](/blog/articles/ma_minecraft/img_7.png)


Block art with Art-Net

We’ve all been there before.
You’re working on a project with some large inherited codebase, from code elder gods who cannot be reasoned with. Or, you’re interacting with a library, that just does this one annoying thing coming from some terrible decision, and now you need to work around it.

Forking it might be an option, but do you really want to take on that responsibility for this one off thing? And do you really want to challenge the senior elder gods? Your situation is feeling more dire with every class you decompile, no way you’re picking all of this up.

The go-to method within modding communities is to setup mixins, or even dabble with the occasional javaassist, which does the job, but is fragile and breaks between versions. Or maybe the code you write now may run in weird configurations or security policies, in which case, you certainly don’t want to work with either.

If only there was some vanilla way in Java, that when mastered, will allow you to bodge through pretty much **anything**.
>  **Foreshadowing: a literary device in which a writer gives an advance hint of what is to come later in the story**

## Bodging
[Tom Scott explained it best in his video](https://youtu.be/lIFE7h3m40U?si=zdAtMWJ-E7tmLZrO). It’s the art of patching something ugly together, that works, just for now, or as a permanent solution if you’re willing to ignore the war crimes you committed in the form of code.
Java is no stranger to either, and though that can lead some pitfalls and boilerplate abstraction hell, this is one of the cases where you can play god, and literally bend classes at your will during runtime.

## Introduction to proxies
Proxies, best known from their JavaScript implementation, are special elements that let you intercept calls and interactions with an object. 
```js
const target = { name: 'John', age: 30 };

const handler = {
  get: (target, prop) => prop in target ? target[prop] : `No such property: ${prop}`,
  set: (target, prop, value) => prop === 'age' && value <= 0 ? false : (target[prop] = value),
  deleteProperty: (target, prop) => prop === 'name' ? false : delete target[prop]
};

const proxy = new Proxy(target, handler);

console.log(proxy.name);     // John
proxy.age = 25;              // Sets age to 25
console.log(proxy.age);      // 25
proxy.age = -5;              // Invalid age, doesn't change
console.log(proxy.age);      // 25
```
This is something that works really well in JS land, types are only a suggestion after all. There’s no governing body running around, telling you that they wouldn’t be compatible.

But, what if I tell you that Java is more forgiving than you might think? And it would let you do similar gymnastics as well. You might not care as much, but this stuff absolutely rocked my world.

## Java Types - why is this so powerful
Types in java are strict. You may simply not cast `7` to a `2001 Toyota corolla` no matter how hard you try. If a field is marked as a specific object type, then there’s no way to change it’s type to a different family (maybe if you were to play with [Unsafe](https://github.com/openjdk/jdk/blob/master/src/jdk.unsupported/share/classes/sun/misc/Unsafe.java), but we’re talking about bodging here, not inventing new means of self destruction).

The only way to overwrite existing behavior is to extend the type you’re targeting and only implement the methods you’re trying to manipulate, passing the rest back to super. This works, but is technical debt when the target type is outside of your control and may be updated (with new methods, signatures, or even worse).

## Java Proxies
Yes, they exist, and they work.
It’s one of the exceptions where java will actually generate new class definitions during runtime, and it is awesome.

They work by creating a new real class during runtime, and registering it directly in [JavaLanguageAccess](https://code.yawk.at/java/15/java.base/jdk/internal/access/JavaLangAccess.java#jdk.internal.access.JavaLangAccess%23defineClass(java.lang.ClassLoader%2Cjava.lang.String%2Cbyte%5B%5D%2Cjava.security.ProtectionDomain%2Cjava.lang.String)) (which is really interesting on it own, and i highly recommend looking into), and it is even smart enough to cache the generated class for future use (when you want to generate a new proxy of the same type).

To get started, we need to create a new `InvocationHandler` that will intercept the calls to the object we’re trying to proxy. This is a functional interface, so we can use a lambda to implement it.
```java
InvocationHandler handler = (proxy, method, args) -> {
    if (method.getName().equals("toString")) {
        return "Hello from the proxy!";
    }
    return null;
};
```

Next, we need to create the proxy itself. This is done by using the `Proxy.newProxyInstance` method, which takes a classloader, an array of interfaces we want to proxy, and the handler we just created. For the examples moving forward, we'll be manipulating a bukkit Player object, and we'll be using the `Player` interface as the target.
```java
Player player = (Player) Proxy.newProxyInstance(
    Player.class.getClassLoader(),
    new Class[] { Player.class },
    handler
);
```

And voila, we have a proxy object that will intercept calls to the `Player` object, and return `Hello from the proxy!` when `toString` is called.
But, this isn't useful in the real world, because this will explode when you try to call any other method on the object, but this is also where the beauty of proxies comes in. You can now implement the methods you want to intercept, and pass the rest back to the original object dynamically, without having to implement the entire interface or even worrying about future changes because it'll be evaluated during runtime, and your used api version doesn't matter.

Where is this useful?

# Example 1: Faking permissions
This may sound far fetched, but is a diabolical thing I had to do in the past. I was calling out to a library which was not worth the effort to fork, and it **insisted** that the passed player had to be OP. Though it's possible to just wrap that call with a setOP invocation, I'd rather not give the player any OP permissions *ever*

Well, we can use a proxy to work around this! I wrote this little middleware, that only intercepts the isOp method, and returns our custom value, but still passes the rest of the calls to the original object.
```java
public class PlayerOpProxy implements InvocationHandler {

    private boolean opStatus;
    private final Player originalPlayer;
    private Player proxiedPlayer;

    public PlayerOpProxy(Player originalPlayer, boolean isOp) {
        this.originalPlayer = originalPlayer;

        this.proxiedPlayer = (Player) Proxy.newProxyInstance(
                originalPlayer.getClass().getClassLoader(),
                originalPlayer.getClass().getInterfaces(),
                this
        );
    }

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        if (method.getName().equals("isOp")) {
            return this.opStatus;
        } else if (method.getName().equals("setOp")) {
            this.opStatus = (boolean) args[0];
            return null;
        } else {
            return method.invoke(this.originalPlayer, args);
        }
    }

    public Player getProxiedPlayer() {
        return this.proxiedPlayer;
    }
}
```

# Example 2: Function middleware (caching)
This is a more practical example, where we want to cache the result of a function call, and return the cached value if the function is called. Normally we would have to implement a caching strategy ourself, but with a proxy we can abstract all of this as if it were a code generator (which it is, in a way).

If we use a proxy based method, then all we need to do is
```java
public interface UserService {
    @CacheUtil.Cache(retention = 5)
    String getUserName(int id);
}

// Creating a proxy
UserService service = new UserServiceImpl();
UserService proxiedService = CacheUtil.createProxy(service, UserService.class);

// Using the proxied service
String result = proxiedService.getUserName(1); // calls the original method
String cachedResult = proxiedService.getUserName(1); // returns the cached value, never actually reaching the original method
```

And this entirely works without any external dependencies, and without requiring ahead of time binary manipulation or mixins. It's a simple, clean, and effective way to manipulate objects at runtime, and it's a tool that I'm surprised isn't used more often.

Here's my CacheUtil implementation, as a freebie for reading this far.
```java
public final class CacheUtil {
    
    /**
     * Annotation to specify caching behavior for methods.
     * Methods annotated with {@code @Cache} will have their return values cached
     * for the specified retention time in seconds.
     */
    @Retention(RetentionPolicy.RUNTIME)
    @Target(ElementType.METHOD)
    public @interface Cache {
        /**
         * Specifies how long the cached value should be retained in seconds.
         * @return retention time in seconds
         */
        int retention() default 5;
    }

    /**
     * Creates a proxy instance that implements caching behavior for annotated methods.
     *
     * @param <T> the type of the interface to be proxied
     * @param target the object to be proxied
     * @param interfaceClass the interface class that the proxy will implement
     * @return a proxied instance with caching behavior
     * @throws IllegalArgumentException if target or interfaceClass is null
     */
    @SuppressWarnings("unchecked")
    public static <T> T createProxy(T target, Class<T> interfaceClass) {
        if (target == null || interfaceClass == null) {
            throw new IllegalArgumentException("Target and interface class cannot be null");
        }
        return (T) Proxy.newProxyInstance(
            interfaceClass.getClassLoader(),
            new Class<?>[] { interfaceClass },
            new CacheHandler(target)
        );
    }

    private static class CacheEntry {
        private final Object value;
        private final long expirationTime;

        CacheEntry(Object value, long expirationTime) {
            this.value = value;
            this.expirationTime = expirationTime;
        }

        boolean isExpired() {
            return System.currentTimeMillis() > expirationTime;
        }

        Object getValue() {
            return value;
        }
    }

    private static class CacheHandler implements InvocationHandler {
        private final Object target;
        private final Map<String, CacheEntry> cache = new ConcurrentHashMap<>();

        CacheHandler(Object target) {
            this.target = target;
        }

        @Override
        public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
            Cache cacheAnnotation = method.getAnnotation(Cache.class);
            if (cacheAnnotation == null) {
                return method.invoke(target, args);
            }

            String cacheKey = generateCacheKey(method, args);
            CacheEntry entry = cache.get(cacheKey);
            
            if (entry != null && !entry.isExpired()) {
                return entry.getValue();
            }

            Object result = method.invoke(target, args);
            long expirationTime = System.currentTimeMillis() + (cacheAnnotation.retention() * 1000L);
            cache.put(cacheKey, new CacheEntry(result, expirationTime));

            return result;
        }

        private String generateCacheKey(Method method, Object[] args) {
            return method.getName() + ":" + (args != null ? Arrays.deepHashCode(args) : "");
        }
    }
}
```


Bodging in Java - crimes against the runtime

# Two things have always been true about me:
- I love sharing things I work on
- I am incredibly lazy and drop anything that has the slightest bit of friction

And sadly, these two things are not compatible.
So I decided to build a blog. But not just any blog, a blog that is so overengineered that there's a really fair chance that I spent more time building it than I will ever spend writing on it.

In a perfect world, I would just have a folder where I could quack out some markdown files and have them automagically appear on the internet. And there are many tools that do just that, leagues better than what I could ever build. But that's not the point. The point is to build something myself, and to learn from it.
The last site I built with this idea (the OpenAudioMc docs) required me to manually register markdown files in a JSON file, provide metadata, and then painfully wait for Next.js to rebuild the site. It was a pain, but it worked. And it was fun to build.

Naturally, I asked myself: "How can I make this even more painful?".
The answer was simple: "Let's build a vite plugin that does this".
Why should vite have anything to do with this? Because fuck it, that's why.

# Drafting
I started building the site normally, quick router, sprinkle in some [ReactMarkdown](https://github.com/remarkjs/react-markdown), hardcode a text blob, and, then, well shit, what now?

Well, markdown "supports" comments, not every parser respects them, but you can abuse this mechanism to embed metadata in a markdown file itself!
A quick
```yaml
---
title: Overbuilding a blog
date: '2024-01-27'
excerpt: Probably the worst way to build a blog
tags:
  - Personal
coverImage: /blog/articles/welcome/scheveningen.png
author:
  name: Mats
  avatar: https://github.com/Mindgamesnl.png
  role: The king
---
```
then, pull it through [Gray Matter](https://www.npmjs.com/package/gray-matter) (or any other yaml  parser), and congratulations! You are now the proud owner of some goofy ahh metadata system!

# Indexing
Like I said, I didn't want to write yucky manual indexes for my files, but I also didn't want to setup a full CMS or crud (I want this to be a dumb static site, no fancy serverless nonsense).

Vite already bundles my garbage JS code, so why not also abuse it to bundle my markdown?
And with that, I don't mean bundling it as huge fuckass strings/blobs, but building a custom plugin that reads markdown files, parses it's embedded metadata, listens for changes, and then makes a virtual module to export all this data.
So I did just that.

## What are Virtual Modules?
Virtual Modules are a concept by the Module government, to sell you more modules without needing to write out a file. You can dynamically generate them in a plugin, and then access them through a special namespace/prefix in your code.

We can implement the resolve method in our vite plugin, to tell it that an import statement should actually be using our virtual module instead!
```js
resolveId(id) {
    if (id === 'virtual:blog-data') {
        return '\0virtual:blog-data';
    }
}
```
Vite will then later call the load method, where we can return the actual content of our virtual module.
```js
load(id) {
    if (id === '\0virtual:blog-data') {
        return `export const exampleData = ${JSON.stringify({ someObject: 'thingy' })};`;
    }
}
```

## Implementing a virtual blog module
Now that we know that we can dynamically generate bundles, we can make one that:
- Lists all my markdown files
- Parses its content and metadata yaml blob
- is pretty slow but I don't care yet

```js
const processArticle = (filePath) => {
  const source = fs.readFileSync(filePath, 'utf-8');
  const { data: frontmatter, content } = matter(source);
  const slug = path.basename(filePath, '.md');

  return {
    slug,
    content,
    ...frontmatter,
    title: frontmatter.title || 'Untitled',
    date: frontmatter.date || new Date().toISOString(),
    excerpt: frontmatter.excerpt || '',
    readTime: calculateReadTime(content),
    author: {
      name: frontmatter.author?.name || 'Mats',
      avatar: frontmatter.author?.avatar || 'https://github.com/Mindgamesnl.png',
      role: frontmatter.author?.role || '',
      bio: frontmatter.author?.bio || ''
    },
    tags: frontmatter.tags || [],
    coverImage: frontmatter.coverImage || '/blog/articles/cover.png'
  };
};

const loadAllArticles = () => {
  const blogDir = path.resolve('src/content/blog/articles');
  const files = fs.readdirSync(blogDir)
    .filter(file => file.endsWith('.md'));

  const articles = files.map(file => {
    const filePath = path.join(blogDir, file);
    return processArticle(filePath);
  });

  articles.sort((a, b) => new Date(b.date) - new Date(a.date));
  return articles;
};
```

which we can then use in our module, where we'll embed it as a json object (passed as string, because vite doesn't like objects). You could also provide function implementations here, and import them like you would any other function from any other module, but this already feels incredibly dirty (it is), and even I have some standards.

So, with that, our load method will look like this:
```js
load(id) {
  if (id === '\0virtual:blog-data') {
    const articles = loadAllArticles();

    return `
      const articles = ${JSON.stringify(articles, null, 2)};
      export { articles };
    `;
  }
}
```

# Consuming the data
Now that we have our data, we can import it in our React components, and use it to render our blog posts.
I wrote a quick little hook that fetches the data from our virtual module, and returns it as a normal object.
```jsx
import { useMemo } from 'react';
import { articles } from 'virtual:blog-data';

export const useBlogArticles = () => {
  return articles;
};

export const useBlogArticle = (slug) => {
  return useMemo(() => {
    if (!slug) return undefined;
    const article = articles.find(article => article.slug === slug);
    return article || null;
  }, [slug]);
};
```

And that's it! this will work just like any other hook. But I don't like it.
This is forgetting my main goal: to let myself be as lazy as possible.

I want hot reloading, I want to be able to write a markdown file, and see it appear on my site without needing to refresh the page. And I want to do this without needing to write a single line of code or god forbid, tab out of my editor.

# Self-reloading
Making the plugin listen for changes is the easy part, we add markdown files, sprinkle in a listener, and we're gaming
```js
const blogDir = path.resolve('src/');

server.watcher.add(path.resolve(blogDir, '**/*.md'));

server.watcher.on('change', (file) => {
  if (file.endsWith('.md')) {
    console.log(`[blog] File changed: ${file}`);
    const article = processArticle(file);
  }
});
```
but how do we then get this to the client?
We don't have to! Our magnificent vite overlords got us covered here too!
Vite already runs a websocket server under the hood which is attached and available to us in our plugin and hook, 
all we need to do is send a custom event, catch it in the hook, and we're absolutely golden!

```js
server.watcher.on('change', (file) => {
  if (file.endsWith('.md')) {
    console.log(`[blog] File changed: ${file}`);
    const article = processArticle(file);
    server.ws.send({
      type: 'custom',
      event: 'blog-article-update',
      data: {
        slug: article.slug,
        event: 'blog-article-update',
        article
      }
    });
  }
});
```

# End result
And that's it! I can now build my blog by writing markdown files, and have them automagically appear on my site, even while running a development instance with hot reloading.
All it took was this (final) vite blog plugin:
```js
import fs from 'fs';
import path from 'path';
import matter from 'gray-matter';

const calculateReadTime = (content) => {
  const wordsPerMinute = 200;
  const words = content.trim().split(/\s+/).length;
  const minutes = Math.ceil(words / wordsPerMinute);
  return minutes === 1 ? '1 minute read' : minutes < 1 ? 'Quick read' : `${minutes} minutes read`;
};

export default function blogPlugin() {
  const virtualModuleId = 'virtual:blog-data';
  const resolvedVirtualModuleId = '\0' + virtualModuleId;

  const processArticle = (filePath) => {
    const source = fs.readFileSync(filePath, 'utf-8');
    const { data: frontmatter, content } = matter(source);
    const slug = path.basename(filePath, '.md');

    return {
      slug,
      content,
      ...frontmatter,
      title: frontmatter.title || 'Untitled',
      date: frontmatter.date || new Date().toISOString(),
      excerpt: frontmatter.excerpt || '',
      readTime: calculateReadTime(content),
      author: {
        name: frontmatter.author?.name || 'Mats',
        avatar: frontmatter.author?.avatar || 'https://github.com/Mindgamesnl.png',
        role: frontmatter.author?.role || '',
        bio: frontmatter.author?.bio || ''
      },
      tags: frontmatter.tags || [],
      coverImage: frontmatter.coverImage || '/blog/articles/cover.png'
    };
  };

  const loadAllArticles = () => {
    const blogDir = path.resolve('src/content/blog/articles');
    const files = fs.readdirSync(blogDir)
      .filter(file => file.endsWith('.md'));

    const articles = files.map(file => {
      const filePath = path.join(blogDir, file);
      return processArticle(filePath);
    });

    articles.sort((a, b) => new Date(b.date) - new Date(a.date));
    return articles;
  };

  return {
    name: 'vite-plugin-blog-data',
    enforce: 'pre',

    configureServer(server) {
      const blogDir = path.resolve('src/');
      server.watcher.add(path.resolve(blogDir, '**/*.md'));

      server.watcher.on('change', (file) => {
        if (file.endsWith('.md')) {
          console.log(`[blog] File changed: ${file}`);
          const article = processArticle(file);
          server.ws.send({
            type: 'custom',
            event: 'blog-article-update',
            data: {
              slug: article.slug,
              event: 'blog-article-update',
              article
            }
          });
        }
      });
    },

    resolveId(id) {
      if (id === virtualModuleId) {
        return resolvedVirtualModuleId;
      }
    },

    load(id) {
      if (id === resolvedVirtualModuleId) {
        const articles = loadAllArticles();

        return `
          const articles = ${JSON.stringify(articles, null, 2)};
          export { articles };
        `;
      }
    }
  };
}
```

and this hook:
```js
import { useState, useEffect, useMemo } from 'react';
import { articles as initialArticles } from 'virtual:blog-data';

// Shared state between hooks
let currentArticles = [...initialArticles];

export function useBlogArticles() {
  const [articles, setArticles] = useState(currentArticles);

  useEffect(() => {
    // Only set up hot reloading in development
    if (!import.meta.hot) return;

    const handleUpdate = (data) => {
      if (data.event === 'blog-article-update') {
        const updatedArticle = data.article;
        const index = currentArticles.findIndex(
          article => article.slug === updatedArticle.slug
        );
        
        if (index !== -1) {
          currentArticles[index] = updatedArticle;
        } else {
          currentArticles.push(updatedArticle);
        }

        // Update component state
        setArticles([...currentArticles]);
      }
    };

    import.meta.hot.on('blog-article-update', handleUpdate);
    return () => import.meta.hot.off('blog-article-update', handleUpdate);
  }, []);

  return articles;
}

export function useBlogArticle(slug) {
  const articles = useBlogArticles();

  return useMemo(() => {
    if (!slug) return undefined;
    return articles.find(article => article.slug === slug) || null;
  }, [slug, articles]);
}

if (import.meta.env.DEV) {
  window.__blogArticles = currentArticles;
}
```


Abusing virtual modules for a blog


# The 80% Problem

Running a large-scale Minecraft theme park comes with some interesting performance challenges. The most sobering metric we discovered? **80% of our server's processing power** was being consumed by a single system: our ride mechanics.

This overhead was happening **every single tick**, 20 times per second, regardless of whether anyone was actually riding the attractions. We had a serious performance bottleneck that was limiting everything else we could build.

## The Ticking Time Bomb

Here's how Minecraft works: everything that needs to happen in the game world gets 50 milliseconds to complete, 20 times per second. Miss that deadline by even 1 millisecond, and the entire server starts falling behind. Players see rides freeze mid-air, shows desync, and the magic disappears faster than you can say "server lag."

Our profilers painted a grim picture. While player movement, commands, animatronics, and literally everything else on the server combined took up about 20% of resources, our ride system was hogging the remaining 80% doing things like:

- Calculating entity positions every tick for hundreds of vehicles
- Computing track orientations and banking in real-time  
- Managing entity tracking and interactions
- Triggering effects and redstone systems
- Handling collision detection

It was death by a thousand calculations, and we were running out of computational headroom fast.

## The Relationship That Couldn't Last

[TrainCarts](https://www.spigotmc.org/resources/traincarts.39592/) was absolutely essential to ImagineFun's existence. When we opened in June 2018, TrainCarts and its companion plugin TCCoasters were revolutionary—they literally enabled the second wave of Minecraft theme park servers.

But as our ambitions grew and our player base expanded, we started hitting fundamental limitations. The system that got us started wasn't going to scale to where we wanted to go.

We needed to completely rethink how we handle entities in Minecraft. Not just optimize the existing system—replace it entirely.

## The Nuclear Option: Going Straight to the Protocol

Most sane developers would have tried optimizing the existing TrainCarts system first. We decided to be insane instead.

Instead of working within Minecraft's entity system, we decided to bypass it completely. What if we could intercept all network traffic between the server and clients, and handle our own entities without Minecraft even knowing they exist?

Here's the key insight: Minecraft clients don't actually care where entity data comes from. They just want to receive the right packets at the right time. So we built COBALT to work entirely at the network layer, intercepting traffic and managing our own parallel universe of entities.

When a player interacts with a COBALT entity, the Minecraft server doesn't know something happened. When we spawn 1,000 entities for a massive ride, the server continues operating normally. We essentially created a way to have extensive entity interactions that are completely invisible to the main server process.

### Under the Hood: The Protocol Hijack

The technical implementation of this network interception is actually quite elegant. We hook directly into Minecraft's networking layer using [Netty](https://netty.io/)—the same framework that powers Minecraft's own networking.

Every packet that flows between server and client gets a chance to be intercepted, modified, or completely replaced by COBALT. When we need to spawn an entity, we generate the exact same packets that Minecraft would send, but we do it from our own thread pool without any of Minecraft's overhead.

This gives us some incredible superpowers:

- **Chunk tracking**: We know exactly which chunks each player has loaded
- **Zero server overhead**: COBALT entities don't exist in Minecraft's entity tracker
- **Perfect compatibility**: Clients can't tell the difference between real and virtual entities
- **Horizontal scaling**: COBALT runs in its own thread pool, completely separate from Minecraft's tick loop

The result? We can spawn 100,000 entities and the Minecraft server thinks it's managing zero. We essentially created an invisible parallel entity system that operates alongside the main server.

### The API: Making the Complex Look Simple

Here's the beautiful part—despite all the complexity under the hood, the API for developers is dead simple. Let's compare spawning an entity the traditional way versus the COBALT way:

**Traditional Bukkit API:**
```java
ArmorStand armorStand = (ArmorStand) loc.getWorld().spawn(loc, ArmorStand.class);
```

**COBALT API:**
```java
ArmorStand armorStand = new VirtualArmorStand(location, entityHandle);
```

Both have the exact same client-side behavior, but one runs in Minecraft's entity system while the other runs in our parallel universe. The beauty is that other developers on our team don't need to learn a completely new API—it follows the same patterns they already know.

## The Animation Revolution: From Real-Time to Pre-Baked

The old system calculated every single frame of animation in real-time. This meant constantly computing positions, rotations, and interactions for every vehicle on every ride, 20 times per second.

COBALT takes a fundamentally different approach. Instead of calculating positions on the fly, we pre-bake everything. Every animation becomes a data file that specifies exactly where each entity should be at every moment in time, what effects should trigger when, and how smooth transitions should happen.

This approach eliminates virtually all runtime calculation. When a ride starts, COBALT reads from the animation file and sends the appropriate packets. No physics calculations, no pathfinding, no real-time decision making—just efficient data playback.

## The Architecture: Building Virtual Reality Inside Minecraft

COBALT's architecture is built around layers of abstraction that provide flexibility while maintaining performance.

At the core, every entity is a `VirtualEntity` that carries its own `EntityNetworkManager`. This network manager acts as a translator between our high-level APIs and the low-level Minecraft protocol, handling all the messy details of packet generation and client communication.

But here's where it gets interesting: we built different types of virtual entities for different use cases:

**Vehicle Entities**: The stars of the show, these handle all the ride logic. They can have custom behavior like the spinning teacups in *Roger Rabbit's Car Toon Spin* or the banking on *Big Thunder Mountain*.

**Animatronic Entities**: Static display entities that don't need player interaction but can have complex animations and effects.

**Show Elements**: Special entities with custom rendering rules, optimized render distances, and performance tweaks.

Each vehicle can contain multiple virtual entities—seats, model parts, particle effects, you name it. The front car of Big Thunder Mountain, for example, has separate entities for the locomotive model, the smoke particles, and each individual seat.

![Roger Rabbit's vehicle with custom onSteer handler for spinning](/blog/articles/cobalt/roger-rabbit-vehicle.png)

*Roger Rabbit's vehicle with custom onSteer handler for spinning*

![Class hierarchy of a Chicken](/blog/articles/cobalt/chicken-hierarchy.png)

*Class hierarchy of a Chicken - showing the layered abstraction approach*

![Big Thunder Mountain vehicle](/blog/articles/cobalt/big-thunder-vehicle.png)

*Big Thunder Mountain vehicle structure*

![Vehicle handler for the front train from Big Thunder Mountain with smoke particles](/blog/articles/cobalt/big-thunder-smoke.png)

*Vehicle handler for the front train from Big Thunder Mountain with smoke particles*

![Screenshot of COBALT entity tracker visualization showing chunks and entity counts](/blog/articles/cobalt/radar.png)

*COBALT's in-game entity tracker, showing chunks around a player and the number of tracked entities in each chunk.*

## The Blender Connection: When Art Meets Code

Creating animations for COBALT isn't just about writing code—it's about giving our creative team the tools they deserve. We integrated directly with [Blender](https://www.blender.org/), the industry-standard 3D animation software that Disney and Pixar use for their movies.

Our Imagineers can now design rides with the same tools used to create *Toy Story* and *The Incredibles*. They get proper 3D viewport, professional animation tools, and all the creative freedom they could want.

But here's the problem we ran into: we already had dozens of existing rides that were built with the old TrainCarts system. Converting each one manually would have taken months and probably driven our creative team to quit.

So naturally, we built *another* pipeline that could automatically convert TrainCarts rides into COBALT animations. This conversion process:

- Loads existing TrainCarts path nodes
- Discovers event triggers from sign placement  
- Converts TrainCarts animations into our format
- Applies post-processing and optimization

Converting a ride from TrainCarts to COBALT now takes about 10 minutes instead of 10 weeks. This automated conversion process was crucial for migrating our existing content.

Here's a converted render of the Big Thunder Mountain dataset (note that it isn't running in real-time for debugging purposes):

<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/t0EuaVgH-Ck?si=IdIcPni5i8EPTTjk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## Performance: The Numbers Don't Lie

After months of development and testing, we finally had our answer to the 80% problem. The results were so dramatic that we initially thought our profilers were broken.

COBALT manages 500+ entities per server while consuming processing time measured in **microseconds**. The old system was consuming 80% of server resources; COBALT uses less than 1%.

That's not a typo. We achieved roughly **200x performance improvement** over the original TrainCarts system.

![Screenshot of production performance metrics showing microsecond-level processing times](/blog/articles/cobalt/grafana.png)

*Production metrics from July 8th, 2022, showing COBALT handling 500 entities with tick times in microseconds.*

But the performance gains go beyond just raw numbers. Because COBALT entities don't exist in Minecraft's entity tracker, they don't interfere with other plugins, don't cause chunk loading issues, and don't contribute to entity cramming or other Minecraft limitations.

We went from a system that was severely limiting our server capacity to one that barely registers on performance monitors. The improvement opened up possibilities we hadn't even considered before.

## Future-Proofing: Building Tomorrow's Foundation

COBALT wasn't just built to solve today's problems—it was designed to enable features we couldn't even dream of with the old system. The modular architecture means we can easily add new entity types, behaviors, and optimizations without rebuilding from scratch.

We're already experimenting with:

- **Dynamic render distances** based on player movement and attention
- **Predictive entity loading** for smoother experiences  
- **Cross-server entity synchronization** for massive events
- **Real-time animation editing** for rapid iteration

The best part? None of these features require changes to the core engine. COBALT's plugin-style architecture means new capabilities can be added as modules without touching the foundation.

## The Human Impact: When Tech Meets Magic

All this technical wizardry might sound abstract, but it has very real impacts on player experience. Before COBALT, we had to carefully manage how many rides could run simultaneously. Large events required disabling attractions to keep the server stable.

Now? We can run every single ride at full capacity while handling hundreds of players across multiple experiences simultaneously. The difference is night and day.

Players notice the difference immediately. Rides run smoothly, shows stay synchronized, and the overall experience feels much more polished. Good technology should be invisible to the end user, and COBALT achieves that goal.

## Battle-Testing: When Theory Meets Reality

COBALT has been running in production at ImagineFun since mid-2022, handling millions of ride cycles and thousands of player hours. We've stress-tested it with events that would have caused problems with the old system, and it continues to perform reliably.

One of the most satisfying moments was running a server-wide event with every attraction operating simultaneously while maintaining excellent performance. It demonstrated how far we'd come from the original 80% overhead problem.

## The Ripple Effect: Beyond Just Rides

COBALT started as a ride system, but it's evolved into something much bigger. The entity management capabilities are now used throughout ImagineFun for:

- **Animatronics** in restaurants and shops
- **Dynamic show elements** for parades and events  
- **Interactive displays** that respond to player actions
- **Custom vehicles** for transportation around the park

What began as a solution to one specific problem became the foundation for an entirely new level of creative possibilities.

## Behind the Scenes: Development Screenshots

![Development screenshot 1](/blog/articles/cobalt/render-1.png)

![Development screenshot 2](/blog/articles/cobalt/render-2.png)

![Development screenshot 3](/blog/articles/cobalt/render-3.png)

## FAQ: The Technical Deep Dive

**Q: How does COBALT handle edge cases like player disconnections or chunk unloading?**

A: COBALT maintains its own chunk tracking system and gracefully handles all the edge cases that Minecraft's entity system struggles with. When a player disconnects, COBALT immediately stops sending packets to that client. When chunks unload, COBALT pauses entity updates for that region until players return.

**Q: Can other servers use COBALT?**

A: COBALT was built specifically for ImagineFun's needs and is tightly integrated with our infrastructure. While the concepts could be applied elsewhere, the implementation is custom-tailored to our specific use case and requirements.

**Q: How do you debug entities that don't exist in Minecraft?**

A: We built extensive debugging tools including real-time entity visualizers, performance profilers, and logging systems that track every aspect of COBALT's operation. It's actually easier to debug than traditional Minecraft entities because we have complete control over the data flow.

**Q: What's next for COBALT?**

A: We're continuously evolving the system based on our growing ambitions. Future plans include enhanced cross-server synchronization and even more sophisticated animation capabilities. The modular architecture means we can keep pushing boundaries without rebuilding the foundation.
