The year is 2023. It's a little over 2 years ago that I haphazardly decided to leave the city I was born and raised in, and move to the other side of the country.

One month before I was supposed to enroll at HVA, I pulled the trigger and decided that the "easy" way wasn't for me (yes, about as good a decision as you would expect from a 20-year-old).

This had some direct ramifications, like suddenly having to deal with a 4 hour daily commute. "But that's fine", he kept saying to himself, "the national railway service is solid" (which was another obvious lie). But, against all odds, I managed to get through the first year of my studies without losing my mind.

It wasn't until I started to pick up more student activities (our local student pub, the student association, and a few other things) that I started to realize that commuting was not going to work out for me. I started missing out on a lot of things, and started showing up too tired to do anything.

It was time for another change, and it was time to commit to Den Haag.

The landscape

Finding a place to live in the Netherlands is hard. Who would've thought? The housing market is a mess, and the Dutch entrepreneurial spirit is reaping all the benefits. Between fake listings, scams, shady figures, and a scary amount of just straight-up fraud, there are a million ways to get screwed over.

You know it's bad based on the news reports alone, but it isn't until you start looking around that the existential dread sets in.

Dissecting the options

The first step towards any good solution, is to dissect the problem and do your research. A lot of it.

There are a few "solid" options for finding your new mancave, which all seem fine on the surface. It won't be long until you find one of the following:

• Facebook groups: Often invite only, where people will share "their" listings.
• Room.nl: A site that's contracted by local housing corporations to list their available rooms.
• Pararius: A site that lists all available rooms, but is mostly used by real estate agents. Think funda, but for rentals.
• Real estate agents: The traditional way of finding a place to live, but with a hefty price tag.

Where the fun begins

Aside from the more "official" options, there are hundreds of sites that list available rooms, and almost all of them require some form of paid membership to access the listings.

I've spent a pretty penny on these sites, and there was just something giving me the ick. Most of these sites had the same listings, and there was never a guarantee that you would ever hear back from the landlord. An obvious red flag, but not hard to believe when you're being told that the space is highly competitive.

FUN FACT! Did you know that agents are well aware of how scam-ridden the housing market is? To the point where they charge a monthly fee just to visit their site? Just so you, as a mere peasant, can see the listings that they have available.

It wasn't until one of these sites messed up that the facade started to crack.

A listing on a site that I trusted had a funny little detail in its description. The listing said something like this:

Interested? Send an email to <a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="(DATA)">[email protected]</a></td>

That isn't a normal email address notation. It's the smoking gun that started to unravel the entire thing for me.

That, right there, is a Cloudflare email obfuscation technique. It's a service website owners can install over their website to prevent data from being collected by robots (which is how your email ends up on spam lists, for example). You should never see visible artifacting from it, because your browser will automatically decode it for you. But poorly written crawlers, are not browsers.

And oh my fucking god. It clicked.

They are all crawlers

I had been using these sites for weeks, and I never even considered that I was being played from all sides.

The double listings, never hearing back from landlords, why all these sites were paid—it all makes sense. It's a money-printing machine. And that day, I declared war on the housing market.

If there are so many fake sites with subscriptions, then that means there is profit to be made. For this to be even remotely interesting in such a competitive market, there must be a crapload of people falling for it. People in the exact same situation.

So how does one get a competitive edge in such a market?

Fighting fire with fire, of course. It is time, to outcrawl the crawlers.

Businesses on top of businesses

I really wanted to move out, and thus, cast as wide a net as possible.

My original thought was to periodically scrape all the sites I could find, and automatically apply (or send a push notification) to new listings that became available.

If the space of rental sites is so competitive, then surely the gap in the market is big enough to warrant a business to provide all of it. So, instead of trying to adapt to every individual site, why don't I target the platform they all use?

It isn't hard to see that sites like Woonnet Haaglanden (providing general public housing) and Roommatch/Room.nl (providing student housing) look exactly the same. This is because both of them are built on top of the same platform. And luckily for me, they all use the exact same REST API.

Outcrawling the crawlers

Time to build a crawler suite to spam the everloving crap out of the sites that I could find. And in my own personal style of bullshit, I wanted to over-architect the hell out of it.

For this particular adventure, I picked my preferred poison of Go because it's easy to deploy and has a great standard library for HTTP requests. The plan was simple:

• Build a standard library to interact with various APIs. I had no clue how long I would be looking for a place, so I wanted everything to be easily maintainable in the future.
• Periodically crawl all targets that I had implemented
• Send a push notification to my phone when a new listing is available
• Automatically apply to new listings, if possible

I started with setting up my types, which were extremely simple. I only needed a few fields to get started:

type Makelaar interface {
	// makelaars have a check() method which returns multiple ResultRoom structs
	Check() []models.DiscoveredRoom

	// also a name() method which returns the name of the makelaar
	Name() string
}

type DiscoveredRoom struct {
	Name        string  `json:"name"`
	MonthlyRate float64 `json:"monthly_rate"`
	City        string  `json:"city"`
}

func (r DiscoveredRoom) String() string {
	return r.Name + " (" + r.City + ") - " + strconv.Itoa(int(r.MonthlyRate)) + " EUR/month"
}

And from there, I implemented 21 different real estate agents, each with their own implementation of the Check() method. Some of them were simple curl wrappers, while others relied on chromedp to puppeteer an actual Chrome instance (often enough to get around Cloudflare "protection").

I used ntfy, an amazing little service that allows you to send push notifications to your phone with a simple HTTP request, to send me a notification when a new listing became available. The whole point of this was to save resources—no way in hell I was going to set up an Apple notification service for this.

Paying it forward

It only took a few weeks before I had a few hundred listings in my database and struck absolute gold with one of them (where I currently live).

But I was also aware that this was a problem that many people were facing, and I wanted to help them out. So, I set up a Twitter bot that would post new listings the second they became available on multiple trusted sites, so my fellow students could get notifications and shoot their shot at a new place to live.

This bot was called @IkWilEenkamer, and ended up posting over 6,000 listings before daddy Elon decided that APIs are for pussies and shut it down.

The privilege of outsmarting a broken system

Let me be crystal clear about something: everything I just described is absolutely fucked up, and I should never have needed to do any of this in the first place.

The fact that a 22-year-old student can build a crawler that outperforms the "official" housing market shows just how fundamentally broken the system is. While I was having fun playing tech wizard, thousands of other students were getting scammed, paying subscription fees to fake sites, and sleeping on couches because they couldn't compete with automated systems.

Think about the absurdity of it all: I spent weeks building a system to automate applying to apartments because the market is so competitive that humans can't keep up. This isn't innovation—it's dystopian bullshit. We've created a system where having programming skills gives you a massive advantage in finding basic shelter. What's next, needing to pass a coding interview to get groceries?

The housing market has become a game where the rules are rigged against ordinary people. Real estate agents are making bank on subscription fees for access to crawled data. Landlords are overwhelmed with applications and just pick whoever responds first. And meanwhile, students are taking out loans not just for education, but to pay for the privilege of maybe seeing a listing that might be real.

The worst part? Most people can't even fight back the way I did. Not everyone has the time, skills, or resources to build their own crawler army. The system literally rewards technical privilege while punishing everyone else for not being able to code their way out of homelessness.

This is what happens when we let market forces run wild without regulation. Housing—a basic human need—becomes a commodity to be gamed, scraped, and automated. The Netherlands prides itself on being progressive, but we've created a rental market that would make Silicon Valley landlords blush.

And that's just the tech tip of the iceberg. May I point out that you've spent a couple of minutes reading this, and we haven't even covered sketchy figures? S.O. to my bestie, prins bernhard.

Epilogue: The house always wins (except when it doesn't)

So there you have it. Did I solve the housing crisis? Absolutely not. Did I get a decent apartment and help a few other students along the way? You bet your overpriced security deposit I did.

The real kicker is that this whole adventure took me about two weeks of coding, while some of my friends spent months refreshing Kamernet and paying for premium subscriptions to sites that were literally just reselling the same crawled data. The system is so broken that a sleep-deprived student with a laptop can outperform the entire "professional" housing market.

But hey, at least I learned some valuable life lessons along the way: always read the HTML source code of sketchy websites, never trust a site that charges money just to look at listings, and sometimes the best way to beat the system is to become the system (but with better ethics and fewer subscription fees).

Now if you'll excuse me, I need to go update my LinkedIn to add "Housing Market Disruptor" to my skill set. Who knows? Maybe I'll pivot to a career in PropTech and really lean into the whole "move fast and break things" mentality—except instead of breaking things, I'll caching in my share of a fundamentally broken market.

Or maybe I'll just stick to regular software development. After all, debugging code is way easier than debugging society.

P.S. - If you're a real estate agent reading this and getting angry, maybe consider not charging people money to see listings that you didn't even create. Just a thought.

The Sauce Code (#OpenSauceSoftware)

It's a little crappy, but still works thanks to the fact that real estate agents are lazy and don't change their APIs often. So here are 3 prominent examples of simple crawler implementations that don't even need a Chromium instance to run.

Devilee

package makelaars

import (
	"encoding/json"
	"fmt"
	"net/http"
	"room-notifications/models"
	"room-notifications/rules"
)

type DevileeResponse []struct {
	URL            string  `json:"url"`
	Photo          string  `json:"photo"`
	Address        string  `json:"address"`
	State          string  `json:"state"`
	Zipcode        string  `json:"zipcode"`
	City           string  `json:"city"`
	District       string  `json:"district"`
	Country        string  `json:"country"`
	IsSales        bool    `json:"isSales"`
	IsRentals      bool    `json:"isRentals"`
	Price          string  `json:"price"`
	SalesPrice     int     `json:"salesPrice"`
	RentalsPrice   int     `json:"rentalsPrice"`
	LivingSurface  int     `json:"livingSurface"`
	PlotSurface    int     `json:"plotSurface"`
	Volume         int     `json:"volume"`
	Rooms          int     `json:"rooms"`
	Bedrooms       int     `json:"bedrooms"`
	IsFurnished    bool    `json:"isFurnished"`
	IsDecorated    bool    `json:"isDecorated"`
	IsShell        bool    `json:"isShell"`
	IsRenovated    bool    `json:"isRenovated"`
	IsInvestment   bool    `json:"isInvestment"`
	IsBought       bool    `json:"isBought"`
	IsRented       bool    `json:"isRented"`
	IsExpected     bool    `json:"isExpected"`
	IsAuction      bool    `json:"isAuction"`
	IsLowestOffer  bool    `json:"isLowestOffer"`
	HasOpenHouse   bool    `json:"hasOpenHouse"`
	IsNew          bool    `json:"isNew"`
	Status         string  `json:"status"`
	StatusOrig     string  `json:"statusOrig"`
	MainType       string  `json:"mainType"`
	BuildType      string  `json:"buildType"`
	Lat            float64 `json:"lat"`
	Lng            float64 `json:"lng"`
	Added          int     `json:"added"`
	Changed        int     `json:"changed"`
	Tags           []any   `json:"tags"`
	Spotlight      string  `json:"spotlight"`
	Categories     bool    `json:"categories"`
	CategoriesOrig []any   `json:"categoriesOrig"`
}

// set of known urls
type Devilee struct {
	Makelaar
}

// name() method which returns the name of the makelaar
func (d Devilee) Name() string {
	return "Devilee"
}

var devileeFoundIds = []string{}

// check() method which returns multiple ResultRoom structs
func (d Devilee) Check() []models.DiscoveredRoom {
	response, err := http.Get("https://www.devilee.nl/nl/realtime-listings/consumer")
	if err != nil {
		fmt.Println("Error making GET request:", err)
		return []models.DiscoveredRoom{}
	}
	defer response.Body.Close()

	var items DevileeResponse
	err = json.NewDecoder(response.Body).Decode(&items)
	if err != nil {
		fmt.Println("Error decoding JSON:", err)
		return []models.DiscoveredRoom{}
	}

	// check if we have any new rooms (not in knownUrls)
	var newRooms []models.DiscoveredRoom
	var foundButNotNew = 0
	for _, item := range items {
		if !contains(devileeFoundIds, item.URL) {

			r := models.DiscoveredRoom{
				Name:        item.Address,
				MonthlyRate: float64(item.RentalsPrice),
				City:        item.City,
			}

			// match rules, based on what im interested in
			if item.StatusOrig != "under_option" && item.StatusOrig != "rented" && item.StatusOrig != "sold" && item.IsRentals && item.Volume > 0 && rules.IsInteresting(r) {
				newRooms = append(newRooms, r)
			}

			devileeFoundIds = append(devileeFoundIds, item.URL)
		} else {
			foundButNotNew++
		}
	}

	fmt.Println("DEVILEE: Found", len(items), "rooms, of which", foundButNotNew, "were already known")

	// parse into ResultRoom structs
	return newRooms
}

Immovita

package makelaars

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"room-notifications/models"
	"room-notifications/rules"
)

type Immovita struct {
	Makelaar
}

func (i Immovita) Name() string {
	return "Immovita"
}

var immovitaFoundIds = []int{}

func (i Immovita) Check() []models.DiscoveredRoom {
	url := "https://immovita.nl/wp-content/themes/immovita/pararius/index.php?script&fields=lat,lng,firstphoto,state,street,price,city&lang=nl"

	response, err := http.Get(url)
	if err != nil {
		fmt.Println("Error:", err)
		return []models.DiscoveredRoom{}
	}
	defer response.Body.Close()

	body, err := ioutil.ReadAll(response.Body)
	if err != nil {
		fmt.Println("Error:", err)
		return []models.DiscoveredRoom{}
	}

	var jsonStr = string(body)
	// replace "var data = " with "" to make it valid json
	jsonStr = jsonStr[11:]

	// convert back to byte array
	var jsonBytes = []byte(jsonStr)

	var properties []ImmovitaProperty
	err = json.Unmarshal(jsonBytes, &properties)
	if err != nil {
		fmt.Println("Error:", err)
		return []models.DiscoveredRoom{}
	}

	var foundButNotNew = 0
	var rooms []models.DiscoveredRoom
	for _, prop := range properties {
		r := models.DiscoveredRoom{
			Name:        prop.Street,
			MonthlyRate: float64(prop.Price),
			City:        prop.City,
		}

		// is this a new room?
		if !containsInt(immovitaFoundIds, prop.ID) {
			// is it in our price range?
			if rules.IsInteresting(r) {
				rooms = append(rooms, r)
			}
			immovitaFoundIds = append(immovitaFoundIds, prop.ID)
		} else {
			foundButNotNew++
		}
	}

	fmt.Println("IMMOVITA: Found", len(rooms), "new rooms on", i.Name(), "(", foundButNotNew, "found but not new)")

	return rooms
}

type ImmovitaProperty struct {
	ID         int    `json:"id"`
	Lat        string `json:"lat"`
	Lng        string `json:"lng"`
	FirstPhoto string `json:"firstphoto"`
	Street     string `json:"street"`
	Price      int    `json:"price"`
	City       string `json:"city"`
}

Woltershousing

package makelaars

import (
	"encoding/json"
	"fmt"
	"net/http"
	"room-notifications/models"
	"room-notifications/rules"
)

type WoltersHousingResponse []struct {
	URL            string  `json:"url"`
	Photo          string  `json:"photo"`
	Address        string  `json:"address"`
	State          string  `json:"state"`
	Zipcode        string  `json:"zipcode"`
	City           string  `json:"city"`
	District       string  `json:"district"`
	Country        string  `json:"country"`
	IsSales        bool    `json:"isSales"`
	IsRentals      bool    `json:"isRentals"`
	Price          string  `json:"price"`
	SalesPrice     int     `json:"salesPrice"`
	RentalsPrice   int     `json:"rentalsPrice"`
	LivingSurface  int     `json:"livingSurface"`
	PlotSurface    int     `json:"plotSurface"`
	Volume         int     `json:"volume"`
	Rooms          int     `json:"rooms"`
	Bedrooms       int     `json:"bedrooms"`
	IsFurnished    bool    `json:"isFurnished"`
	IsDecorated    bool    `json:"isDecorated"`
	IsShell        bool    `json:"isShell"`
	IsRenovated    bool    `json:"isRenovated"`
	IsInvestment   bool    `json:"isInvestment"`
	IsBought       bool    `json:"isBought"`
	IsRented       bool    `json:"isRented"`
	IsExpected     bool    `json:"isExpected"`
	IsAuction      bool    `json:"isAuction"`
	IsLowestOffer  bool    `json:"isLowestOffer"`
	HasOpenHouse   bool    `json:"hasOpenHouse"`
	IsNew          bool    `json:"isNew"`
	Status         string  `json:"status"`
	StatusOrig     string  `json:"statusOrig"`
	MainType       string  `json:"mainType"`
	BuildType      string  `json:"buildType"`
	Lat            float64 `json:"lat"`
	Lng            float64 `json:"lng"`
	Added          int     `json:"added"`
	Changed        int     `json:"changed"`
	Tags           []any   `json:"tags"`
	Spotlight      string  `json:"spotlight"`
	Categories     bool    `json:"categories"`
	CategoriesOrig []any   `json:"categoriesOrig"`
}

// set of known urls
type WoltersHousing struct {
	Makelaar,
	knownUrls []string
}

// name() method which returns the name of the makelaar
func (d WoltersHousing) Name() string {
	return "WoltersHousing"
}

// check() method which returns multiple ResultRoom structs
func (d WoltersHousing) Check() []models.DiscoveredRoom {
	response, err := http.Get("https://www.woltershousing.nl/nl/realtime-listings/consumer")
	if err != nil {
		fmt.Println("Error making GET request:", err)
		return []models.DiscoveredRoom{}
	}
	defer response.Body.Close()

	var items WoltersHousingResponse
	err = json.NewDecoder(response.Body).Decode(&items)
	if err != nil {
		fmt.Println("Error decoding JSON:", err)
		return []models.DiscoveredRoom{}
	}

	// check if we have any new rooms (not in knownUrls)
	var newRooms []models.DiscoveredRoom
	var foundButNotNew = 0
	for _, item := range items {
		if !contains(d.knownUrls, item.URL) {

			r := models.DiscoveredRoom{
				Name:        item.Address,
				MonthlyRate: float64(item.RentalsPrice),
				City:        item.City,
			}

			// match rules, based on what im interested in
			if item.StatusOrig != "under_option" && item.StatusOrig != "rented" && item.StatusOrig != "sold" && item.IsRentals && item.Volume > 0 && rules.IsInteresting(r) {
				newRooms = append(newRooms, r)
			}

			d.knownUrls = append(d.knownUrls, item.URL)
		} else {
			foundButNotNew++
		}
	}

	fmt.Println("WoltersHousing: Found", len(items), "rooms, of which", foundButNotNew, "were already known")

	return newRooms
}