Develoraptor Blog


Indexing the Weird Wild Web

Posted by

In the days when the web was much smaller, before search engines became the de facto way to find things online, many people used web directories to navigate the internet. These were lists of links, often categorised, maintained manually by human editors.

As the size of the world wide web began to grow faster, directory operators found it hard to keep their lists up-to-date. At the same time, search engines were becoming more accurate and powerful, and very good at finding the specific thing you were looking for. Predictably enough, the directory sites began to drop off the internet or otherwise fade into obscurity, and search became king.

These days, however, the web is a different place. Most users probably only want to visit a handful of well-known websites to socialise or do some shopping, read the news or find out about the latest trend. Search engines have adapted to this, but at the expense of the more weirder, wilder side of the web. Now smaller sites and niche interests struggle to claw back search ranking from clickbait news articles and online mega shops, and the internet becomes a colder place every day.

Perhaps then it's time to bring back web directories, as a small but vibrant network of human-curated links to things you wouldn't otherwise find. Whether it's a small list of favourite blogs or a meticulously categorised database of sites without cookies or ads - suddenly the web seems more interesting again.

In fact I'm sure there are plenty of them out there - I just wouldn't know how to find them.