Link Rot: The Web is Decaying

This is a guest blog post from the Decentralised Public Library, an Arweave initiative.

What is link rot?

Link rot is surprisingly common — across major social media platforms 30% of shared links are totally dead within two years. We all imagine the internet to be timeless — after all, sometimes it seems like removing something from the internet is nearly impossible. However, this is simply not true — the traditional web is shockingly fragile and not at all immortal.

Over 20 years, 98.4% of web links suffer from rot, becoming totally inaccessible to future generations.

It’s not just casually-shared social media links that are at risk of rot though. Even the most prestigious institutions in the world really struggle with link rot — 50% of U.S. Supreme Court opinions contain dead links, and so do 70% of Harvard academic journals. Both legal judgements and academic research rely very heavily on access to historic evidence decades into the future — link rot jeopardises this.

Why do links rot?

Websites don’t maintain themselves, and ultimately unless people work to preserve them and keep them accessible, they will all fall offline eventually.

Something as simple as a site changing its domain name will trigger any links back to the old domain to rot away, often with no way for the reader to find the new location.

Although more rare, there are also deeply sinister, political causes of link rot, like government censorship. Some governments dedicate a huge amount of time and resources to information control, attempting to take down websites to hide information from their citizens. For instance, China’s government has targeted scientific and academic publications extensively for takedowns of specific articles in recent years, and both Springer and Cambridge University Press have bowed to this pressure. Even the USA has been accused of censoring climate change information from government websites for political reasons.

How can we fight link rot?

Firstly, we should be aware of the important work of projects like the Internet Archive’s Wayback Machine, which crawls the web and makes backups of millions of websites. If you encounter a 404 error out in the wild, you should always check the URL with the Wayback Machine to see if there’s an archived version available. They even have a nifty browser extension (for Chrome and Firefox) that automatically replaces broken links with archived sites, if they happen to have a backup of that page. There are also national web archives that perform similar functions — for example, the UK Web Archive, though they focus heavily on UK websites, and aim to capture pages just once per year. Remember, most web archiving projects accept requests so let them know if a site you need backed up is missing!

Also, you can begin to permanently archive the decaying traditional web yourself! One option is the Arweave web extension for Chrome, which allows you to permanently capture snapshots of webpages storing them immediately on the ‘permaweb’.

The permaweb is similar to the traditional web, except permaweb content cannot be lost, altered, or purposefully deleted. The links never change, so you know that you won’t suffer from link rot!

Also, for content creators and developers it’s brilliant, because you don’t have to waste your time configuring servers, maintaining domains, or repeatedly paying to host content year after year. You can check out more about the permaweb here, and play around with some free archiving tokens here. As you may have noticed, most of the links in this article are actually permaweb snapshots of traditional webpages, placing them beyond the reach of accidental and intentional alteration and loss.

Once you’ve claimed your free tokens, it’s super easy to start archiving with the Arweave web extension. Check out our handy step-by-step guide here: https://docs.arweave.org/info/archiving/step-by-step-beginners-guide

No more 404!

A novel data storage blockchain protocol enabling a permanent serverless web and creating truly permanent data storage for the first time.