Arweave+IPFS: Persistence for the InterPlanetary File System
TL;DR: Arweave is launching an IPFS incentivisation layer, providing permanent storage for the network. The Arweave+IPFS bridge has been live on the Arweave mainnet and in testing with selected partners for months — now it is ready for the wider decentralised web community.
With the current web becoming increasingly centralised and fragile, an ecosystem of decentralised data storage platforms is emerging — one of these being IPFS. Started by Protocol Labs in 2014, the IPFS project aims to enable the original vision of the open, distributed web by offering a content- rather than location-addressed alternative to the centralised web.
With traditional location-based addressing, if you want to retrieve a file or piece of data on the current web, you will input the IP address/domain name which makes a request to the server, and your content will be downloaded and rendered. However, if the specific server you’re trying to access is down and you can’t get the piece of data you want, there’s still a chance that somebody, somewhere else on the web has this content. The problem is, you can’t access it because your computer doesn’t know where to find it. With a platform like IPFS, instead of asking “Is this content on your server, Alice?” as you do on the traditional web, you say to the network: “Hey everyone, please send me this content” by using the unique hash.
Currently, IPFS suffers from a major problem: storage in the network cannot be easily incentivised. This lack of incentivisation means that in the vast majority of cases, only one copy of data is stored within IPFS. There is also little incentive for nodes to maintain long term backups of data on the network. Coupled with this, theoretically, files could also ‘disappear’ over time, as nodes may choose to clear their cached data to save space. At current adoption levels this isn’t a significant issue but in the long term, backing up large amounts of data requires strong economic incentives. For example, Pinata offers a centralised data ‘pinning’ service, and Filecoin have been working on a temporary file storage solution, but truly persistent and decentralised storage incentivisation has remained elusive.
Subsequently, we are happy to release the Arweave+IPFS bridge: a mechanism for harnessing some of the Arweave’s data permanence incentives for content stored inside the IPFS ecosystem. This Arweave+IPFS bridge allows you to have truly permanent backing of your data using Arweave, while also making it available in IPFS. It is integrations like these that will form the foundations of the future open, decentralised web and its surrounding ecosystem.
At its core, the Arweave+IPFS bridge system is split into two components: a user-side ‘upload service’, and a node-side ‘IPFS extension’. When a user wants to incentivise the storage of a piece of content on IPFS, they access an uploader service (initially via a permaweb app, or an Arweave node’s console) which grabs the data through an IPFS gateway. Once the data has been retrieved from IPFS, the uploader service signs it into a new Arweave transaction with an ‘IPFS-Add’ tag, then dispatches it to the Arweave network. Once the transaction has been mined into an Arweave block, nodes running the Arweave+IPFS extension add and pin the data to their local IPFS node. In this way, data is retrieved from the IPFS network, then re-uploaded and exposed through many more nodes on both IPFS and Arweave. Developers can also hook into the Arweave+IPFS bridge by building their own uploading services, allowing simple programmatic access to incentivised IPFS storage.
After successfully running several Arweave+IPFS bridge nodes for over 6 months, we’re now ready to open the system up to the public and begin onboarding many more users. The main limitation of the current implementation is that, as with the proposed bridge between Filecoin+IPFS, it lacks explicit data transfer incentives to encourage the bridge nodes to serve the data they’re hosting back into IPFS itself. However, even if a very small proportion of nodes in the Arweave network choose to run the optional ‘+IPFS’ extension, this is still sufficient to successfully ‘seed’ the data inside IPFS. Further, we will consult with the community about the possibility of building compatibility with the core IPFS data access APIs into Arweave itself — allowing nodes to enforce the usual Wildfire data distribution incentives to Arweave+IPFS.
Alone, no project or protocol will be able to meet all the needs of the decentralised web ecosystem. In order for the ecosystem to succeed in reaching mainstream adoption, it must meet all of the varied needs of developers and their users. By releasing the first version of the Arweave+IPFS bridge system, we are excited to be adding a permanence incentivisation layer to further contribute to the emerging decentralised web ecosystem.
If you’re a developer interested in getting involved with building the decentralised web, check out our ongoing open source hackathon!
-arweave-team
