Sia Technical Update and Roadmap
Hard for me to believe this, but we’ve been working on Sia now for more than 4 years. Even harder for me to believe, of the 4 major competitors that existed in 2014 (Sia, Storj, Filecoin, Maidsafe), only one has successfully launched a fully decentralized platform for cloud storage. That was us, and we pulled it off in 2015.
That said, you’d have a difficult time convincing anyone that what we released in 2015 was a finished product. We’ve come a long way since then, and we have a long way to go.
Today, Sia is able to store multiple terabytes of data from a single client, achieve sustained speeds above 100mbps for both upload and download, and can manage tens of thousands of files. And, in v1.3.3 we will be releasing official API support for video streaming from the Sia platform.
In the near road ahead, we will be focusing in the short term on the accuracy of renter spending reports. Today, the broad reporting is outright incorrect, and it appears that the per-contract reporting is also incorrect, often overlooking spending. We’ve fixed some of this already, but are performing a full audit of the contractor spending reporting (in addition to expanding the test suite to more extensively cover reporting) to make sure that accounting is correct from v1.3.4 forward (some of the fixes will not be ready in time for v1.3.3).
Another thing that we are focusing on is crash-resilience of the contractor. Crash resilience ends up being extremely difficult to achieve for decentralized networks, especially ones as complicated as Sia. The core problem is consumer hard drives. If a computer suddenly loses power or is turned off abruptly, data may only be partially written to disk, causing corruption. To make matters worse, the hard drive and operating system will actually report to the code that a write is complete even if the data is not safe from a sudden loss of power. Sia is currently very good in this regard — most everything that happens on Sia is resilient to unexpected crashes, however we need to get the final testing in place for the contractor.
The primary goal for v1.3.4 is going to be file backups. Today on Sia, if the host machine is corrupted, all of the data gets corrupted as well. A developer (myself or my cofounder) would be able to take an old backup of the metadata and recover the files, but it would be a lot of manual, highly expert work. We’re going to be changing that in v1.3.4, making it possible to back up your metadata such that you can recover your files in the event that the host machine crashes, removing the single point of failure that currently exists for most people.
As many people are aware, file backups and file sharing on Sia are actually roughly the same thing. You can think of filesharing as giving a file backup to a friend. Or you can think of a file backup as sharing a file with yourself. We will be focused on making the endpoints for backups in v1.3.4, and then providing basic support for file sharing in v1.3.5. Our target for v1.3.6 is a recursive metadata compression technique that should make full system backups small enough to be put on the blockchain itself. We can encrypt those backups with your seed and put them on the blockchain such that you will be able to recover your data using nothing but your wallet seed. This will require making infrequent transactions on-chain for backups, and also means that backups will not be continuous, however it’s a big step forward.
The changes to the file format to enable file backups, file sharing, and seed-based file recovery will also lead to greater file scalability for the renter, increasing the renter’s capabilities from tens of thousands of files to millions of files. These changes should also substantially increase the total amount of data that the renter is capable of supporting.
In the very long term, we do know how to make seed based file recovery continuous. It’s… a huge pain and a massive engineering effort. I’m guessing it’s going to be more work than creating the entirety of the Sia platform to date. That said, we know how to do it, and we know how to make it reasonably efficient. That would mean that at any point, you could log into any computer, install Sia, and gain access to your full, up-to-date data. This is something that is definitely on our roadmap, but not something we are likely to complete in 2018 or even 2019.
We have hired a full time developer who will be working on the UI starting in the next month. UI development has been all but frozen over the past year due to our primary UI developer leaving, however this gap in our team has been filled! Expect good things moving forward.
Within the broader cryptocurrency community, we’re going to start reaching out to more projects such as RNDR and Blockstack so that Sia may be the file backbone for the cryptocurrency space. The biggest thing holding back many projects today is filesharing and backups, and we will be enabling both of those in the coming months.
One other huge focus for us over the next few months is going to be an on-chain scalability solution that I’ve been working on quietly over the past year. I’ve called the solution microchains, and I believe we can leverage it to gain 10,000x on-chain scalability for Sia, among a myriad of other benefits. I’ll be releasing regular blogposts explaining microchains and exploring the various advantages they have over traditional blockchains. If all goes well, we hope to be writing some microchains based code by the end of the summer.
Overall, 2018 is going to be a busy year for us. As always, we’re looking at what we believe is the most long-term critical elements for the platform, and building Sia to be a platform that’s likely to still be in use 50 years from now.
