Making Sync Faster

Dear Smeshers,

As many of you are by now aware, it’s gotten pretty hard to sync a Spacemesh node, and to stay in sync. Today we’re excited to release a new “quick sync” mode for go-spacemesh and Smapp that will make this easier. Before I explain how this works and the associated tradeoffs, let me explain a little bit about how we got here and how we’re going to fix this.

Tldr: the Spacemesh network has grown much, much faster than any of us anticipated. To put things in perspective, we thought we’d see on the order of 1,000-10,000 ATXs (miner identities) during the first year of operation. We designed the network to handle around 100,000 ATXs and figured we’d have plenty of time to redesign and improve things long before that number was exceeded. And we set 1,000,000 ATXs as our long term, “someday maybe” stretch goal.

In the current epoch, epoch 15, a mere half year post-genesis, we’ve already exceeded 1,000,000 ATXs. This is truly exciting. It means the network is much more active and much more secure than it would be if it were smaller. It means more people know about and are a part of Spacemesh than we imagined would be true at this point in time. But it’s also made keeping up with the breakneck pace of growth—around 40% epoch-on-epoch—enormously challenging.

We’re really proud and excited that, in spite of the unprecedented and unexpected nature of this growth, we’ve been able to keep up and to date the network has experienced negligible downtime. It may not always be immediately obvious because so many of the upgrades we’ve made are “under the hood” and invisible to the casual user. The Spacemesh protocol and network have evolved enormously since we launched and are far more mature than they were! But one of the unfortunate consequences of this growth is that it’s become harder and harder for node operators to keep up, especially those running nodes at home.

One of the main issues is that ATXs contain miner eligibility proofs (proofs of space-time), and these proofs are computationally expensive to verify. It takes around 150-200ms to verify a single proof on a modern CPU, which means that it currently takes almost two days to fully process and verify all of the ATXs and miner proofs for a single epoch on a single core (proportionally faster on multiple cores, but still too darn long).

Another issue is that our P2P network and sync protocol weren’t designed to handle so much data! The sync protocol naively assumed that you could safely encode all of the ATXs for a given epoch into a single object, which works fine with 10,000 ATXs but not so well with two orders of magnitude more!

We’re working on several major upgrades that collectively will address these issues once and for all:

• Streaming sync will allow arbitrarily large data sets, even those containing millions of ATXs, to be synced successfully across the network
• Distributed verification will allow each individual node to verify only a tiny fraction of those millions of ATXs each epoch, distributing the load among all the miners
• Set reconciliation will allow sync to happen faster and more effectively and will do a better job of making sure that every node sees every data item
• Better P2P decentralization will allow faster discovery of other peers in the network

You can learn more about each of these upgrades and track their progress on our public dev team kanban. We expect all of these to be rolled out over the next month.

Back to quick sync. Today we’re rolling out quick sync as a quick-and-dirty workaround to the sync issues that many miners are experiencing, and that won’t be fully addressed until the above upgrades are in place in a few weeks. Normally when syncing a new Spacemesh node, the node downloads all historical data—all blocks, transactions, ATXs, etc.—for all of history from peers, independently verifies each of them, and reconstructs the current state from scratch. Quick sync will allow a node to bypass this process entirely and instead download a snapshot of the current state from a trusted source (from us, the Spacemesh core team, or from another source you trust). It will allow a node to go from zero to being in sync in a matter of hours rather than taking days or, in some cases, being entirely unable to sync from scratch due to the issues outlined above.

Cypherpunks and blockchain purists may wonder, is it really okay to download a state snapshot rather than downloading and verifying every proof, transaction and state transition for yourself? The honest answer is “no.” As we’ve said many times, blockchain is all about trust minimization. You shouldn’t take our word, or anyone else’s word, about the state of the blockchain or the state of the world. You should do the work to verify it for yourself. That’s what we’re all here for, after all.

But this pure approach needs to be balanced against the reality of the situation that it’s too darn hard to sync a Spacemesh node right now. Quick sync offers a practical shortcut for node operators that are happy to take our word for the current state. Importantly, it doesn’t remove their ability to independently verify that state—that can be done in the background, after downloading the state snapshot. In the future quick sync will do this work for you automatically.

So, how do you use quick sync? If you’re running Smapp it will kick in automatically when you first sync or if your node falls too far behind. You’ll see a window like this with a link to a document explaining the tradeoffs:

If you’re running go-spacemesh manually you’ll need to use the standalone quicksync-rs tool. See the README in that repository for more information on how to use the tool.

Quick recap:

• It’s pretty hard to sync a Spacemesh node from scratch today. We know how to make it easier and we’re working on it but it will take a few weeks to fix.
• As a short-term fix, we’re rolling out quick sync today. We do not recommend its use to those operating critical infrastructure, such as an exchange, that rely on always knowing the precisely accurate state at every moment in time. We do recommend it to casual home node operators who are otherwise struggling to stay in sync.
• We’ll continue to work on improving quick sync, adding features like automatic background verification of all downloaded data and integrating it more cleanly into go-spacemesh.

As always, don’t hesitate to reach out with comments, concerns, or ideas on Discord or on our Community Forum.