We are specialists in building digital products that require blockchain implementation.
17.3.2022. Polygon outage postmortem
Shard LabsMar 17
The following post-mortem is written by the Shard Labs team. It highlights all the information that the community needs to know about the latest events that happened with the Polygon network and Shard Labs validator — the events that have caused an interruption in Lido accruing rewards Mar 10–14, 2022.
Before we get into a detailed analysis, here’s a short summary of the events that occurred:
our validator stopped working temporarily
bridging was disabled but is now functional again
no funds were lost
On Mar 14, 2022 we made a public announcement on our Twitter regarding the issue that occurred.
?️ The @0xPolygon PoS had an implementation problem that caused the bridge to stop working.
A short term fix already applied and Lido for Polygon is accruing rewards again. ? @LidoFinance
On March 10, 2022 Polygon announced that the Polygon PoS chain had an implementation problem that caused the bridge to stop working. More information available here.
After this happened, rewards stopped accruing for the whole validator set because the Heimdall layer was not able to submit checkpoints to Ethereum mainnet (more details here).
This issue lasted for ~3 days until the hotfix went live. More information about the hotfix is available here.
This means that stMATIC did not gain in value from March 10–14, 2022.
A short-term fix was applied ASAP, and Lido on Polygon is accruing rewards again.
This issue happened unfortunately because the Polygon network is currently difficult to maintain due to short block time and numerous chain reorganizations. Because of this, Shard Labs’ validator, which is currently the only node operator in Lido on Polygon, got the wrong chain state data after applying the hotfix.
Conclusion and Next Steps
As an interim fix from March 14th, 2022 onwards, Shard Labs’ infrastructure is being operated by one of the most reliable Polygon node operators until Shard Labs manages to fix the state sync issue and rolls out its more robust infrastructure.
In order to anticipate similar future scenarios such as this, Shard Labs is working on having a self-hosted failover solution for timelier recovery.
In Phase 2 of Lido on Polygon (mid-April, 2022), the first batch of additional node operators will be onboarded, which will improve the reliability and robustness of the whole protocol. If one of the validators has an issue, others will still receive rewards and the overall effect on stMATIC yield will be lessened.
We want to thank everyone that contributed to identifying the issue and finding a solution.
As always, we will give our best to update you with all the changes in a prompt manner. You can keep up with all the updates through our Twitter account. If you have any questions, join this Discord channel!
This article was originally published by Shard Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.
With the recent market events, you probably noticed that the term Proof-of-Reserves or PoR emerges every time any crypto exchange is mentioned. To learn what is PoR, how it’s used, and why it matters continue reading this article. What is Proof-of-Reserves (PoR)? * It is not a consensus mechanism. Proof-of-Reserves, or short PoR is a […]
It is very clear that there is a global downturn in the markets, and we expect 18 months up to 3 years of downturn, potential recession on the global markets, etc. This is the first time ever that a global market recession overlaps with a crypto bear market. As we are approaching stage 3 of […]
If you came across some of Shard Labs’ tweets or articles, you’ve probably heard the big news: Now you can stake $MATIC in a decentralized, secure way and immediately get the representation of their share in the form of stMATIC token without maintaining staking infrastructure. Awesome, right? ? ? ? LIDO is coming to #[email protected]_labs […]