We are specialists in building digital products that require blockchain implementation.
17.3.2022. Polygon outage postmortem
Shard LabsMar 17
The following post-mortem is written by the Shard Labs team. It highlights all the information that the community needs to know about the latest events that happened with the Polygon network and Shard Labs validator — the events that have caused an interruption in Lido accruing rewards Mar 10–14, 2022.
Before we get into a detailed analysis, here’s a short summary of the events that occurred:
our validator stopped working temporarily
bridging was disabled but is now functional again
no funds were lost
On Mar 14, 2022 we made a public announcement on our Twitter regarding the issue that occurred.
🗣️ The @0xPolygon PoS had an implementation problem that caused the bridge to stop working.
A short term fix already applied and Lido for Polygon is accruing rewards again. 💧 @LidoFinance
On March 10, 2022 Polygon announced that the Polygon PoS chain had an implementation problem that caused the bridge to stop working. More information available here.
After this happened, rewards stopped accruing for the whole validator set because the Heimdall layer was not able to submit checkpoints to Ethereum mainnet (more details here).
This issue lasted for ~3 days until the hotfix went live. More information about the hotfix is available here.
This means that stMATIC did not gain in value from March 10–14, 2022.
A short-term fix was applied ASAP, and Lido on Polygon is accruing rewards again.
This issue happened unfortunately because the Polygon network is currently difficult to maintain due to short block time and numerous chain reorganizations. Because of this, Shard Labs’ validator, which is currently the only node operator in Lido on Polygon, got the wrong chain state data after applying the hotfix.
Conclusion and Next Steps
As an interim fix from March 14th, 2022 onwards, Shard Labs’ infrastructure is being operated by one of the most reliable Polygon node operators until Shard Labs manages to fix the state sync issue and rolls out its more robust infrastructure.
In order to anticipate similar future scenarios such as this, Shard Labs is working on having a self-hosted failover solution for timelier recovery.
In Phase 2 of Lido on Polygon (mid-April, 2022), the first batch of additional node operators will be onboarded, which will improve the reliability and robustness of the whole protocol. If one of the validators has an issue, others will still receive rewards and the overall effect on stMATIC yield will be lessened.
We want to thank everyone that contributed to identifying the issue and finding a solution.
As always, we will give our best to update you with all the changes in a prompt manner. You can keep up with all the updates through our Twitter account. If you have any questions, join this Discord channel!
This article was originally published by Shard Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.
If you missed BlockSplit last year, then this year it should definitely be on your list! Why? For two reasons: BlockSplit Conference is a premier event that brings together blockchain and cryptocurrency enthusiasts, industry leaders, developers, and innovators from around the world. The conference features a diverse range of speakers who are experts in various […]
Blockchain may seem like a daunting technology to learn, but just like everything else – it is not hard if you start on the right track and with the right resources 😊 So, if you want to make a career as a blockchain developer here is where you should start! Skills Before we start with […]
As many of you know, ETHDenver is an annual Ethereum hackathon + conference that takes place in Denver, Colorado. This event brings together developers, designers, and entrepreneurs from around the world, to meet and build decentralized applications and other innovative solutions on the Ethereum blockchain. This year it was held from Feb 24 to Mar […]