We are specialists in building digital products that require blockchain implementation.
17.3.2022. Polygon outage postmortem
Shard LabsMar 17
The following post-mortem is written by the Shard Labs team. It highlights all the information that the community needs to know about the latest events that happened with the Polygon network and Shard Labs validator — the events that have caused an interruption in Lido accruing rewards Mar 10–14, 2022.
Before we get into a detailed analysis, here’s a short summary of the events that occurred:
our validator stopped working temporarily
bridging was disabled but is now functional again
no funds were lost
On Mar 14, 2022 we made a public announcement on our Twitter regarding the issue that occurred.
🗣️ The @0xPolygon PoS had an implementation problem that caused the bridge to stop working.
A short term fix already applied and Lido for Polygon is accruing rewards again. 💧 @LidoFinance
On March 10, 2022 Polygon announced that the Polygon PoS chain had an implementation problem that caused the bridge to stop working. More information available here.
After this happened, rewards stopped accruing for the whole validator set because the Heimdall layer was not able to submit checkpoints to Ethereum mainnet (more details here).
This issue lasted for ~3 days until the hotfix went live. More information about the hotfix is available here.
This means that stMATIC did not gain in value from March 10–14, 2022.
A short-term fix was applied ASAP, and Lido on Polygon is accruing rewards again.
This issue happened unfortunately because the Polygon network is currently difficult to maintain due to short block time and numerous chain reorganizations. Because of this, Shard Labs’ validator, which is currently the only node operator in Lido on Polygon, got the wrong chain state data after applying the hotfix.
Conclusion and Next Steps
As an interim fix from March 14th, 2022 onwards, Shard Labs’ infrastructure is being operated by one of the most reliable Polygon node operators until Shard Labs manages to fix the state sync issue and rolls out its more robust infrastructure.
In order to anticipate similar future scenarios such as this, Shard Labs is working on having a self-hosted failover solution for timelier recovery.
In Phase 2 of Lido on Polygon (mid-April, 2022), the first batch of additional node operators will be onboarded, which will improve the reliability and robustness of the whole protocol. If one of the validators has an issue, others will still receive rewards and the overall effect on stMATIC yield will be lessened.
We want to thank everyone that contributed to identifying the issue and finding a solution.
As always, we will give our best to update you with all the changes in a prompt manner. You can keep up with all the updates through our Twitter account. If you have any questions, join this Discord channel!
This article was originally published by Shard Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.
Disclaimer: This article is meant to provide general information and to help educate people. It is not financial advice of any sort, and it should not be taken in any way as such. Shard Labs would like to remind you that keeping it safe is ALWAYS vital. Never give your wallet keys or access codes […]
We have some exciting news to share! Shard Labs is now a Verified Operator of ssv.network’s Limited Mainnet Launch, which is a big step in keeping the Ethereum validators’ infrastructure safe and reliable. We are pleased to announce our official participation in @ssv_network as a Mainnet Verified Operator! What does this mean? 🧵👇 pic.twitter.com/PMMU3iizvw — […]
Disclaimer: This article is meant to provide general information and to help wider educate people. It is not financial advice of any sort, and it should not be taken in any way as such. Shard Labs would like to remind you that keeping it safe is ALWAYS vital. Never give your wallet keys or access […]