The following post-mortem is written by the Shard Labs team. It highlights all the information that the community needs to know about the latest events that happened with the Polygon network and Shard Labs validator — the events that have caused an interruption in Lido accruing rewards Mar 10–14, 2022.

Before we get into a detailed analysis, here’s a short summary of the events that occurred:

• our validator stopped working temporarily
• bridging was disabled but is now functional again
• no funds were lost

Detailed Analysis

On Mar 14, 2022 we made a public announcement on our Twitter regarding the issue that occurred.

🗣️ The @0xPolygon PoS had an implementation problem that caused the bridge to stop working.

A short term fix already applied and Lido for Polygon is accruing rewards again. 💧 @LidoFinance

— Shard Labs (@shard_labs) March 14, 2022

On March 10, 2022 Polygon announced that the Polygon PoS chain had an implementation problem that caused the bridge to stop working. More information available here.

After this happened, rewards stopped accruing for the whole validator set because the Heimdall layer was not able to submit checkpoints to Ethereum mainnet (more details here).

This issue lasted for ~3 days until the hotfix went live. More information about the hotfix is available here.

This means that stMATIC did not gain in value from March 10–14, 2022.

A short-term fix was applied ASAP, and Lido on Polygon is accruing rewards again.

This issue happened unfortunately because the Polygon network is currently difficult to maintain due to short block time and numerous chain reorganizations. Because of this, Shard Labs’ validator, which is currently the only node operator in Lido on Polygon, got the wrong chain state data after applying the hotfix.

Conclusion and Next Steps

As an interim fix from March 14th, 2022 onwards, Shard Labs’ infrastructure is being operated by one of the most reliable Polygon node operators until Shard Labs manages to fix the state sync issue and rolls out its more robust infrastructure.

In order to anticipate similar future scenarios such as this, Shard Labs is working on having a self-hosted failover solution for timelier recovery.

In Phase 2 of Lido on Polygon (mid-April, 2022), the first batch of additional node operators will be onboarded, which will improve the reliability and robustness of the whole protocol. If one of the validators has an issue, others will still receive rewards and the overall effect on stMATIC yield will be lessened.

We want to thank everyone that contributed to identifying the issue and finding a solution.

As always, we will give our best to update you with all the changes in a prompt manner. You can keep up with all the updates through our Twitter account. If you have any questions, join this Discord channel!

This article was originally published by Shard Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.