We are specialists in building digital products that require blockchain implementation.

New York
18:23
Tokyo
07:23
Zagreb
00:23

Blog

17.3.2022. Polygon outage postmortem

Shard Labs Mar 17

The following post-mortem is written by the Shard Labs team. It highlights all the information that the community needs to know about the latest events that happened with the Polygon network and Shard Labs validator — the events that have caused an interruption in Lido accruing rewards Mar 10–14, 2022.

Before we get into a detailed analysis, here’s a short summary of the events that occurred:

  • our validator stopped working temporarily
  • bridging was disabled but is now functional again
  • no funds were lost

Detailed Analysis

On Mar 14, 2022 we made a public announcement on our Twitter regarding the issue that occurred.

On March 10, 2022 Polygon announced that the Polygon PoS chain had an implementation problem that caused the bridge to stop working. More information available here.

After this happened, rewards stopped accruing for the whole validator set because the Heimdall layer was not able to submit checkpoints to Ethereum mainnet (more details here).

This issue lasted for ~3 days until the hotfix went live. More information about the hotfix is available here.

This means that stMATIC did not gain in value from March 10–14, 2022.

A short-term fix was applied ASAP, and Lido on Polygon is accruing rewards again.

This issue happened unfortunately because the Polygon network is currently difficult to maintain due to short block time and numerous chain reorganizations. Because of this, Shard Labs’ validator, which is currently the only node operator in Lido on Polygon, got the wrong chain state data after applying the hotfix.

Conclusion and Next Steps

As an interim fix from March 14th, 2022 onwards, Shard Labs’ infrastructure is being operated by one of the most reliable Polygon node operators until Shard Labs manages to fix the state sync issue and rolls out its more robust infrastructure.

In order to anticipate similar future scenarios such as this, Shard Labs is working on having a self-hosted failover solution for timelier recovery.

In Phase 2 of Lido on Polygon (mid-April, 2022), the first batch of additional node operators will be onboarded, which will improve the reliability and robustness of the whole protocol. If one of the validators has an issue, others will still receive rewards and the overall effect on stMATIC yield will be lessened.

We want to thank everyone that contributed to identifying the issue and finding a solution.

As always, we will give our best to update you with all the changes in a prompt manner. You can keep up with all the updates through our Twitter account. If you have any questions, join this Discord channel!


This article was originally published by Shard Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Similar stories

🤔 Bear Market or Builder Market?

It is very clear that there is a global downturn in the markets, and we expect 18 months up to 3 years of downturn, potential recession on the global markets, etc. This is the first time ever that a global market recession overlaps with a crypto bear market. As we are approaching stage 3 of […]

17 June 2022

What is MATIC/stMATIC token?

If you came across some of Shard Labs’ tweets or articles, you’ve probably heard the big news: Now you can stake $MATIC in a decentralized, secure way and immediately get the representation of their share in the form of stMATIC token without maintaining staking infrastructure.  Awesome, right? ? ? ? LIDO is coming to #[email protected]_labs […]

02 June 2022

Onboarding Node Operators

Lido on Polygon is onboarding the first batch of node operators. The Lido Node Operator Subgovernance Group (LNOSG) decided to nominate 5 applicants for the proposed shortlist. More details about this shortlist are available here. April 2022 Node Operator onboarding round summery statistics It has been said that when you welcome a new family member, […]

21 April 2022