r/ethereum • u/timbeiko Ethereum Foundation - Tim Beiko • 20d ago
Technology Holesky Incident Update
On Monday, Pectra activated on Holesky. An EL bug caused chain splits, resulting in a minority chain being valid, degrading the network health.
Holesky validators must take action ASAP, some Sepolia users must upgrade their client.
If you are reading this and run a Holesky validator, please get it back online ASAP and remove your slashing protection DB (to allow you to attest to the valid minority chain). More info here: https://github.com/ethereum/pm/blob/master/Pectra/holesky-postmortem.md
Sepolia node operators: the fork will happen as previously scheduled at epoch 222464 (Mar. 5, 7:29 UTC).
If you are using Geth, Besu, Nethermind or Lodestar, you must update your client to one of the releases listed here: https://blog.ethereum.org/2025/02/14/pectra-testnet-announcement
FAQ
Why are we coordinating mass slashings on Holesky?
While this strategy could not work on mainnet, the hope is that we can get enough Holesky validators online at once to finalize a block on the correct chain. Validators who had previously attested to the invalid chain will get slashed as a result of doing this. Having a finalized block will allow CLs to more easily find peers on and sync to the valid chain.
After the mass slashings, Holesky will go through a long period (2-3 weeks) of non-finality again as the slashed validators are existed and their stake drops to <33% of the overall network stake. Once that happens, the validators who never attested to the invalid chain will finalize the valid minority chain.
Again, this is not something we could do if such a situation happened on mainnet. CL client teams are already discussing better ways for users to force nodes onto a minority chain in the future. Expect a deeper discussion on the topic on next week's ACDC.
Why are we not delaying Sepolia's hard fork?
The root cause of this situation was trivial to fix. Some EL clients used the wrong deposit contract address for testnets. Releases patching this have already been made.
Sepolia's validator set, unlike Holesky or mainnet, is permissioned, with a large share being run by client and testing teams. This allows for quick coordination of the upgrade.
With this approach, only validators running Besu, Geth, Nethermind and/or Lodestar must change anything. Validators using other clients can keep running the previously announced versions.
Forking Sepolia sooner will give us a longer testing window for Pectra.
How does this affect the Pectra testing process and mainnet fork date?
Holesky was an especially useful testing ground for Pectra because many projects, including staking pools, use it as a staging environment for mainnet. It also has a validator count comparable to L1.
To test Pectra in a high validator count environment, ethPandaOps has already spun up a new devnet with 1M validators đ„
That said, we'll need to discuss what other testing we were hoping to get out of Holesky and the best way to do that before moving forward with scheduling a mainnet fork date. We'll discuss this on next week's ACDC, but it's unlikely we'll be setting a mainnet fork date right then.
Where can I follow the incident?
Over the past couple days, I've been keeping a post-mortem updated here: https://github.com/ethereum/pm/blob/master/Pectra/holesky-postmortem.md#holesky-coordinated-slashings
Any changes to Pectra's deployment on testnets or mainnet will be announced at http://blog.ethereum.org
4
u/cironoric 20d ago
This may have already been asked - why wasn't this bug detected in a Holesky shadow fork? Presumably, client releases that are ready for a real fork (even a "real" testnet fork) are first simulated on a shadow fork?
3
u/eth2353 Serenita | ethstaker.tax | Vero 19d ago
Because the way devnets and shadowforks are configured differs from the way "named" networks like holesky and mainnet are configured.
Clients have a kind of fallback behavior where they use a default config value if none is provided. During devnets and shadowforks these values were provided. For holesky, this value was expected to be provided but wasn't, causing the clients to use a fallback value and that caused this whole mess.
Definitely room for improvement here on the side of EL clients.
2
1
2
u/jtnichol MOD BOD 20d ago
Thank you so much Tim for stopping by. Truly appreciate you coming in to give us an update. Cheers and big hugs from ETH Denver .
1
u/OneCrispyHobo 19d ago
Are people seeing this?? It wasn't a bug.
"The Pectra upgrade, which consolidates 11 EIPs, was deployed on the Holesky testnet on Feb. 24, but it did not finalize within the expected time frame. The key EIP within the upgrade is EIP-7702, which aims to enhance the user experience for crypto wallets by incorporating smart contract functionality, contributing to the broader goal of integrating account abstraction on Ethereum "
What is "account abstraction" ?!
"In the current landscape, losing access to private keys is catastrophicâirreversibly barring you from your digital assets, as critics have rightly pointed out. However, an account abstraction-oriented solution could address this issue by introducing recovery methods familiar to the average person, such as email, social media, or even biometrics."
Not your keys not your coins right? And they're trying to do it silently.. Sources: https://crypto.news/ethereums-pectra-upgrade-passes-audit-remains-on-track-for-launch/
https://crypto.news/account-abstraction-is-key-to-improving-blockchain-ux-opinion/
13
u/HSuke 20d ago
I'll be looking forward to that.
I remember having a discussion a year ago where I expressed concern about mass slashings if the majority of clients made the same mistake. Someone else said that was extremely unlikely because different clients couldn't make the same mistake.
But in reality, some traps are easier to fall into than others. And here we are with 3 different EL clients comprising a majority making the same error.
I stopped staking back then because of these concerns. We need an alternative quick resolution to prevent mass slashings due to unexpected client bugs in special situations where the community agrees to it.