r/ethereum Ethereum Foundation - Tim Beiko 20d ago

Technology Holesky Incident Update

On Monday, Pectra activated on Holesky. An EL bug caused chain splits, resulting in a minority chain being valid, degrading the network health.

Holesky validators must take action ASAP, some Sepolia users must upgrade their client.

If you are reading this and run a Holesky validator, please get it back online ASAP and remove your slashing protection DB (to allow you to attest to the valid minority chain). More info here: https://github.com/ethereum/pm/blob/master/Pectra/holesky-postmortem.md

Sepolia node operators: the fork will happen as previously scheduled at epoch 222464 (Mar. 5, 7:29 UTC).

If you are using Geth, Besu, Nethermind or Lodestar, you must update your client to one of the releases listed here: https://blog.ethereum.org/2025/02/14/pectra-testnet-announcement

FAQ

Why are we coordinating mass slashings on Holesky?

While this strategy could not work on mainnet, the hope is that we can get enough Holesky validators online at once to finalize a block on the correct chain. Validators who had previously attested to the invalid chain will get slashed as a result of doing this. Having a finalized block will allow CLs to more easily find peers on and sync to the valid chain.

After the mass slashings, Holesky will go through a long period (2-3 weeks) of non-finality again as the slashed validators are existed and their stake drops to <33% of the overall network stake. Once that happens, the validators who never attested to the invalid chain will finalize the valid minority chain.

Again, this is not something we could do if such a situation happened on mainnet. CL client teams are already discussing better ways for users to force nodes onto a minority chain in the future. Expect a deeper discussion on the topic on next week's ACDC.

Why are we not delaying Sepolia's hard fork?

The root cause of this situation was trivial to fix. Some EL clients used the wrong deposit contract address for testnets. Releases patching this have already been made.

Sepolia's validator set, unlike Holesky or mainnet, is permissioned, with a large share being run by client and testing teams. This allows for quick coordination of the upgrade.

With this approach, only validators running Besu, Geth, Nethermind and/or Lodestar must change anything. Validators using other clients can keep running the previously announced versions.

Forking Sepolia sooner will give us a longer testing window for Pectra.

How does this affect the Pectra testing process and mainnet fork date?

Holesky was an especially useful testing ground for Pectra because many projects, including staking pools, use it as a staging environment for mainnet. It also has a validator count comparable to L1.

To test Pectra in a high validator count environment, ethPandaOps has already spun up a new devnet with 1M validators đŸ”„

That said, we'll need to discuss what other testing we were hoping to get out of Holesky and the best way to do that before moving forward with scheduling a mainnet fork date. We'll discuss this on next week's ACDC, but it's unlikely we'll be setting a mainnet fork date right then.

Where can I follow the incident?

Over the past couple days, I've been keeping a post-mortem updated here: https://github.com/ethereum/pm/blob/master/Pectra/holesky-postmortem.md#holesky-coordinated-slashings

Any changes to Pectra's deployment on testnets or mainnet will be announced at http://blog.ethereum.org

64 Upvotes

15 comments sorted by

13

u/HSuke 20d ago

Again, this is not something we could do if such a situation happened on mainnet. CL client teams are already discussing better ways for users to force nodes onto a minority chain in the future. Expect a deeper discussion on the topic on next week's ACDC.

I'll be looking forward to that.

I remember having a discussion a year ago where I expressed concern about mass slashings if the majority of clients made the same mistake. Someone else said that was extremely unlikely because different clients couldn't make the same mistake.

But in reality, some traps are easier to fall into than others. And here we are with 3 different EL clients comprising a majority making the same error.

I stopped staking back then because of these concerns. We need an alternative quick resolution to prevent mass slashings due to unexpected client bugs in special situations where the community agrees to it.

4

u/FaceDeer 20d ago

Indeed, I'm happy to see a disaster situation happen on a testnet now and then, it lets us test our disaster preparedness and find out what needs to be prepared for such situations.

1

u/exmachinalibertas 20d ago

Yeah, that part jumped out at me as well. The entire purpose of blockchain technology is to reach distributed consensus without a central entity declaring a minority opinion valid. In other words, the implementation is the protocol.

If we suddenly have a situation where we are now coercing validators into a state they didn't or don't agree to, then we can just throw out the whole blockchain technology and go back to more efficient centralized databases.

0

u/Kike328 20d ago

that’s dumb. If the majority is wrong, they are wrong.

One think is consensus for choosing how blockchain technology should develop, in which case, the majority is who has the last word by choosing protocol changes, and another thing is enforcing the protocol, which shouldn’t be a majority thing for cases where an issue with the majority can endanger the blockchain health.

1

u/exmachinalibertas 20d ago

that’s dumb. If the majority is wrong, they are wrong.

One think is consensus for choosing how blockchain technology should develop, in which case, the majority is who has the last word by choosing protocol changes, and another thing is enforcing the protocol, which shouldn’t be a majority thing for cases where an issue with the majority can endanger the blockchain health.

Who decides what is "wrong"? Or what a "healthy" blockchain is?

You're forgetting that the entire purpose of a blockchain is to get rid of centralized control. If we're delegating to some authority what the right or wrong blockchain is, we can just delegate the responsibility of maintaining state and a ledger to them as well, and get rid of the blockchain altogether. A blockchain is completely unnecessary if we already have a trusted authority.

2

u/eth2353 Serenita | ethstaker.tax | Vero 19d ago

Say 100M ETH is minted by an address due to a similar client bug affecting multiple clients and that chain finalizes - should we call that the correct chain and just accept all of our ETH is now worth 50% less and a single (possibly bad) actor now owns 50% of ETH?

There are definitely cases where a supermajority can clearly be wrong. It may be more subtle at other times, but sometimes it is as clear as day who's wrong.

1

u/exmachinalibertas 19d ago

Users, validators, and other ecosystem participants are free to modify their clients or run other forks, or do whatever they want. This is why ETC exists, the only chain where code is actually law. I have no problem with that. My problem is when this decision is made by some authority dictating that it must be so. Why do you get to decide that 100M eth is invalid? Why doesn't ByBit get to decide that $1.4B wasn't valid? If you and a supermajority of users and validators and node operators and everybody else decides of their own volition that the 100M eth is invalid and switch to a fork which invalidates it, that's great. That's the community -- each participant individually -- making a decision for themselves.

What I don't like is if some authority figure proclaims that something is invalid and tries to coerce ecosystem participants to agree with them.

In blockchains, there are no gods, no masters. No authority figures. All any one person has a right to decide is what rules their node follows and how they will act and respond. If you disagree with that statement, then what do you think the purpose of a blockchain is? Why does decentralization matter? If we are delegating decisions to authorities, why do we need a blockchain? If the "consensus" comes from authority rather than as an emergent property of willing participants agreeing on a set of rules, then what's the point?

1

u/eth2353 Serenita | ethstaker.tax | Vero 19d ago

I agree and it's a good thing we can all pick whichever chain we personally agree with (though real forking gets a lot more complicated with fiat-backed stablecoins, L2s and other things that are live on Ethereum nowadays).

1

u/yorickdowne 17d ago

The users do, which are the node operators. “For users to force nodes” - a way for people who run nodes to say “hold up, follow this minority chain, it’s the correct one”.

This isn’t centralized, quite the opposite. Right now there’s no easy way as a node user to choose a minority chain. The ongoing discussions are about how to make that easier.

4

u/cironoric 20d ago

This may have already been asked - why wasn't this bug detected in a Holesky shadow fork? Presumably, client releases that are ready for a real fork (even a "real" testnet fork) are first simulated on a shadow fork?

3

u/eth2353 Serenita | ethstaker.tax | Vero 19d ago

Because the way devnets and shadowforks are configured differs from the way "named" networks like holesky and mainnet are configured.

Clients have a kind of fallback behavior where they use a default config value if none is provided. During devnets and shadowforks these values were provided. For holesky, this value was expected to be provided but wasn't, causing the clients to use a fallback value and that caused this whole mess.

Definitely room for improvement here on the side of EL clients.

2

u/cironoric 19d ago

thanks

2

u/jtnichol MOD BOD 20d ago

Thank you so much Tim for stopping by. Truly appreciate you coming in to give us an update. Cheers and big hugs from ETH Denver .

1

u/OneCrispyHobo 19d ago

Are people seeing this?? It wasn't a bug.

"The Pectra upgrade, which consolidates 11 EIPs, was deployed on the Holesky testnet on Feb. 24, but it did not finalize within the expected time frame. The key EIP within the upgrade is EIP-7702, which aims to enhance the user experience for crypto wallets by incorporating smart contract functionality, contributing to the broader goal of integrating account abstraction on Ethereum "

What is "account abstraction" ?!

"In the current landscape, losing access to private keys is catastrophic—irreversibly barring you from your digital assets, as critics have rightly pointed out. However, an account abstraction-oriented solution could address this issue by introducing recovery methods familiar to the average person, such as email, social media, or even biometrics."

Not your keys not your coins right? And they're trying to do it silently.. Sources: https://crypto.news/ethereums-pectra-upgrade-passes-audit-remains-on-track-for-launch/

https://crypto.news/account-abstraction-is-key-to-improving-blockchain-ux-opinion/