AllCoreDevs - Testing #095

gm

The cool (cl) kids leave us!

Reacted to "The cool (cl) kids leave us!" with 😆

Reacted to "The cool (cl) kids l..." with 😆

gm gm😄

https://github.com/ethereum/pm/issues/2209#issuecomment-5483605197

gm gm

📋 Agenda for today https://github.com/ethereum/pm/issues/2209

Reacted to "📋 Agenda for today https://github.com/ethereum/pm/issues/2209" with 👍

-> glamsterdam-devnet-8 (Platåberget) updates - led by @qu0b Platåberget test case knocked out 12/13 Besu nodes and lagged Erigon + Ethrex: https://github.com/ethereum/pm/issues/2209#issuecomment-5568033889 https://notes.ethereum.org/@ethpandaops/glamsterdam-devnet-8

Reacted to "https://github.com/ethereum/pm/issues/2209#issuecomment-5483605197" with 👍

Payloads missing is also me

What hardware are nodes running on? Recommended, more, or less?

8 cores 16 GB RAM

digital ocean

Seems a bit below recommended hardware specs? https://eips.ethereum.org/EIPS/eip-7870 Not saying is not important to tackle ofc. But trying to understand severity

Do nodes have tracing enabled on these nodes?

Replying to "Do nodes have tracin..." yes

Replying to "Do nodes have tracin..." That will make things much harder tbh

Replying to "Do nodes have tracin..." i've now disabled traces and debug on some of the besu nodes

Replying to "Do nodes have tracin..." tracoor is requesting traces from every node

Replying to "Do nodes have tracin..." You guys really hate us :(

Reacted to "You guys really hate..." with 🥶

Do we have a shared place where we document all these devnet "attacks"? Just so we can add them to future devnets automatically, similar for some of Toni's etc

Reacted to "You guys really hate..." with 🥶

Antwoord verzenden naar "Do we have a share..." We should move these spamoor files to a specific places so we can take those "attacks" from there. I am not sure where to put those though

Otherwise they are scattered over different folders/repos

Replying to "Do we have a shared ..." Maybe a plugins repo or similar?

nimbus seems fine to me

Replying to "Do we have a shared ..." there are spammer configs included in the repo that are accessible from the frontend. sounds like a good place - but public

Reacted to "there are spammer ..." with 🙏

Antwoord verzenden naar "Do we have a share..." Ok great, I will add these attacks there :)

Heeft gereageerd op "there are spammer ..." met 🙏

Replying to "Do we have a shared ..." Spamoor is not flexible enough

Reacted to "Spamoor is not flexible enough" with 👍

@jochem-brouwer , how many accounts have you deployed for this attack?

Replying to "Do we have a shared ..." For EL it is no? What are u missing there?

Antwoord verzenden naar "@jochem-brouwer , ..." 100k

Replying to "Do we have a shared ..." Attacks are cross layer

Antwoord verzenden naar "@jochem-brouwer , ..." Effectively I only target 66.667 accounts

Antwoord verzenden naar "@jochem-brouwer , ..." This is the spamoor attack: Deploys a JUMPDEST-contract caller that cycles through indices 4000..70678 only, then hits it with full-block zero-calldata txs that keeps loop-calling these contracts and saves where it ended up at, such that the next tx can pick that up. Targets contracts were created by spamoor ID 100. Signed off by Jochem.

Replying to "Do we have a shared ..." Problem is that they are multiple gbs

Replying to "@jochem-brouwer , ho..." Ok, so clients should be using their cache for later blocks, right?

Replying to "Do we have a shared ..." Jochem’s one wasn’t. I agree it doesn’t solve everything. But covers a decent range of EL stuff with plugins. We can try supporting BALs. CL is ofc hard

there was something about Nimbus? missed it sorry

🩹 glamsterdam-devnet-9 (the planned non-finality devnet) updates + post-mortems - led by @qu0b Planned non-finality trigger not reached; ~28 h of unplanned non-finality then recovery on Sep 3 tested it anyway. https://notes.ethereum.org/@ethpandaops/glamsterdam-devnet-9

Antwoord verzenden naar "@jochem-brouwer , ..." Yes, which makes this even more scary :)

Antwoord verzenden naar "@jochem-brouwer , ..." So we loop over the same 66.667 ceil(200_000_000 / 3000) accounts and most clients struggle with this (after a while)

Antwoord verzenden naar "@jochem-brouwer , ..." So this seems to indicate this is not an IO or state problem

Replying to "Do we have a shared ..." Yeah jochem one works fine but mine are morw CL level that affect the EL. E.g. publishing a heavy payload into a different slot.

Replying to "@jochem-brouwer , ho..." memory issue?

I just mentioned I dumped stuff in benchmark channel, it is in the all-el channel

Reacted to "Yeah jochem one work..." with 👍

Interesting. We should definitely do this in the mainnet fork devnet as well

Replying to "🩹 glamsterdam-devne..." Lighthouse OOM RCA: <https://panda-uploads-production.devops-539.workers.dev/panda/uploads/724a20/devnet-9-lighthouse-oom-rca-2026-09-04.html>

Reacted to "memory issue?" with 🤔

Reacted to "I just mentioned I d..." with 👍

Does the scenario trigger a write to the account? Or just the loading and jumpdest?

Antwoord verzenden naar "@jochem-brouwer , ..." Only loading jumpdest :) CALLVALUE=0. Let me ping you on discord also

Reacted to "Only loading jumpdest :) CALLVALUE=0. Let me ping you on discord also" with 👍

Has the prysm issue been fixed?

https://github.com/OffchainLabs/prysm/pull/17382 it’s not merged in yet

D We'Re waiting for prysm for devnet-11 too...

So it’s still not in trunk

Replying to "Has the prysm issue ..." ASAIK it was just that they where not up to date on alpha.14 on their trunk

Antwoord verzenden naar "@jochem-brouwer , ..." Pinged you on discord. The similar benchmark test is: `benchmark/stateful/bloatnet/test_account_query.py::test_account_access[fork_Amsterdam-blockchain_test_stateful_engine-code_size_65536-opcode_CALL-value_sent_0-account_mode_AccountMode.EXISTING_CONTRACT_JUMPDEST-overhead_baseline_False-cache_strategy_CacheStrategy.NO_CACHE-benchmark-gas-value_200M]` There might be slight differences in the way the caller attack is setup, but the target accounts are all the same

Replying to "Has the prysm issue ..." https://github.com/OffchainLabs/prysm/pull/17382 this needs to merge in to trunk

Replying to "Has the prysm issue ..." Sorry I can’t talk

Reacted to "I just mentioned I dumped stuff in benchmark channel, it is in the all-el channel" with 👍

Replying to "Has the prysm issue ..." Yes

Antwoord verzenden naar "@jochem-brouwer , ..." Maybe worth noting: this also creates a somewhat big BAL, I recall something like 1.3 MiB pulling a large block (with lots of these txs in it) from the nodes

Reacted to "Yes" with 👍

Replying to "Has the prysm issue ..." It updates to alpha 14

Replying to "Has the prysm issue ..." But there’s still review feedback I believe

Replying to "Has the prysm issue ..." Should be merged in soon

🚀 glamsterdam-devnet-11 launch readiness - led by @qu0b EL client Checklist: - 7610 removed in tests-glamsterdam-devnet@v8.1.3 + tests@v20.0.2; - 8037 merge-time state-gas repayment (EIPs#12265) in tests-glamsterdam-devnet@v8.1.4

Replying to "feedback from anyone..." does someone from pandaops do the deposit attack there, or should i do it?

Replying to "feedback from anyone..." I can share my scripts privately

Replying to "feedback from anyone..." please don'T do on this network. L2s want to test the happy path

How many days from genesis until gloas?

Antwoord verzenden naar "🚀 glamsterdam-d..." Are you here? :)

Replying to "feedback from anyone..." ok ok 😂

Reacted to "Are you here? :)" with 👋

Antwoord verzenden naar "🚀 glamsterdam-d..." Ok cool, was getting slighty worried :)

https://github.com/OffchainLabs/prysm/pull/17440 needs to merge in develop too if you are trying builder deposits

There’s a few other ones depending on the testing ..

CL breakout: https://ethereumfoundation.zoom.us/j/89441658268?pwd=yjqf4TheEb1R7ZSuesCdOAy5tdyVk7.1

🔀 Splitting into breakouts now. EL stays here. CL with @jtraglia: <https://ethereumfoundation.zoom.us/j/89441658268?pwd=yjqf4TheEb1R7ZSuesCdOAy5tdyVk7.1>

Reacted to "CL breakout: https://ethereumfoundation.zoom.us/j/89441658268?pwd=yjqf4TheEb1R7ZSuesCdOAy5tdyVk7.1" with 👆

Reacted to "CL breakout: https..." with 👆

https://gist.github.com/jochem-brouwer/72b97452ba440da909e5da5bdf876817

Ship EIP-8253 (bump nonce of zero-nonce storage accounts) with Glamsterdam? Companion to the EIP-7610 removal / BAL edge case <https://eips.ethereum.org/EIPS/eip-8253> Jochem's TL;DR gist: <https://gist.github.com/jochem-brouwer/72b97452ba440da909e5da5bdf876817>

Reacted to "Ship EIP-8253 (bump nonce of zero-nonce storage accounts) with Glamsterdam? Companion to the EIP-7610 removal / BAL edge case <https://eips.ethereum.org/EIPS/eip-8253> Jochem's TL;DR gist: <https://gist.github.com/jochem-brouwer/72b97452ba440da909e5da5bdf876817>" with 👍

We hit that actually with PBT

I have draft PR open for spec+tests in EELS https://github.com/ethereum/execution-specs/pull/3535

May it be someone tracks such state transition? Should it be reflected in BALs?

This BAL change is in EIP-8253

-> hegota

It seems too late

Reacted to "-> hegota" with 👍

Reacted to "It seems to late" with ➕

Reacted to "It seems to late" with 👍

Reacted to "It seems to late" with 👍

Reacted to "It seems to late" with ➕

It is fine to do it in hegota. Imo this would work in Glamsterdam only if all (or most) clients prefer it.

Reacted to "It is fine to do it ..." with 👍

Reacted to "It is fine to do i..." with 👍

Reacted to "It is fine to do it ..." with 🙏

Pre-fork EIP-8253 before Glamsterdam? :) (joking)

Oh lol

Reacted to "Oh lol" with 😂

Reacted to "Oh lol" with 😂

Reacted to "Oh lol" with 😂

Reacted to "Oh lol" with 😂

Replying to "Oh lol" Creative thinking 😄

Reacted to "It seems too late" with ➕

Reacted to "Pre-fork EIP-8253 before Glamsterdam? :) (joking)" with 👍

Btw, there are no storage changes into the BAL added. It is only nonce changes. I might have messed this up by saying something differently (implying storage changes to BAL) earlier

So we bump nonce 0->1 for 28 accounts, see: https://eips.ethereum.org/EIPS/eip-8253

Replying to "Btw, there are no storage changes into the BAL added. It is only nonce changes. I might have messed this up by saying something differently (implying storage changes to BAL) earlier" I just mean if we go either way, the worst case isn't terrible for the BAL

🗂️ Close EIPs#12249 (EIP-8037 SELFDESTRUCT state-gas refill)? Declined at ACDT #94 <https://github.com/ethereum/EIPs/pull/12249> <https://forkcast.org/calls/acdt/094/>

Sorry joined late and likely missed the discussion about EIP-8253. I would be in favour of doing it in Glamsterdam if there is bandwidth on the testing side (it is mostly more effort to set up the testing in a shadow fork), but wouldn’t mind if that gets done in Hegota either. Happy with both, slight preference to just get it done now so we don’t have to talk about this again.

Reacted to "Sorry joined late an..." with 👍

Heeft gereageerd op "Sorry joined late ..." met 👍

Reacted to "Sorry joined late ..." with 🙏

Heeft gereageerd op "Sorry joined late ..." met 🙏

Antwoord verzenden naar "Sorry joined late ..." Lets discuss async. If you could take a look of adding these changes in your code, we are taking care of the test side of this (not shadow fork itself but the EELS tests+spec: https://github.com/ethereum/execution-specs/pull/3535 )

Reorg testing for ELs - brought up by @CPerezz <https://github.com/ethereum/pm/issues/2209#issuecomment-5567858868> <https://discord.com/channels/1359927674746835211/1538923536356352072/1538923541498695681>

https://discord.com/channels/1359927674746835211/1538923536356352072/1538923541498695681

Friendly reminder that the next frame tx breakout call is also tmrw. Add to the agenda if you like: https://github.com/ethereum/pm/issues/2212

We wanted to work towards this reorg window: https://eips.ethereum.org/EIPS/eip-8252

Reacted to "Friendly reminder ..." with ❤️

This might help, @CPerezz

So Neth and Lukasz are for Hive IIUC. This means no other EL has any preferences I assume. So we will start building such a suite in hive most likely then. And trying to add all PBT reorg testing there as well as MPT Awesome. Thanks!

Hive seems to make sense

Reacted to "Hive seems to make s..." with 👍

Reacted to "This might help, @CP..." with ❤️

Replying to "This might help, @CP..." Ye I know! I wanted to have perf tests for that too

🌐 eth/72 txsize is ambiguous for type-3 transactions (carried over from ACDT #94) <https://github.com/ethereum/devp2p/pull/281> <https://github.com/ethereum/pm/issues/2204#issuecomment-5479074806> EIP-8070 size_i follow-up: <https://github.com/ethereum/EIPs/pull/12275> Besu fix merged: <https://github.com/besu-eth/besu/pull/11203>

What's cool about eels is that any fixture can be run outside hive using a cli per client, if we could have genesis/payload dump from hive that might speedup reproducing the issues.

https://github.com/ethereum/devp2p/pull/281

Reacted to "Lets discuss async. If you could take a look of adding these changes in your code, we are taking care of the test side of this (not shadow fork itself but the EELS tests+spec: https://github.com/ethereum/execution-specs/pull/3535 )" with 👍

Reacted to "https://github.com..." with 🙏

Replying to "Sorry joined late an..." Sounds good, feel free to ping if you need help with anything. I’ll try to prep a branch for it tomorrow

🔍 Heads up: EIP-8037 state gas in debug traces (execution-apis#852) is on the RPC Standards Call today 15:00 UTC <https://github.com/ethereum/execution-apis/pull/852> <https://github.com/ethereum/pm/issues/2213>

I like making it a <stategas, execution > vector

Reacted to "I like making it a <..." with 👍

Reacted to "I like making it a <..." with 🙏

Antwoord verzenden naar "Sorry joined late ..." Thanks! Any test case we should include please send them to me so we directly include it in the first release of these tests. For instance the balance should not change, so things coming to mind is poking this balance, sending balance in the prior block, sending withdrawal to this account, etc etc

Antwoord verzenden naar "Sorry joined late ..." Or directly write them