AllCoreDevs — Testing #072
Transcript
Ask about this call
Answered from this call's summary and transcript only.
AI-generated — verify against the recording.
Call summary
Targets
- •Wednesday - bal-devnet-3 launch pending final client readiness — 00:25:52
- •Wednesday - EPBS DevNet 0 launch with Prism/Teku/Lodestar if tests pass — 00:28:04
- •Today/tomorrow - perf-devnet-3 benchmarking suite starts with new snapshots — 00:40:14
Decisions
- •Skip expensive Blob DevNet; test Fusaka features on Holesky/Sepolia instead — 00:10:09
- •Use dedicated CPU instances for 1-2 days on bal-devnet-2 for benchmarking — 00:14:59
- •EPBS DevNet 0: stable network without sync/malicious testing initially — 00:31:26
- •Payload attestations will be ignored in EPBS DevNet 0; not breaking clients — 00:36:19
- •State Actor tool runs parallel to BloatNet for different testing purposes — 00:48:52
Highlights
- State Bloat:
- ·perf-devnet-3 deploying depth-based worst cases today; reaching 1.5-2TB state — 00:40:14
- ·Next stop 4TB; no more foreseeable worst-case scenarios after this — 00:40:27
- ·Slow block tracing PRs ready across clients; Nethermind reworking internal metrics — 00:41:01
- ·Nethermind syncing issues under investigation; Reth in crash loop on storage hashing — 00:42:42
- ·State Actor tool creates big state from block zero without Genesis downloads — 00:47:03
- Epbs Devnet 0:
- ·Prysm passing Kurtosis, finalizing robustly; sync support by today/tomorrow — 00:28:26
- ·Lighthouse breaks when payload arrives before block; needs queuing fix — 00:30:00
- ·Teku similar to Prysm status; finalized payload storage work ongoing — 00:32:21
- ·Lodestar reviewing code for main branch merge — 00:33:21
- ·Grandine needs couple more weeks before client interop readiness — 00:34:06
- ·Nimbus finalizing well; fixing peer scoring/downscoring networking issue — 00:35:00
- Fusaka Updates:
- ·Blob DevNet 0: CL teams to release hidden flags for Sepolia/Holesky testing — 00:09:13
- ·Skipping expensive DevNet; testing directly on Holesky to save resources — 00:10:09
- Gas Limit Eth70:
- ·Besu eth/70 ready; Reth eth/70 ready; both can join nft-devnet-10 — 00:37:57
- Testing Infrastructure:
- ·Test fixture structure changed: one fork per JSON to stop file growth — 00:07:09
- ·New top-level fork directories (Prague, Osaka) organize fixtures by fork — 00:07:22
- Glamsterdam Bal Devnet 2:
- ·bal-devnet-2 running with Etherax and Aragon; Aragon needs proposing fixes — 00:11:34
- ·Need configurable flags for blob prefetching/batch I/O benchmarks — 00:12:02
- ·Parallel execution benchmarking ready; can run on dedicated DevNet2 instances — 00:14:00
- Glamsterdam Bal Devnet 3:
- ·Execution APIs PRs 726 and 746 ready to merge today — 00:20:25
- ·Test fixture metadata field change: adds block access list to each block — 00:21:47
- ·Nethermind targeting Wednesday; Besu needs 8037 review; Reth missing 8037 only — 00:25:52
Action Items
- •All client teams — Feedback on test fixture structure change via Ether R&D thread — 00:08:12
- •Execution APIs maintainers — Merge execution-apis PRs 726 and 746 today — 00:19:20
- •Client teams consuming fixtures — Async review of test vector metadata field addition this week — 00:23:16
Meeting chat97
- Justin Traglia
Gm
- Stefan Starflinger
Hello
- Bhargava Shastry
Gm
- Enrico Del Fante (tbenr)
gm
- nixo
gm
- dionysuz.eth
gm
- nixo
Reacted to "Gm" with 🌞
- nixo
Reacted to "gm" with 😎
- nixo
Reacted to "gm" with ☀️
- Mario Vega
gm
- felix (eest)
Reacted to "Hello" with 🌞
- Ben Adams
ssz tests :)
- jochem-brouwer
Can also do it on empty state for pure compute ones
- jochem-brouwer
Benchmarkoor uses a snapshot so its not a live network. We can fill on Osaka and Glamsterdam
- CPerezz
Reacted to "Benchmarkoor uses a ..." with 👍
- Maria Silva
Reacted to "Benchmarkoor uses a ..." with 👍
- Parithosh Jayanthi
- CPerezz
Replying to "Benchmarkoor uses a ..." Except for execute-remote dependent tests
- Barnabas
today *
- Parithosh Jayanthi
- Parithosh Jayanthi
- jochem-brouwer
Antwoord verzenden naar "Benchmarkoor uses ..." This is a good point, I think it's there but I have to check that we can choose the fork to build these benchs on
- jochem-brouwer
Antwoord verzenden naar "https://github.com..." broken url :(
- Parithosh Jayanthi
Replying to "https://github.com/e..." fixed
- Parithosh Jayanthi
Replying to "https://github.com/e..." Trailing )
- jochem-brouwer
Heeft gereageerd op "fixed" met 🔥
- spencer-tb
Ship it?
- Stefan Starflinger
Reacted to "Ship it?" with 👍
- Marius
we haven't yet
- Barnabas
very quiet audience 😂
- Marius
Its optional, so its not a block
- Barnabas
I listed on the spec sheet as optional
- Parithosh Jayanthi
Reacted to "I listed on the spec..." with 👍🏽
- spencer-tb
Reacted to "very quiet audience ..." with 🍿
- Marius
sorry again?
- lightclient
Reacted to "very quiet audience ..." with 🍿
- Parithosh Jayanthi
Wen bal-devnet-3
- wolovim
Replying to "I listed on the spec..." https://notes.ethereum.org/@ethpandaops/bal-devnet-3
- nixo
Reacted to "very quiet audience ..." with 🍿
- Marius
We are working on a release today, afterwards we will be able to merge some stuff
- Mario Vega
Reacted to "We are working on a ..." with 🔥
- Marius
And then create teh devnet-3 branch
- Stefan Starflinger
Reacted to "We are working on a …" with 🔥
- Mario Vega
Replying to "Wen bal-devnet-3" EELS test release: https://github.com/ethereum/execution-spec-tests/releases/tag/bal@v5.2.0
- Barnabas
Potuz: “we seem to be robust” Potuz: “we cannot sync”
- Parithosh Jayanthi
Reacted to "Potuz: “we seem to b..." with 😂
- felix (eest)
Reacted to "Potuz: “we seem t..." with 😂
- Justin Traglia
Reacted to "Potuz: “we seem to b..." with 😅
- nflaig
Reacted to "Potuz: “we seem to b..." with 😂
- chrisberry
Reacted to "Potuz: “we seem to b..." with 😂
- jochem-brouwer
Heeft gereageerd op "Potuz: “we seem t..." met 😅
- felix (eest)
Replying to "Potuz: “we seem t..." this is fine
- Toni Wahrstätter
Should the payload ever arrive before the block? I guess we wouldn't expect this to happen in the happy path?
- CPerezz
Reacted to "aljpc8.jpg" with 👀
- Parithosh Jayanthi
Reacted to "aljpc8.jpg" with 👀
- Stefan Starflinger
Reacted to "aljpc8.jpg" with 👀
- Stefan Bratanov
we (Teku) have a delay of 500 ms after the block when we publish self built payload
- Justin Traglia
Reacted to "aljpc8.jpg" with 👀
- jvn
Reacted to "aljpc8.jpg" with 👀
- Mario Vega
Reacted to "Potuz: “we seem to b..." with 😂
- Potuz
@Stefan Bratanov what’s the branch? You can test against Prysm, it should handle early payloads
- Potuz
I can test against you guys if you do handle them
- Parithosh Jayanthi
Reacted to "I can test against y..." with 🔥
- Barnabas
teku is master
- Stefan Bratanov
I need to merge one PR, then we can test
- Stefan Bratanov
by the way we send and pack payload attestations in blocks, that's not issue for other clients right. they will be ignored for devnet-0? Just want to make sure it doesn't break clients
- nflaig
Replying to "I can test against y..." lodestar should be able to handle early payload
- Justin Traglia
Reacted to "lodestar should be a..." with 🔥
- Stefan Bratanov
kk thanks, I can turn it off easily if it causes issues on the gossip side when we run the devnets
- Potuz
Replying to "I can test against y..." Ah nice, what is the image I should use in Kurtosis to test against you guys?
- nflaig
Replying to "I can test against y..." ethpandops/lodestar:epbs-devnet-0
- Jen
Reth is ready w/ eth70
- Potuz
Replying to "I can test against y..." Thanks will test today after I cleanup my open PRs
- nflaig
Reacted to "Thanks will test tod..." with 👍
- Barnabas
we can include besu and reth today
- Justin Traglia
Reacted to "Thanks will test tod..." with ❤️
- Jen
Reacted to "we can include besu ..." with 🙏
- Parithosh Jayanthi
Reacted to "we can include besu ..." with 🙏
- Parithosh Jayanthi
nft-devnet-10
- Barnabas
nethermind’s host is still being investigated for now
- Barnabas
Can we hold off another day?
- CPerezz
- Barnabas
are we syncing reth in full or archive mode?
- Jen
Can you guys share the logs with us?
- CPerezz
Did Jenn not finish with it?
- Barnabas
you should have access to the servers
- CPerezz
Kk MB.
- CPerezz
Ty @pk910
- CPerezz
https://github.com/nerolation/state-actor LINK TO THE TOOL
- CPerezz
FYI
- Ameziane Hamlat
What is the status of besu nodes on bloat network, I see that only geth is proposing blocks ? I’m trying to sync again from my side as a besu follower node but how about validator nodes ?
- CPerezz
Replying to "What is the status o..." Will check now. Still, you should have access. All IPs to ssh are here: https://github.com/ethpandaops/perf-devnets/blob/master/ansible/inventories/devnet-3/inventory.ini
- Ameziane Hamlat
Reacted to "Will check now. Stil..." with 👍
- CPerezz
Bloatnet covers main net compatibility, syncing testing
- jochem-brouwer
Heeft gereageerd op "Bloatnet covers ma..." met 👍
- CPerezz
Rest can mostly be done with this tool
- CPerezz
Like Geth’s Bintrie
Key decisions
Skip expensive Blob DevNet; test Fusaka features on Holesky/Sepolia instead
To save resourcesUse dedicated CPU instances for 1-2 days on bal-devnet-2 for benchmarking
For benchmarking purposesEPBS DevNet 0: stable network without sync/malicious testing initially
For initial testingPayload attestations will be ignored in EPBS DevNet 0; not breaking clients
To avoid breaking clientsState Actor tool runs parallel to BloatNet for different testing purposes
For different testing purposes