feat: allow to toggle Celestia in op-batcher and op-node by kien-rise · Pull Request #498 · celestiaorg/optimism

kien-rise · 2025-05-09T03:57:23Z

~~This PR introduces the --da.enabled flag, allowing Celestia support to be toggled in both op-node and op-batcher.~~

This PR disables Celestia unless all of the following flags are set: --da.rpc, --da.auth_token, --da.namespace.

func (c CLIConfig) IsEnabled() bool {
	return c.Rpc != "" && c.AuthToken != "" && c.Namespace != ""
}

In addition, it reverts several changes within op-node/rollup/derive, simplifying the process of rebasing onto the upstream repository.

Finally, it restores functionality for end-to-end (E2E) tests — the make test command now runs successfully.

Note to reviewers: You're encouraged to review the commits individually for clarity.

tuxcanfly

This reads correct but we'd want to make sure that the blobdata fallback mode continues to work as expected.

We would also need to support multi-frame transactions as implemented in #439.

kien-rise · 2025-05-21T19:52:38Z

We would also need to support multi-frame transactions as implemented in #439.

Reproducing the scenario took me more time than expected. I will recreate the scenario with OP_BATCHER_MULTI_FRAME_TXS=true and OP_BATCHER_DA_FALLBACK_MODE=calldata tomorrow.

At this point, I'm confident that this PR handles multi-frame transactions correctly, though I haven't gathered concrete evidence yet.

kien-rise · 2025-05-22T21:18:58Z

@tuxcanfly I ran tests with OP_BATCHER_MULTI_FRAME_TXS=true and OP_BATCHER_DA_FALLBACK_MODE=calldata, and everything worked as expected.

This outcome isn't surprising, since my changes are independent of the MULTI_FRAME_TXS functionality.

Scenario 1: Posting to Celestia succeeds

t=2025-05-22T20:53:49+0000 lvl=info msg="Building Celestia transaction candidate" size=117022
t=2025-05-22T20:53:57+0000 lvl=info msg="celestia: blob successfully submitted" id=cbd20b0000000000f0dd00d4005ccad7d95ecc587f955a923329ce4bbb2c6e15d93844923d4c49c4
t=2025-05-22T20:53:57+0000 lvl=info msg="Building Calldata transaction candidate" size=41
t=2025-05-22T20:53:57+0000 lvl=info msg="tx candidate" ID=4889d5b1eeecd50a91a2054fc2fba7e1:0+1+2 len(txdata.frames)=3 txdata.asBlob=false
t=2025-05-22T20:54:13+0000 lvl=info msg="Handling receipt" id=4889d5b1eeecd50a91a2054fc2fba7e1:0+1+2
t=2025-05-22T20:54:13+0000 lvl=info msg="Transaction confirmed" tx_id=4889d5b1eeecd50a91a2054fc2fba7e1:0+1+2 tx=0xdc7f92964ca8ddca502e2c47b0716dd37e5edc4dcb62837dab21d03db75cf9ac block=0xd1ae38e39648819a6581adf37b831d4c185f59221b59b2df0b913f20b1418ab0:26
t=2025-05-22T20:54:13+0000 lvl=info msg="Channel is fully submitted" id=4889d5b1eeecd50a91a2054fc2fba7e1 min_inclusion_block=26 max_inclusion_block=26

The :0+1+2 in the ID proved that there are 3 frames.

Scenario 2: Posting to Celestia fails and it falls back to calldata

I simulated the case when posting to Celestia fails.

t=2025-05-22T20:45:53+0000 lvl=error msg="celestia: blob submission failed" err="celestia: manually disabled"
t=2025-05-22T20:45:53+0000 lvl=info msg="Building Calldata transaction candidate" size=90140
t=2025-05-22T20:45:53+0000 lvl=info msg="tx candidate" ID=4624364b88b4ccd98a456997da076444:0+1+2 len(txdata.frames)=3 txdata.asBlob=false

The :0+1+2 in the ID proved that there are 3 frames.

In both cases, op-node still worked properly. The safe head was constantly increasing. It proved that op-node was able to decode the L1 transactions properly.

Here is a relevant op-node log in Scenario 1

t=2025-05-22T21:08:25+0000 lvl=info msg="Advancing bq origin" origin=0x67d7163f0fe822be780ab6c164d0089119ea6ed389f9bd8575de41f7da38a845:22 originBehind=false
t=2025-05-22T21:08:25+0000 lvl=debug msg="fetching next piece of data"
t=2025-05-22T21:08:25+0000 lvl=info msg="> CelestiaDataSource::Next"
t=2025-05-22T21:08:25+0000 lvl=info msg="> BlobDataSource::Next"
t=2025-05-22T21:08:25+0000 lvl=info msg="> BlobDataSource::open"
t=2025-05-22T21:08:25+0000 lvl=info msg="> dataAndHashesFromTxs"
t=2025-05-22T21:08:25+0000 lvl=info msg=": dataAndHashesFromTxs" len(data)=1 hashes=[]
t=2025-05-22T21:08:25+0000 lvl=info msg="< dataAndHashesFromTxs"
t=2025-05-22T21:08:25+0000 lvl=info msg="< BlobDataSource::open"
t=2025-05-22T21:08:25+0000 lvl=info msg="< BlobDataSource::Next"
t=2025-05-22T21:08:25+0000 lvl=info msg=": CelestiaDataSource::Next" s.comm=0x6fd30b0000000000d8d94a61f8b49d825e3a3d94df4a1bd2d24c1f9658044ab15117c5e60689ea11
t=2025-05-22T21:08:29+0000 lvl=info msg=": CelestiaDataSource::Next" len(blobs[0])=110208
t=2025-05-22T21:08:29+0000 lvl=info msg="< CelestiaDataSource::Next"
t=2025-05-22T21:08:29+0000 lvl=info msg="created new channel" stage=channel origin=0x67d7163f0fe822be780ab6c164d0089119ea6ed389f9bd8575de41f7da38a845:22
t=2025-05-22T21:08:29+0000 lvl=info msg="decoded span batch from channel" batch_type=SpanBatch batch_timestamp=1747947858 parent_check=0xccce77a6b6904b6a3f327a7d4504b5da66a722be origin_check=0xa0ff7af4495f5c4448d3a5f4e6dc5a06d9acc3b2 start_epoch_number=7 end_epoch_number=7 block_count=1 txs=0 compression_algo=zlib stage_origin=0x67d7163f0fe822be780ab6c164d0089119ea6ed389f9bd8575de41f7da38a845:22
t=2025-05-22T21:08:29+0000 lvl=info msg="Found next valid span batch" origin=0x67d7163f0fe822be780ab6c164d0089119ea6ed389f9bd8575de41f7da38a845:22 epoch=0xa0ff7af4495f5c4448d3a5f4e6dc5a06d9acc3b2c3ec085a18ed93aadd57b307:7 batch_type=SpanBatch batch_timestamp=1747947858 parent_check=0xccce77a6b6904b6a3f327a7d4504b5da66a722be origin_check=0xa0ff7af4495f5c4448d3a5f4e6dc5a06d9acc3b2 start_epoch_number=7 end_epoch_number=7 block_count=1 txs=0
t=2025-05-22T21:08:29+0000 lvl=debug msg="pop next batch from the cached span batch"
t=2025-05-22T21:08:29+0000 lvl=debug msg="Found next singular batch" origin=0x67d7163f0fe822be780ab6c164d0089119ea6ed389f9bd8575de41f7da38a845:22 epoch=0xa0ff7af4495f5c4448d3a5f4e6dc5a06d9acc3b2c3ec085a18ed93aadd57b307:7 batch_type=SingularBatch batch_timestamp=1747947858 parent_hash=0xccce77a6b6904b6a3f327a7d4504b5da66a722be9b1ca0d2293cf3f51f8b4d04 batch_epoch=0xa0ff7af4495f5c4448d3a5f4e6dc5a06d9acc3b2c3ec085a18ed93aadd57b307:7 txs=1588
t=2025-05-22T21:08:29+0000 lvl=info msg="generated attributes in payload queue" txs=1589 timestamp=1747947858
t=2025-05-22T21:08:31+0000 lvl=debug msg="Updating pending safe" pending_safe=0xe6961130c990fa50ea0b7070782da8933f0417c4d5816528a2fb4f744cec8399:84 local_safe=0xccce77a6b6904b6a3f327a7d4504b5da66a722be9b1ca0d2293cf3f51f8b4d04:83 unsafe=0xef787db1fcbb511a61e7644c64c29cbce3b303b7bf812dc95ee77c3f8009ca18:120 concluding=true
t=2025-05-22T21:08:31+0000 lvl=debug msg="dropping stale attributes, requesting new ones" pending=0xe6961130c990fa50ea0b7070782da8933f0417c4d5816528a2fb4f744cec8399:84 attributes_parent=0xccce77a6b6904b6a3f327a7d4504b5da66a722be9b1ca0d2293cf3f51f8b4d04:83
t=2025-05-22T21:08:31+0000 lvl=debug msg="Updating local safe" local_safe=0xe6961130c990fa50ea0b7070782da8933f0417c4d5816528a2fb4f744cec8399:84 safe=0xccce77a6b6904b6a3f327a7d4504b5da66a722be9b1ca0d2293cf3f51f8b4d04:83 unsafe=0xef787db1fcbb511a61e7644c64c29cbce3b303b7bf812dc95ee77c3f8009ca18:120
t=2025-05-22T21:08:31+0000 lvl=debug msg="Local safe head updated" local_safe=0xe6961130c990fa50ea0b7070782da8933f0417c4d5816528a2fb4f744cec8399:84
t=2025-05-22T21:08:31+0000 lvl=debug msg="Updating safe" safe=0xe6961130c990fa50ea0b7070782da8933f0417c4d5816528a2fb4f744cec8399:84 unsafe=0xef787db1fcbb511a61e7644c64c29cbce3b303b7bf812dc95ee77c3f8009ca18:120

len(blobs[0])=110208 proved that this payload correspond to a 3-frame txData (note that I set OP_BATCHER_MAX_L1_TX_SIZE_BYTES=40000 and OP_BATCHER_TARGET_NUM_FRAMES=3)

tuxcanfly · 2025-05-23T06:01:47Z

@kien-rise Thanks for confirming. Will take a deeper look and test the changes.

…not true

…ls to download Celestia blob

tuxcanfly

Looks good! Thanks for your patience and persistence! 🚀

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498)

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client da: use celestia-node client (#504) da: fix batcher submit (#511)

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client da: use celestia-node client (#504) da: fix batcher submit (#511) feat(op-celestia/indexer): add indexed l2 block status (#514)

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client da: use celestia-node client (#504) da: fix batcher submit (#511) feat(op-celestia/indexer): add indexed l2 block status (#514) da: optionally support new tx client (#517) da: fix celestia config da: fix celestia enabled da: update celestia-node v0.26.4 da: log celestia config

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client da: use celestia-node client (#504) da: fix batcher submit (#511) feat(op-celestia/indexer): add indexed l2 block status (#514) da: optionally support new tx client (#517) da: fix celestia config da: fix celestia enabled da: update celestia-node v0.26.4 da: log celestia config gh: delete dependabot

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client da: use celestia-node client (#504) da: fix batcher submit (#511) feat(op-celestia/indexer): add indexed l2 block status (#514) da: optionally support new tx client (#517) da: fix celestia config da: fix celestia enabled da: update celestia-node v0.26.4 da: log celestia config

build(deps): update go-da to v0.1.0 build(deps): update go-da to v0.2.0; add ctx da: add cli flags for da config da: add get timeout to da client build(deps): bump local-celestia-devnet to v0.12.7 fix(cli): fix da rpc check fix(cli): check return err fix(da): blob data source reuse DataFromEVMTransactions da: upgrade to go-da v0.5.0 feat(op-batcher): support disable eth fallback chore(op-batcher): refactor EthFallbackDisabledFlagName da: use env auth token if set feat: enable multi frame txs and frame size setting for calldata txs chore: add log for multiframetxs setting da: try using blobdata for eth fallback da: add gas price flag (#451) da: reuse useblobs for multiframetxs (#452) Add Github CI (#472) Cleanup Github CI workflow (#478) feat: allow to toggle Celestia in op-batcher and op-node (#498) feat: add op-celestia indexer (#503) fix: init op-node client da: use celestia-node client (#504) da: fix batcher submit (#511) feat(op-celestia/indexer): add indexed l2 block status (#514) da: optionally support new tx client (#517) da: fix celestia config da: fix celestia enabled da: update celestia-node v0.26.4 da: log celestia config gh: delete dependabot feat: `op-celestia-indexer` support for 4844 blob indexing (#530) Co-authored-by: Javed Khan <tuxcanfly@gmail.com>

kien-rise marked this pull request as ready for review May 9, 2025 11:03

kien-rise force-pushed the 1055-make-test branch from b3713fb to 7369e74 Compare May 11, 2025 19:19

tuxcanfly force-pushed the celestia-develop branch from 380fd31 to 45c4154 Compare May 12, 2025 13:40

kien-rise force-pushed the 1055-make-test branch from 7369e74 to e8fed94 Compare May 12, 2025 14:40

kien-rise changed the title ~~fix: make test~~ fix: let make test work again + feat: add --da.enabled to op-node May 12, 2025

tuxcanfly reviewed May 21, 2025

View reviewed changes

Comment thread op-node/rollup/derive/celestia_data_source.go

Comment thread op-node/rollup/derive/data_source.go

tuxcanfly suggested changes May 30, 2025

View reviewed changes

kien-rise changed the title ~~fix: let make test work again + feat: add --da.enabled to op-node~~ feat: allow to enable/disable Celestia in op-batcher and op-node Jun 3, 2025

kien-rise added 4 commits June 5, 2025 00:47

test: let "make test" work again

a3408e6

feat: add flag --da.enabled to enable/disable Celestia in op-node

dc23b56

feat: op-batcher should not use Celestia if the --da.enabled flag is …

4a08154

…not true

test: re-enable disabled tests

9654882

kien-rise force-pushed the 1055-make-test branch from 433ea48 to 9654882 Compare June 4, 2025 17:50

kien-rise changed the title ~~feat: allow to enable/disable Celestia in op-batcher and op-node~~ feat: add flag --da.enabled to toggle Celestia in op-batcher and op-node Jun 4, 2025

kien-rise requested a review from tuxcanfly June 10, 2025 11:07

tuxcanfly suggested changes Jun 26, 2025

View reviewed changes

Comment thread op-celestia/cli.go Outdated

Comment thread op-node/rollup/derive/celestia_data_source.go Outdated

feat: remove flag --da.enabled

f423d08

kien-rise changed the title ~~feat: add flag --da.enabled to toggle Celestia in op-batcher and op-node~~ feat: allow to toggle Celestia in op-batcher and op-node Jul 10, 2025

kien-rise requested a review from tuxcanfly July 10, 2025 20:24

kien-rise added 2 commits July 11, 2025 03:32

refactor: print bs.UseCelestiaDA at initialization

044c3b3

fix: return NewTemporaryError instead of NewResetError if op-node fai…

b4f0630

…ls to download Celestia blob

kien-rise force-pushed the 1055-make-test branch from a51dfc9 to b4f0630 Compare July 10, 2025 20:32

kien-rise added 2 commits July 11, 2025 11:53

refactor: use s.log instead of log

4bde8b0

feat: validate namespace

af3eb2c

tuxcanfly approved these changes Jul 28, 2025

View reviewed changes

tuxcanfly merged this pull request into celestiaorg:celestia-develop Jul 28, 2025
5 of 40 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: allow to toggle Celestia in op-batcher and op-node#498

feat: allow to toggle Celestia in op-batcher and op-node#498
tuxcanfly merged 9 commits into
celestiaorg:celestia-developfrom
kien-rise:1055-make-test

kien-rise commented May 9, 2025 •

edited

Loading

tuxcanfly left a comment

Uh oh!

Uh oh!

kien-rise commented May 21, 2025

kien-rise commented May 22, 2025

tuxcanfly commented May 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tuxcanfly left a comment

Uh oh!

Labels

2 participants

Uh oh!

Conversation

kien-rise commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuxcanfly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kien-rise commented May 21, 2025

kien-rise commented May 22, 2025

Scenario 1: Posting to Celestia succeeds

Scenario 2: Posting to Celestia fails and it falls back to calldata

tuxcanfly commented May 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tuxcanfly left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

2 participants

kien-rise commented May 9, 2025 •

edited

Loading