Validate what happens if aggregator crashes after `submitBlockToDA()`

[jbowen93]

What happens if submitBlockToDA was successful but the node crashes immediately after that? Can it still recover? What about the other steps in between?

Originally posted by @liamsi in #442 (comment)

Response in PR by @jbowen93

I'll admit I'm not 100% sure.

The BlockResponse, State, and Validators are never read from Store so those calls don't currently matter.

If it crashes before m.store.SetHeight(block.Header.Height) or m.executor.Commit(ctx, newState, block, responses) are called (which is immediately after submitBlockToDA()) then there may be an issue.

The sequencer will have written a block to the DA layer but will not have broadcast a header for other nodes to retrieve this block. Since state hasn't been committed to the proxyApp (via m.executor.Commit()) I think the sequencer will simply try again with the same block as the block height hasn't been incremented.

I think the primary consequence here is that the sequencer will have to pay Celestia to submit a block multiple times. This could be handled better by having the sequencer check if the block is already present in the DA layer by using fetchBlock(daHeight).

[distractedm1nd]

I made a mock test to verify what happens when an error is handled directly after a successful submitBlockToDA - meaning that publishBlock returns early, but continues to be called in AggregationLoop. I got the following logs (the error is thrown processing blockHeight 3):

I[2022-07-22|11:08:55.689] Creating and publishing block                module=BlockManager height=3
D[2022-07-22|11:08:55.689] block info                                   module=BlockManager num_tx=0
I[2022-07-22|11:08:55.689] submitting block to DA layer                 module=BlockManager height=3
D[2022-07-22|11:08:55.689] Submitting block to DA layer!                module=da_client height=3 dataLayerHeight=1
I[2022-07-22|11:08:55.689] successfully submitted optimint block to DA layer module=BlockManager optimintHeight=3 daHeight=1
E[2022-07-22|11:08:55.689] Failed to submit block to DA Layer           module=BlockManager
D[2022-07-22|11:08:55.689] Gossiping block header                       module=p2p len=282
D[2022-07-22|11:08:55.689] header received                              from=12D3KooWPfGYg7aRmQ2CzFxGFBgZhNVGCZ3ichhbbYzg7d7gh9CP bytes=282
D[2022-07-22|11:08:55.689] block header received                        module=BlockManager height=3 hash="[54 190 226 4 29 66 198 123 230 21 148 40 110 75 121 224 30 144 109 253 249 79 54 204 161 88 68 161 39 163 102 143]"
D[2022-07-22|11:08:55.689] retrieve                                     module=BlockManager daHeight=1
D[2022-07-22|11:08:55.689] trying to retrieve block from DA             module=BlockManager daHeight=1
I[2022-07-22|11:08:55.700] Using pending block                          module=BlockManager height=3
I[2022-07-22|11:08:55.700] submitting block to DA layer                 module=BlockManager height=3
D[2022-07-22|11:08:55.700] Submitting block to DA layer!                module=da_client height=3 dataLayerHeight=1
I[2022-07-22|11:08:55.700] successfully submitted optimint block to DA layer module=BlockManager optimintHeight=3 daHeight=1
D[2022-07-22|11:08:55.700] Gossiping block header                       module=p2p len=282
I[2022-07-22|11:08:55.700] indexed block                                module=txindex height=3
D[2022-07-22|11:08:55.700] indexed block txs                            module=txindex height=3 num_txs=0
D[2022-07-22|11:08:55.700] header received                              from=12D3KooWPfGYg7aRmQ2CzFxGFBgZhNVGCZ3ichhbbYzg7d7gh9CP bytes=282
D[2022-07-22|11:08:55.700] Gossiping block header                       module=p2p len=282
D[2022-07-22|11:08:55.700] block header received                        module=BlockManager height=3 hash="[54 190 226 4 29 66 198 123 230 21 148 40 110 75 121 224 30 144 109 253 249 79 54 204 161 88 68 161 39 163 102 143]"
D[2022-07-22|11:08:55.700] header received                              from=12D3KooWPfGYg7aRmQ2CzFxGFBgZhNVGCZ3ichhbbYzg7d7gh9CP bytes=282
D[2022-07-22|11:08:55.700] block header received                        module=BlockManager height=3 hash="[54 190 226 4 29 66 198 123 230 21 148 40 110 75 121 224 30 144 109 253 249 79 54 204 161 88 68 161 39 163 102 143]"

If I've interpreted this correctly, it means that the sequencer pays the DA layer again like @jbowen93 predicted? It just tries again with the same block since the height hasn't increased ofc. It will be a bit harder to figure out what happens if optimint panics here, but will report back with some more information soon.