Block not submitted to DA after node restart

[tzdybal]

Following scenarios results in inconsistency in DA, that prohibits sync of full nodes:

1. DA block submission fails for prolonged time, and node panics. After restarting node last block is not submitted to DA.
2. If node is restarted after applying block, but before successful DA submission, block won't be submitted to DA after restart.

In both cases, block producer is able to continue producing blocs, but other full nodes are not able to sync, because not all blocks can be retrieved from DA.

[jbowen93]

An example of this causing syncing issues in Ethermint+Optimint

4:58PM INF current height currentHHeight=149 module=BlockManager
4:58PM INF updating state daHeight=52440 module=BlockManager
4:58PM INF minted coins from module account amount=2059780980273360763aphoton from=mint module=x/bank
4:58PM INF commit synced commit=436F6D6D697449447B5B3131382034302037332035372031323820313738203231362031373720313239203131382031313920313834203820313220343720313520313333203820313931203130322036352031323720313936203139362031303120323239203239203234342031373820313134203135342036395D3A39367D
4:58PM INF indexed block height=150 module=txindex
4:58PM INF block body retrieved from DALC hash=[236,140,46,140,143,46,2,120,158,59,32,19,19,245,156,67,120,33,24,2,107,118,4,90,182,241,108,25,252,225,64,22] height=151 module=BlockManager
4:58PM INF current height currentHHeight=150 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[236,140,46,140,143,46,2,120,158,59,32,19,19,245,156,67,120,33,24,2,107,118,4,90,182,241,108,25,252,225,64,22] height=151 module=BlockManager
4:58PM INF current height currentHHeight=150 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[236,140,46,140,143,46,2,120,158,59,32,19,19,245,156,67,120,33,24,2,107,118,4,90,182,241,108,25,252,225,64,22] height=151 module=BlockManager
4:58PM INF current height currentHHeight=150 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[236,140,46,140,143,46,2,120,158,59,32,19,19,245,156,67,120,33,24,2,107,118,4,90,182,241,108,25,252,225,64,22] height=151 module=BlockManager
4:58PM INF current height currentHHeight=150 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[105,208,175,147,229,204,6,64,24,192,168,116,157,67,123,17,215,67,53,59,173,123,175,24,165,211,56,238,7,195,155,147] height=152 module=BlockManager
4:58PM INF current height currentHHeight=150 module=BlockManager
4:58PM INF updating state daHeight=52445 module=BlockManager
4:58PM INF minted coins from module account amount=2059781349040190419aphoton from=mint module=x/bank
4:58PM INF commit synced commit=436F6D6D697449447B5B313235203138352032313620353420313438203133382031303620323920313038203233332031303020313230203132352032313720323120323136203736203132332037372035362031343920323620363120313437203231322039203136372031373720323134203531203239203135335D3A39377D
4:58PM INF indexed block height=151 module=txindex
4:58PM INF block body retrieved from DALC hash=[77,79,145,43,209,194,163,31,210,246,235,108,247,206,132,203,197,151,60,114,174,100,255,226,184,206,249,48,111,236,253,208] height=153 module=BlockManager
4:58PM INF current height currentHHeight=151 module=BlockManager
4:58PM INF updating state daHeight=52448 module=BlockManager
4:58PM INF minted coins from module account amount=2059781717807041115aphoton from=mint module=x/bank
4:58PM INF commit synced commit=436F6D6D697449447B5B31303920323535203137302038302032303320393420313830203738203139392031323820323135203234352031373220313434203132352037352036362038372032303620323533203230312031303220373420393720313836203534203232352036362035302032313720313230203230345D3A39387D
4:58PM INF block body retrieved from DALC hash=[37,94,251,45,212,63,129,200,125,152,187,19,107,183,35,96,63,30,112,214,127,157,190,242,185,123,113,219,169,131,5,228] height=155 module=BlockManager
4:58PM INF current height currentHHeight=152 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[229,154,174,148,81,7,202,4,4,124,63,127,133,61,66,84,15,188,230,113,142,100,33,149,93,158,190,113,3,180,133,86] height=156 module=BlockManager
4:58PM INF current height currentHHeight=152 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[34,140,50,152,19,230,63,151,233,132,118,241,176,119,205,57,77,139,98,97,130,134,243,232,134,80,176,111,170,226,132,65] height=157 module=BlockManager
4:58PM INF current height currentHHeight=152 module=BlockManager
4:58PM INF block body retrieved from DALC hash=[56,217,145,2,215,45,200,32,74,17,227,2,2,6,213,49,15,115,16,245,12,85,248,64,109,74,172,141,240,194,139,109] height=158 module=BlockManager
4:58PM INF current height currentHHeight=152 module=BlockManager

You can see that since block 154 is missing the node gets stalled at height 152 even though it retrieves blocks 155 to head

[jbowen93]

Possibly related, Yaz is seeing errors with his node getting stuck at block 115. This is because it is getting an apphash mismatch when attempting to apply block 116.

Block Bodies retrieved

9:17PM INF block body retrieved from DALC hash=[35,84,100,165,222,127,251,168,111,60,204,10,153,141,230,227,249,160,217,223,59,90,254,203,230,66,120,87,136,115,31,167] height=113 module=BlockManager
9:17PM INF current height currentHeight=115 module=BlockManager
9:17PM INF block body retrieved from DALC hash=[181,218,89,235,68,204,47,6,144,133,108,16,104,23,36,160,251,182,120,73,220,16,208,196,202,112,180,227,119,185,69,62] height=114 module=BlockManager
9:17PM INF current height currentHeight=115 module=BlockManager
9:17PM INF block body retrieved from DALC hash=[181,218,89,235,68,204,47,6,144,133,108,16,104,23,36,160,251,182,120,73,220,16,208,196,202,112,180,227,119,185,69,62] height=114 module=BlockManager
9:17PM INF current height currentHeight=115 module=BlockManager
9:17PM INF block body retrieved from DALC hash=[71,71,22,157,112,1,77,30,255,248,141,108,135,210,5,70,81,22,43,27,91,201,235,92,85,188,234,79,103,174,78,15] height=115 module=BlockManager
9:17PM INF current height currentHeight=115 module=BlockManager
9:17PM INF block body retrieved from DALC hash=[7,235,109,181,46,214,161,167,11,139,60,115,112,185,148,76,37,6,239,171,38,251,228,95,241,148,244,227,164,185,213,180] height=116 module=BlockManager
9:17PM INF current height currentHeight=115 module=BlockManager
9:17PM INF block body retrieved from DALC hash=[252,49,205,95,57,175,14,140,12,143,182,161,29,131,55,129,125,126,88,35,122,93,169,200,51,196,232,242,39,27,40,20] height=117 module=BlockManager
9:17PM INF current height currentHeight=115 module=BlockManager

Error

9:17PM ERR failed to ApplyBlock error="AppHash mismatch" module=BlockManager

Block

{
    "Data":{"Evidence":{"Evidence":null},"IntermediateStateRoots":{"RawRootsList":null},"Txs":null},
    "Header":{
        "AggregatorsHash":[215,251,24,163,165,52,121,226,201,10,84,221,51,83,137,192,181,181,252,113,217,54,183,124,106,92,252,175,178,20,200,237],
        "AppHash":[33,127,68,24,58,225,137,192,138,10,245,33,60,33,232,139,71,56,238,14,106,98,53,151,244,88,241,72,144,153,145,194],
        "ConsensusHash":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
        "DataHash":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
        "Height":116,
        "LastCommitHash":[182,44,38,165,112,146,188,13,199,148,110,116,112,88,175,111,158,117,155,124,67,226,158,186,100,244,45,202,31,89,111,82],
        "LastHeaderHash":[71,71,22,157,112,1,77,30,255,248,141,108,135,210,5,70,81,22,43,27,91,201,235,92,85,188,234,79,103,174,78,15],
        "LastResultsHash":[227,176,196,66,152,252,28,20,154,251,244,200,153,111,185,36,39,174,65,228,100,155,147,76,164,149,153,27,120,82,184,85],
        "NamespaceID":[0,1,2,3,4,5,6,8],
        "ProposerAddress":"GiRXH+8PZMGEwEvJok3NVYZ9nkI=",
        "Time":1655202751,
        "Version":{"App":0,"Block":11}
    },
    "LastCommit":{
        "HeaderHash":[71,71,22,157,112,1,77,30,255,248,141,108,135,210,5,70,81,22,43,27,91,201,235,92,85,188,234,79,103,174,78,15],
        "Height":115,
        "Signatures":["sBq0YBkv8UDSvPj6TKrj9zB0SoDEZxj0FoQwtzQwZNWrkdcEBTHTcWnoiR6bWTa9DCTplNxyY2q4dJ2zzVWCDQ=="]
    }
} 

State

{
    "AppHash":[100,232,208,84,155,189,51,134,72,166,217,253,171,224,107,57,9,2,103,191,82,37,106,200,179,35,130,130,129,121,58,202],
    "ChainID":"opti_69420-1",
    "ConsensusParams":{"block":{"max_bytes":22020096,"max_gas":10000000,"time_iota_ms":1000},
    "evidence":{"max_age_duration":172800000000000,"max_age_num_blocks":100000,"max_bytes":1048576},
    "validator":{"pub_key_types":["ed25519"]},"version":{}},
    "DAHeight":50549,
    "InitialHeight":1,
    "LastBlockHeight":115,
    "LastBlockID":{
        "hash":"044CD043B9FCBCCA6CA048D0369C2E0AF72167D505EDCD88FD0D73295C26CCD9",
        "parts":{"hash":"","total":0}
    },
    "LastBlockTime":"2022-06-13T14:26:34Z",
    "LastHeightConsensusParamsChanged":1,
    "LastHeightValidatorsChanged":1,
    "LastResultsHash":[227,176,196,66,152,252,28,20,154,251,244,200,153,111,185,36,39,174,65,228,100,155,147,76,164,149,153,27,120,82,184,85],
    "LastValidators":{
        "proposer":{
            "address":"1A24571FEF0F64C184C04BC9A24DCD55867D9E42",
            "proposer_priority":0,
            "pub_key":"c0JjE/d0GZ131Ar4dzTrEokfqUATXAqEZR5VRIv4Z0A=",
            "voting_power":1000000000000000},
            "validators":[
                {
                    "address":"1A24571FEF0F64C184C04BC9A24DCD55867D9E42",
                    "proposer_priority":0,"pub_key":"c0JjE/d0GZ131Ar4dzTrEokfqUATXAqEZR5VRIv4Z0A=",
                    "voting_power":1000000000000000
                }
            ]
        },
    "NextValidators":{
        "proposer":{
            "address":"1A24571FEF0F64C184C04BC9A24DCD55867D9E42",
            "proposer_priority":0,
            "pub_key":"c0JjE/d0GZ131Ar4dzTrEokfqUATXAqEZR5VRIv4Z0A=",
            "voting_power":1000000000000000
        },
        "validators":[
            {
                "address":"1A24571FEF0F64C184C04BC9A24DCD55867D9E42",
                "proposer_priority":0,
                "pub_key":"c0JjE/d0GZ131Ar4dzTrEokfqUATXAqEZR5VRIv4Z0A=",
                "voting_power":1000000000000000
            }
        ]
    },
    "Validators":{
        "proposer":{
            "address":"1A24571FEF0F64C184C04BC9A24DCD55867D9E42",
            "proposer_priority":0,
            "pub_key":"c0JjE/d0GZ131Ar4dzTrEokfqUATXAqEZR5VRIv4Z0A=",
            "voting_power":1000000000000000
        },
        "validators":[
            {
                "address":"1A24571FEF0F64C184C04BC9A24DCD55867D9E42",
                "proposer_priority":0,
                "pub_key":"c0JjE/d0GZ131Ar4dzTrEokfqUATXAqEZR5VRIv4Z0A=",
                "voting_power":1000000000000000
            }
        ]
    },
    "Version":{
        "consensus":{
            "block":11
        },
        "software":"v0.34.14"
    }
}

Mismatch between block apphash

[33,127,68,24,58,225,137,192,138,10,245,33,60,33,232,139,71,56,238,14,106,98,53,151,244,88,241,72,144,153,145,194]

and state apphash

[100,232,208,84,155,189,51,134,72,166,217,253,171,224,107,57,9,2,103,191,82,37,106,200,179,35,130,130,129,121,58,202]

[jbowen93]

Initial attempt to switch order of submission to DA layer and updating ProxyApp results in:

10:48PM ERR failed to broadcast tx error="tx size is too big: 432, max: 0" module=evm-backend

See #442

[tzdybal]

See: #441 (comment)