Rollkit is not submitting blocks to DA

[jcstein]

Overview

When testing out celestia-node v0.7.1 and celestia-app/celestia-core v0.12.0, I was unable to produce blocks from rollkit on either versions rollkit/cosmos-sdk@v0.46.7-rollkit-v0.6.0-no-fraud-proofs or rollkit/cosmos-sdk@v0.46.7-rollkit-v0.7.0-no-fraud-proofs. This is unexpected behavior and no data has been posted to Celestia although Rollkit thinks that it is producing new blocks.

tl;dr

1. Rollkit is not posting blocks on rollkit/cosmos-sdk@v0.46.7-rollkit-v0.6.0-no-fraud-proofs or rollkit/cosmos-sdk@v0.46.7-rollkit-v0.7.0-no-fraud-proofs
2. I have seen similar behavior when we were testing bitcoin+ethermint demo where daHeight was always 0
3. manual transactions to post data to Celestia local DA and testnet are functional as expected
4. here is proof that the manual transactions to post data went through, but PFBs from Rollkit did not

Details to reproduce

node v0.7.1 + core/app v0.12.0 + rollkit/cosmos-sdk@v0.46.7-rollkit-v0.7.0-no-fraud-proofs

When testing the new version of rollkit/cosmos-sdk with node 0.7.1 and core/app 0.12.0 with the hello world tutorial on a local-celestia-devnet (built off of this fork with versions listed above).

The TIA balance didn’t change after 2 transactions where I transferred state on accounts on the rollup:

before 2 transactions:

curl -s -X GET http://0.0.0.0:26659/balance | jq
{
  "denom": "utia",
  "amount": "999995000000000"
}

after 2 transactions:

curl -s -X GET http://0.0.0.0:26659/balance | jq
{
  "denom": "utia",
  "amount": "999995000000000"
}

query key 1 balance on rollup to show transfers were successful:

hellod query bank balances $KEY1
balances:
- amount: "9999999999999999999873793"
  denom: stake
pagination:
  next_key: null
  total: "0"

query key 2 balance on rollup:

hellod query bank balances $KEY2
balances:
- amount: "9999999999999999000126207"
...

node main:latest + core/app v0.12.0 + rollkit/cosmos-sdk@v0.46.7-rollkit-v0.7.0-no-fraud-proofs

Then, in an effort to repro and test on a Celestia testnet, I ran the same rollup on blockspacerace on celestia-node main:latest. Something that I see that is unusual, is I’m providing it with the correct block height by curling the RPC in the init script to start the Rollkit chain (local devnet - is on port 26650 because I was running on same machine as the rollup, see hello world tutorial for explanation) and for blockspacerace - DA_BLOCK_HEIGHT=$(curl https://rpc-blockspacerace.pops.one/block | jq -r '.result.block.header.height'), but every block submitted shows daHeight=0 in this message: 3:29PM INF successfully submitted Rollkit block to DA layer daHeight=0 module=BlockManager rollkitHeight=2

before any rollup txs, the Celestia balance is:

curl -s -X GET http://0.0.0.0:26659/balance | jq
{
  "denom": "utia",
  "amount": "5000000"
}

after a rollup tx, a block should have been produced, posted to Celestia, which would make the balance of TIA go down, in theory:

curl -s -X GET http://0.0.0.0:26659/balance | jq
{
  "denom": "utia",
  "amount": "5000000"
}

Now, I'll check the balance of those rollup transactions to again confirm they went through:

query key 1 balance on rollup:

hellod query bank balances $KEY1
balances:
- amount: "9999999999999999999957931"
...

query key 2 balance on rollup:

hellod query bank balances $KEY2
...
- amount: "9999999999999999000042069"
...

Here are the logs for the second rollup

node main:latest and core/app v0.12.0 and rollkit/cosmos-sdk@v0.46.7-rollkit-v0.6.0-no-fraud-proofs

Then, hoping to isolate the issue to the update of rollkit, I rolled back to https://github.com/rollkit/cosmos-sdk/releases/tag/v0.46.7-rollkit-v0.6.0-no-fraud-proofs and am experiencing the same issue. There is no change in TIA balance when posting blocks and no value other than 0 for daHeight. My start command in the init.sh script does have fee set to 6000 utia, and i also sent a transaction on the rollup this, but appears no blocks are being posted to Celestia.

testing PFD on same setup

confirming that i can post data to blockspacerace with the same node i tried running a rollup on

• PFB tx
• retrieve data
• checking that balance decreased as expected

[jcstein]

More testing, issue persists

node v0.7.2 + core/app v0.12.0 + rollkit/cosmos-sdk@v0.46.7-rollkit-v0.7.0-no-fraud-proofs

1. start local devnet
2. query balance of node
3. start rollup & rollup logs using this script
4. list keys
5. set variables KEY1 and KEY2
6. Create a transaction on the rollup
7. Query key 1 rollup balance
8. Query key 2 rollup balance
9. query balance of celestia node

[gupadhyaya]

fixed by #794

[jcstein]

confirming it's fixed and posting data, running off of rollkit/rollkit commit 8640e67fa57bafe66e6e2db4c7d5cb184ecb6af1 in rollkit/cosmos-sdk and using local version of rollkit/cosmos-sdk in my hello chain 🎉

running without lazy sequencing

1. start chain & chain logs
2. query balance

curl -X GET http://localhost:26659/balance
{"denom":"utia","amount":"999995000000000"}

3. produce 3 blocks
4. check balance of celestia node

curl -X GET http://localhost:26659/balance
{"denom":"utia","amount":"999994999982000"}

5. check balance of rollup account
6. send rollup tx
7. query balance of rollup account again

Run with lazy sequencing [WIP]

1. run the chain with --rollkit.lazy_aggregator true in init script
2. list keys of rollup
3. after setting keys as variables, send transaction, errors from Error: rpc error: code = NotFound desc = rpc error: code = NotFound desc = account cosmos1vwerhacft0v97df5mn7ka482vrkfewfqvln7yl not found: key not found
4. query balance of key, similar error