Restarting Ethermint after crash panics

[jbowen93]

When Ethermint using Optimint crashed on Mamaki (see #419) an attempt to restart it with ethermintd start results in another panic

panic: SetIndex requires index to not be set

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/x/capability/keeper.Keeper.InitializeIndex({{_, _}, {_, _}, {_, _}, _, _, _}, {{0x2b214c8, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.45.1/x/capability/keeper/keeper.go:155 +0x1ea
github.com/cosmos/cosmos-sdk/x/capability.InitGenesis({{0x2b214c8, 0xc0000dc000}, {0x2b347e0, 0xc002cbec00}, {{0x0, 0x0}, {0xc002b448c0, 0xb}, 0x0, {0x1aa014ca, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.45.1/x/capability/genesis.go:12 +0xf8
github.com/cosmos/cosmos-sdk/x/capability.AppModule.InitGenesis({{{0x2b36538, 0xc0000b1fb0}}, {{0x2b31a98, 0xc0000b1fb0}, {0x2b0abf8, 0xc0019a4550}, {0x2b0ac20, 0xc0019a45f0}, 0xc001c1e870, 0xc001c1e8a0, ...}}, ...)
	github.com/cosmos/cosmos-sdk@v0.45.1/x/capability/module.go:130 +0x17c
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x2b214c8, 0xc0000dc000}, {0x2b347e0, 0xc002cbec00}, {{0x0, 0x0}, {0xc002b448c0, 0xb}, 0x0, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.45.1/types/module/module.go:320 +0x29d
github.com/tharsis/ethermint/app.(*EthermintApp).InitChainer(_, {{0x2b214c8, 0xc0000dc000}, {0x2b347e0, 0xc002cbec00}, {{0x0, 0x0}, {0xc002b448c0, 0xb}, 0x0, ...}, ...}, ...)
	github.com/tharsis/ethermint/app/app.go:620 +0x20e
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc0012c7380, {{0x1aa014ca, 0xeda26fef2, 0x0}, {0xc002b448c0, 0xb}, 0xc00058cba0, {0x3ffbef8, 0x0, 0x0}, ...})
	github.com/cosmos/cosmos-sdk@v0.45.1/baseapp/abci.go:62 +0x455
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc002b72180, {{0x1aa014ca, 0xeda26fef2, 0x0}, {0xc002b448c0, 0xb}, 0xc00058cba0, {0x3ffbef8, 0x0, 0x0}, ...})
	github.com/tendermint/tendermint@v0.34.14/abci/client/local_client.go:272 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x0?, {{0x1aa014ca, 0xeda26fef2, 0x0}, {0xc002b448c0, 0xb}, 0xc00058cba0, {0x3ffbef8, 0x0, 0x0}, ...})
	github.com/tendermint/tendermint@v0.34.14/proxy/app_conn.go:77 +0x55
github.com/celestiaorg/optimint/state.(*BlockExecutor).InitChain(0xc0012ec310, 0xc000021080)
	github.com/celestiaorg/optimint@v0.3.0/state/executor.go:59 +0x31f
github.com/celestiaorg/optimint/block.NewManager({0x2b248b8?, 0xc002b50a08}, {0x6fc23ac00, 0x6fc23ac00, 0x0, {0x0, 0x1, 0x2, 0x3, 0x4, ...}}, ...)
	github.com/celestiaorg/optimint@v0.3.0/block/manager.go:111 +0x385
github.com/celestiaorg/optimint/node.NewNode({_, _}, {{0xc000054270, 0x16}, {0xc001163e58, 0x4}, {{0xc002b43340, 0x16}, {0x0, 0x0}}, ...}, ...)
	github.com/celestiaorg/optimint@v0.3.0/node/node.go:129 +0xce5
github.com/tharsis/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2b3e778, 0xc000029620}, {0xc001163b60, 0xb}, {0x2b27490, 0xc0000b1fb0}, ...}, ...)
	github.com/tharsis/ethermint/server/start.go:330 +0xcd0
github.com/tharsis/ethermint/server.StartCmd.func2(0xc000e05180?, {0x3ffbef8?, 0x0?, 0x0?})
	github.com/tharsis/ethermint/server/start.go:122 +0x228
github.com/spf13/cobra.(*Command).execute(0xc000e05180, {0x3ffbef8, 0x0, 0x0})
	github.com/spf13/cobra@v1.4.0/command.go:856 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d8b180)
	github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.4.0/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/cobra@v1.4.0/command.go:895
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc000054270, 0x16})
	github.com/cosmos/cosmos-sdk@v0.45.1/server/cmd/execute.go:36 +0x1eb
main.main()
	github.com/tharsis/ethermint/cmd/ethermintd/main.go:20 +0x38

[tzdybal]

Reproduced also with ignite-cli-based app.

[tzdybal]

There's an issue with state deserialization from KV store. This should be tested by https://github.com/celestiaorg/optimint/blob/147a71c75b40596e4d9e829b22689bc1764d091a/store/store_test.go#L125-L145
But apparently it's too simple case (only one field is used).

Interesting question is: why json.Marshal -> blob -> json.Unmarshal fails.

[tzdybal]

The issue is NextValidators.Validators.PubKey. It is an interface (crypto.PubKey). It can be easily serialized (as string) but cannot be deserialized (as actual interface implementation is unknown).

[tzdybal]

#424 reproduces the issue. Easiest working solution would be to use encoding/gob (tested, it works).

But I'm leaning to serialize with protobuf, and at the same time adopt cosmos-sdk approach to script copying proto files from tendermint. This solution would also resolve #37.