Data race in test - pubsub related

[tzdybal]

Race is detected consistently on #530.

https://github.com/celestiaorg/optimint/actions/runs/3085647836/jobs/4993915277

==================
WARNING: DATA RACE
Write at 0x00c000d44ce8 by goroutine 110:
  github.com/stretchr/testify/mock.(*Mock).MethodCalled()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.0/mock/mock.go:516 +0xffd
  github.com/stretchr/testify/mock.(*Mock).Called()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.0/mock/mock.go:450 +0x1a4
  github.com/celestiaorg/optimint/mocks.(*Application).Commit()
      /home/runner/work/optimint/optimint/mocks/Application.go:59 +0x64
  github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync()
      /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.20/abci/client/local_client.go:264 +0x11a
  github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync()
      /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.20/proxy/app_conn.go:93 +0x42
  github.com/celestiaorg/optimint/state.(*BlockExecutor).commit()
      /home/runner/work/optimint/optimint/state/executor.go:228 +0x1b0
  github.com/celestiaorg/optimint/state.(*BlockExecutor).Commit()
      /home/runner/work/optimint/optimint/state/executor.go:163 +0xfd
  github.com/celestiaorg/optimint/block.(*Manager).trySyncNextBlock()
      /home/runner/work/optimint/optimint/block/manager.go:273 +0x804
  github.com/celestiaorg/optimint/block.(*Manager).SyncLoop()
      /home/runner/work/optimint/optimint/block/manager.go:227 +0xcda
  github.com/celestiaorg/optimint/node.(*Node).OnStart·dwrap·7()
      /home/runner/work/optimint/optimint/node/node.go:226 +0x58

Previous read at 0x00c000d44ce8 by goroutine 114:
  github.com/celestiaorg/optimint/node.TestTxGossipingAndAggregation()
      /home/runner/work/optimint/optimint/node/integration_test.go:136 +0x6a7
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1306 +0x47

Goroutine 110 (running) created at:
  github.com/celestiaorg/optimint/node.(*Node).OnStart()
      /home/runner/work/optimint/optimint/node/node.go:226 +0x850
  github.com/tendermint/tendermint/libs/service.(*BaseService).Start()
      /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.20/libs/service/service.go:144 +0x57b
  github.com/celestiaorg/optimint/node.TestTxGossipingAndAggregation()
      /home/runner/work/optimint/optimint/node/integration_test.go:96 +0x144
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1306 +0x47

Goroutine 114 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1306 +0x726
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1598 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1259 +0x22f
  testing.runTests()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1596 +0x7ca
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1504 +0x9d1
  main.main()
      _testmain.go:285 +0x32b
==================

==================
WARNING: DATA RACE
Write at 0x00c000d45be8 by goroutine 284:
  github.com/stretchr/testify/mock.(*Mock).MethodCalled()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.0/mock/mock.go:516 +0xffd
  github.com/stretchr/testify/mock.(*Mock).Called()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.0/mock/mock.go:450 +0x1a4
  github.com/celestiaorg/optimint/mocks.(*Application).Commit()
      /home/runner/work/optimint/optimint/mocks/Application.go:59 +0x64
  github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync()
      /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.20/abci/client/local_client.go:264 +0x11a
  github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync()
      /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.20/proxy/app_conn.go:93 +0x42
  github.com/celestiaorg/optimint/state.(*BlockExecutor).commit()
      /home/runner/work/optimint/optimint/state/executor.go:228 +0x1b0
  github.com/celestiaorg/optimint/state.(*BlockExecutor).Commit()
      /home/runner/work/optimint/optimint/state/executor.go:163 +0xfd
  github.com/celestiaorg/optimint/block.(*Manager).trySyncNextBlock()
      /home/runner/work/optimint/optimint/block/manager.go:273 +0x804
  github.com/celestiaorg/optimint/block.(*Manager).SyncLoop()
      /home/runner/work/optimint/optimint/block/manager.go:227 +0xcda
  github.com/celestiaorg/optimint/node.(*Node).OnStart·dwrap·7()
      /home/runner/work/optimint/optimint/node/node.go:226 +0x58

Previous read at 0x00c000d45be8 by goroutine 114:
  [failed to restore the stack]

Goroutine 284 (running) created at:
  github.com/celestiaorg/optimint/node.(*Node).OnStart()
      /home/runner/work/optimint/optimint/node/node.go:226 +0x850
  github.com/tendermint/tendermint/libs/service.(*BaseService).Start()
      /home/runner/go/pkg/mod/github.com/tendermint/tendermint@v0.34.20/libs/service/service.go:144 +0x57b
  github.com/celestiaorg/optimint/node.TestTxGossipingAndAggregation()
      /home/runner/work/optimint/optimint/node/integration_test.go:96 +0x144
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1306 +0x47

Goroutine 114 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1306 +0x726
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1598 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1259 +0x22f
  testing.runTests()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1596 +0x7ca
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.17.13/x64/src/testing/testing.go:1504 +0x9d1
  main.main()
      _testmain.go:285 +0x32b
==================

[tzdybal]

This is somehow related to: stretchr/testify#965.

Assertions in the test access Mock.Calls directly, while it's still being updated in another goroutines (nodes are still stopping).

There are two ways to fix this:

• ensure that all nodes stopped
• modify testify to enable thread-safe access to Calls

[tzdybal]

In tests, context.Background() is supplied to NewNode, and this context is never cancelled. because of this, goroutines started by Node are never stopped (they stop with context).