Debug the `ConnectionReverse` warning: `[ERR_STREAM_WRITE_AFTER_END]: write after end`

[joshuakarp]

Describe the bug

We've started to notice a particular warning popping up in various parts of the codebase that utilise the networking domain. Namely, the following warning:

WARN:ConnectionReverse 127.0.0.1:59070:Connection Error: Error [ERR_STREAM_WRITE_AFTER_END]: write after end

My initial thought is that this is potentially just a warning message that is being erroneously propagated up to the console output when a connection is torn down. But further investigation is required.

To Reproduce

We've noticed this occur in a few different areas:

1. When running tests/network/index.test.ts:

   Details

   [nix-shell:~/Documents/MatrixAI/js-polykey]$ npm test -- tests/network/index.test.ts 
   
   > @matrixai/polykey@1.0.0 test /home/josh/Documents/MatrixAI/js-polykey
   > jest "tests/network/index.test.ts"
   
   Determining test suites to run...
   GLOBAL SETUP
   WARN:ConnectionReverse 127.0.0.1:59070:Connection Error: Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    PASS  tests/network/index.test.ts (8.163 s)
     network index
       ✓ integration of forward and reverse proxy (4751 ms)
   
   Test Suites: 1 passed, 1 total
   Tests:       1 passed, 1 total
   Snapshots:   0 total
   Time:        8.217 s
   Ran all test suites matching /tests\/network\/index.test.ts/i.
   GLOBAL TEARDOWN
   

2. dfsd

Expected behavior

No warning message to appear at all (either by resolving some underlying issue that's causing a stream to be written to, or by stopping the warning message from propagating up).

Additional context

• original thread concerning the introduction of network/index.test.ts: https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/178#note_583511389
  • note the error outputs here too - seems that previously some ERROR outputs were also being propagated up to the console output (as per the test output) but this is no longer the case in the most recent iteration of index.test.ts
• another discussion came up when I was looking into the nodes claim process - i thought this was causing failures, but it ended up being something else https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/209#note_664762131
• similarly, with the vaults sharing, @scottmmorris came across this warning too, thinking that it was something that needed to be fixed as well Introducing vault sharing and the restrictions of pulling and cloning with vault permissions #266 (comment)

Notify maintainers

@CMCDragonkai

[CMCDragonkai]

Is this in master?

Any new changes affecting networking domain atm?

[CMCDragonkai]

@joshuakarp have a read of the connection compose logic see where writes could occur.

[joshuakarp]

Is this in master?

Any new changes affecting networking domain atm?

Yeah, in master. This has been around for several months (as early as the test was added back in May here https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/178#note_583511389) - just created this issue to keep track of it.

[CMCDragonkai]

This is the sort of bug we can use our new VSCode debugger powers for. Maybe have a try at it soon @joshuakarp.

[joshuakarp]

Good idea - will take a bit of a look into it now.

[joshuakarp]

Figured I'd log the debugging process I took in case it assists anyone else too.

Firstly, I wanted to figure out exactly what the various parts of the debugging toolbar did:

From left to right:

1. Continue: the normal "move to next breakpoint"
   
   Stops just before executing line 136 here (at the breakpoint set at line 136). Pressing continue completes the execution.
2. Step Over: takes a single "step" in your code (i.e. executes the current line you've stopped at, then stops at the next line)
   
   press "Step Over"
   
3. Step Into: same as Step Over, except if it's a function call, with enter the called function and stop on the first line of the called function
4. Step Out: if currently in a function call, this allows you to immediately jump to and stop wherever the function will return

[joshuakarp]

It's actually rather difficult to use these options to debug something that uses await. As expected, there's a fair bit of plumbing for supporting the async-await pattern, so using "Step Into" on fwdProxy.stop causes us to enter the plumbing functions:

then "Step Into":

You can mitigate this a bit by adding a breakpoint in the called async function itself though. e.g. adding a breakpoint within ForwardProxy.stop and then using "Continue" to skip over the async-await plumbing.

[joshuakarp]

For this particular issue though, the warning message arises after closing the ConnectionForward, during the await sleep call in ForwardProxy.stop:

  public async stop(): Promise<void> {
    this.logger.info('Stopping Forward Proxy Server');
    const serverClose = promisify(this.server.close).bind(this.server);
    await serverClose();
    // Ensure no new connections are created while this is iterating
    await Promise.all(
      Array.from(this.connections.ingress, ([, conn]) => conn.stop()),
    );
    // Delay socket close by about 1 second
    // this gives some time for the end/FIN packets to be sent
    await sleep(1000);
    // Even when all connections are destroyed
    // the utp socket sometimes hangs in closing
    // here we asynchronously close and unreference it
    // in order to speed up the closing
    this.utpSocket.close();
    this.utpSocket.unref();
    this.logger.info('Stopped Forward Proxy Server');
  }

It's also a little tricky to debug this with the debugger because the operations aren't sequential (because of the networking going on under the hood). So often the error will be a timeout error instead if we pause on the breakpoint for too long.

[CMCDragonkai]

Well it looks like you found it. This error can arise due to:

    // Delay socket close by about 1 second
    // this gives some time for the end/FIN packets to be sent
    await sleep(1000);
    // Even when all connections are destroyed
    // the utp socket sometimes hangs in closing
    // here we asynchronously close and unreference it
    // in order to speed up the closing
    this.utpSocket.close();

Which means the socket got closed during the sleep. And then when you call this.utpSocket.close() this tries to send a FIN packet. And you cannot write a fin packet when the socket is already closed.

At least this is what seems to be the case. One way to get around this is maybe checking if the socket is already closed, and if so don't bother calling close again.

I didn't expect the socket to be closed at that point though. Maybe something else is causing the socket to be closed. The sleep is only there to wait for all connections to stop, not necessarily for the "socket" to be closed.

If you check if the socket is closed prior, does the warning still occur?

[CMCDragonkai]

It could also be that we have closed the UTP socket but the connection failed to stop in time. That's why there's a 1000 ms sleep before closing the socket. What happens if that 1000 becomes 2000?

[CMCDragonkai]

The next question is why is the connection taking so long to close? I had investigated this prior but didn't find the reason, hence for the comment I wrote there. This may just be an intermittent error. It may not affect normal operations...

[CMCDragonkai]

The real reason may be in the C++ codebase that is not worth going into until we have swapped out UTP with something more robust like QUIC or webrtc. #234

[scottmmorris]

This is the jest test that it occured on for me:

  beforeAll(async () => {
    localKeynode = await testUtils.setupRemoteKeynode({
      logger: logger.getChild('Local Keynode'),
    });
    await localKeynode.start({
      password: 'password',
    });
    localKeynodeId = localKeynode.keyManager.getNodeId();

    remoteKeynode1 = await testUtils.setupRemoteKeynode({
      logger: logger.getChild('Remote Keynode 1'),
    });
    await remoteKeynode1.start({
      password: 'password',
    });
    remoteKeynode1Id = remoteKeynode1.keyManager.getNodeId();
    remoteKeynode2 = await testUtils.setupRemoteKeynode({
      logger: logger.getChild('Remote Keynode 2'),
    });
    await remoteKeynode2.start({
      password: 'password',
    });
    remoteKeynode2Id = remoteKeynode2.keyManager.getNodeId();
    await testUtils.addRemoteDetails(localKeynode, remoteKeynode2);
    await testUtils.addRemoteDetails(remoteKeynode1, remoteKeynode2);
    await testUtils.addRemoteDetails(remoteKeynode2, localKeynode);
    await testUtils.addRemoteDetails(remoteKeynode2, remoteKeynode1);

    await gestaltGraph.setNode({
      id: remoteKeynode1Id,
      chain: {},
    });
    await gestaltGraph.setNode({
      id: remoteKeynode2Id,
      chain: {},
    });
    await remoteKeynode1.gestaltGraph.setNode({
      id: localKeynodeId,
      chain: {},
    });
    await remoteKeynode1.gestaltGraph.setNode({
      id: remoteKeynode2Id,
      chain: {},
    });
    await remoteKeynode2.gestaltGraph.setNode({
      id: localKeynodeId,
      chain: {},
    });
    await remoteKeynode2.gestaltGraph.setNode({
      id: remoteKeynode1Id,
      chain: {},
    });
    remoteVault = await remoteKeynode1.vaultManager.createVault(vaultName);
    await remoteKeynode1.vaultManager.shareVault(
      remoteVault.vaultId,
      localKeynodeId,
    );
    await remoteKeynode1.vaultManager.shareVault(
      remoteVault.vaultId,
      remoteKeynode2Id,
    );
  });

  afterAll(async () => {
    await testUtils.cleanupRemoteKeynode(remoteKeynode2);
    await testUtils.cleanupRemoteKeynode(remoteKeynode1);
    await testUtils.cleanupRemoteKeynode(localKeynode);
  });
  test(
    'clone and pull from other cloned vaults',
    async () => {
      const clonedVaultRemote2 = await remoteKeynode2.vaultManager.cloneVault(
        remoteKeynode1Id,
        remoteVault.vaultId,
      );
      await localKeynode.acl.setNodePerm(remoteKeynode2Id, {
        gestalt: {
          notify: null,
        },
        vaults: {},
      });
      await remoteKeynode2.vaultManager.shareVault(
        clonedVaultRemote2.vaultId,
        localKeynodeId,
      );
    },
    global.defaultTimeout * 2,
  );
});

When running this in a normal test.ts file the code hung instead of giving the warning. That may have been because I set it up incorrectly though