polykey async-init fixes

[tegefaulkes]

For addressing point 1. #194 (comment)

Problem regarding the usage of src/config.ts and PolykeyAgent.ts. The PK_INGRESS_PORT has to be propagated from env variables to the ReverseProxy.createReverseProxy().

However we noticed during the integration of js-async-init, some things weren't done exactly the way it's supposed to be.

In PolykeyAgent, the structure at the end of createPolykeyAgent should be:

    const pk = new Polykey({ ... });
    await pk.start({ ... });
    return pk;

The reason to have an asynchronous createX static constructor is to call the start and ensure that constructed instances are started already. If this wasn't intended, then you only needed the StartStop decorator, not the CreateDestroyStartStop decorator.

The fact that PolykeyAgent wasn't done this way, may mean that other classes that are using the CreateDestroyStartStop pattern are also doing it incorrectly.

Now because the createX static methods are meant to call the asynchronous start, this means start parameters/options must be available in the createX static methods. The ReverseProxy has these parameters:

  public async start({
    ingressHost = '0.0.0.0' as Host,
    ingressPort = 0 as Port,
    serverHost,
    serverPort,
    tlsConfig,
  }: {
    ingressHost?: Host;
    ingressPort?: Port;
    serverHost: Host;
    serverPort: Port;
    tlsConfig: TLSConfig;
  }): Promise<void> {

These need to be replicated to the createReverseProxy.

@tegefaulkes

Note the difference between constructor parameters and start parameters. In the recent work on EFS and js-db, it's better to put parameters into the constructor if it is expected that they will stick around as class properties. Only if the parameters are temporary to the asynchronous start, then you should have them in the start method. This means there are some changes required to the PK domains that haven't yet been done.

In ReverseProxy, there are some cases where start does require parameters, because for example ingressPort can be set to 0, and then it has be resolved subsequently to the actual port, and that's why it is in the start method.

Tasks

1. Review all uses of CreateDestroyStartStop decorator and ensure that their createX functions are calling start
2. Move any parameter that is meant to be stored as a class property and doesn't require asynchronous operations on them to the constructor instead of start.
3. Propagate all remaining start parameters to the createX functions so that when calling create you can parameterise the asynchronous creation.
   • order of parameters: required constructor, required start, optional constructor, optional start
4. Change to using CDSS when there is underlying state, destroy should destroy the underlying state, while stop doesn't, remember destroy is propagated from the top level
5. CreateDestroyStartStop should be using 3 exceptions: ErrorXRunning, ErrorXDestroyed, and ErrorXRunning, refer to the parameter name of the decorators to see what to use
6. Specify default values in parameters inside the function signature like { x = false }: { x: boolean; }, then for encapsulated dependencies use x = x ?? await X.createX() style. This should reduce the line noise.
7. For CDSS pattern, make sure default values are on the createX but not the constructor. In some cases we will put defaults also on the constructor if we expect the class to be also be constructed without the asynchronous create like for `
8. The info logging should be standardised for createX, destroy, start and stop. Always do this (even if the methods are noops you should still log, this is important to debug the matroska model):

   // in the createX
   logger.info(`Creating ${this.name}`);
   logger.info(`Created ${this.name}`);
   // for the other methods
   logger.info(`Starting ${this.constructor.name}`);
   logger.info(`Started ${this.constructor.name}`);
   // repeat for stop and destroy

9. Child loggers should be propagated with class name logger.getChild(Class.name).
10. Due to recursive dependency for PolykeyAgent instance in agentService, the GRPC related code may need to be SS pattern and not CDSS, check this and investigate. See polykey async-init fixes #281 (comment)
11. Setting WorkerManager should be dynamic, see src/PolykeyAgent.ts.
12. The difference between encapsulated dependencies and just external references is whether the dependency is optional. If optional, then it is an encapsulated dependency, if required, then it is an external reference. So you don't manage it's lifecycle, which means you don't call create or start.
13. The createX should be the dual of destroy. So then the domain managed state should be "created" at createX. And destroy should destroy it. However start() should always be called at the end of createX so that way the proper starting operations can be done. However don't use the recursive option for utils.mkdirExists polykey async-init fixes #281 (comment), but do use recursive option for destroy and also force...
14. Propagate the start, stop, and destroy into encapsulated dependencies but not required dependencies.
15. Make sure to test all of this in accordance to the matroska/babushka doll.
16. update GRPCClient to be inline with the async-init lib updates.
17. Update relevent getters to use @ready decoration.
18. Incorporate readiness testing to each one see tests/sessions/Session.ts.

[tegefaulkes]

Review all uses of CreateDestroyStartStop decorator and ensure that their createX functions are calling start

1. create a list of all the domains using CreateDestroyStartStop.
2. determine if any domains are missing CreateDestroyStartStop pattern.
3. make sure all domains on the list are calling start() inside the createX function.

[tegefaulkes]

Referencing the diagram from https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205#note_688426664.

list of domains using CreateDestroyStartStop.

• FwdProxy
• RevProxy
• NodeManager
• PolykeyAgent
• PolykeyClient
• GRPCServer
• Session - not converted.
• GRPCClient - not converted

[tegefaulkes]

Checking all relevent domains are calling start when being created.

• FwdProxy
• RevProxy
• NodeManager
• PolykeyAgent
• PolykeyClient
• GRPCServer
• Session
• GRPCClient
  • GRPCClientAgent
  • GRPCClientClient

[tegefaulkes]

To finish the last two items on the above list I need to convert them to using CreateDestroyStartStop first.

• Session
• GRPCClient This is an odd case. it was already converted. the interface was defined on this but the decoration with @CreateDestroyStartStop was done when it was extended for the client and the agent version of it.

[CMCDragonkai]

It's also PolykeyAgent. It wasn't propagating the start and stop and create and destroy.

[tegefaulkes]

I'm not sure we should have the networking modules start at creation. It means NodeMnanager can't be started before the services, GRPC and proxies are started. but also we're getting a dependency cycle where the clientService depends on PolykeyAgent, Polykeyagent depends on nodeManager and nodemanager depends on the service.
PolykeyAgent<-agentService<-grpcServer<-rev,fwdProxy<-NodeManager<-PolykeyAgent.

Since the networking modules are the most expensive to start up it makes sense that they and by extension PolykeyAgent should start independent of creation.

[CMCDragonkai]

@tegefaulkes did you make any WIP commits on the MR: https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/213 or was there some unpushed work?

[CMCDragonkai]

The reason clientService depends on PolykeyAgent is due to all the agent-related commands right? Lke start and stop?

There may be a way to abstract this the there's circularity here.

[CMCDragonkai]

The CreateDestroy that is applied on KeyManager is also problematic.

The CreateDestroy should not have a create method. Only a createX static method should exist.

See the tests that use CreateDestroy, there's no mention of create method.

[CMCDragonkai]

See this: https://gist.github.com/CMCDragonkai/1dbf5069d9efc11585c27cc774271584. The gist explains the different uses of CreateDestroy.

[joshuakarp]

Moving the async-init fixes todo list from the top-level description of #194. Most of these points are already in the task list in this issue, but they're a little more fleshed out here:

Details

1. Fix usage of the CreateDestroyStartStop decorator, as per Testnet Node Deployment (testnet.polykey.io) #194 (comment) (please read through this comment carefully). This will involve:

   • - Review all uses of CreateDestroyStartStop decorator and ensure that their createX functions are calling start
     For example:

     public static async createPolykeyAgent( ... ) {
       ...
       const pk = new Polykey({ ... });
       await pk.start({ ... });
       return pk;
     }

   That is, the assumption is that a class using the CreateDestroyStartStop pattern must return a started instance after the call to createX. If this wasn't intended, then you only needed the StartStop decorator, not the CreateDestroyStartStop decorator.

   • - Move any parameter that is meant to be stored as a class property and doesn't require asynchronous operations on them to the constructor instead of start.
   • - Propagate all remaining start parameters to the createX functions so that when calling create you can parameterise the asynchronous creation.
     For example, the ReverseProxy should be something like:

   static async createReverseProxy({
     ingressHost = '0.0.0.0' as Host,
     ingressPort = 0 as Port,
     serverHost,
     serverPort,
     tlsConfig,
     connConnectTime = 20000,
     connTimeoutTime = 20000,
     logger,
   }: {
     connConnectTime?: number;
     connTimeoutTime?: number;
     logger?: Logger;
   }): Promise<ReverseProxy> {
     const logger_ = logger ?? new Logger('ReverseProxy');
     ...
     const revProxy = new ReverseProxy( ... );
     await revProxy.start( ... );
     return revProxy;
   }
   
   constructor({
     connConnectTime,
     connTimeoutTime,
     logger,
   }: {
     connConnectTime: number;
     connTimeoutTime: number;
     logger: Logger;
   }) {
     logger.info('Creating Reverse Proxy');
     this.logger = logger;
     this.connConnectTime = connConnectTime;
     this.connTimeoutTime = connTimeoutTime;
     this.logger.info('Created Reverse Proxy');
   }
   
   public async start({
     ingressHost = '0.0.0.0' as Host,
     ingressPort = 0 as Port,
     serverHost,
     serverPort,
     tlsConfig,
   }: {
     ingressHost?: Host;
     ingressPort?: Port;
     serverHost: Host;
     serverPort: Port;
     tlsConfig: TLSConfig;
   })

   • - Ensure that createX is calling start on any encapsulated dependencies too. Ensure that their lifecycles are also managed in stop and destroy (i.e. call their respective stop/destroy functions here).
     • Note the distinction between an encapsulated dependency and an external reference
     • An encapsulated dependency is any optional parameter in createX/start (because if they aren't passed, we have to create them ourselves). The class is expected to manage the lifecycle of these dependencies.
     • An external reference is any required parameter. Therefore, no management of lifecycle is required (it's managed elsewhere - do not call start/stop/create/destroy on these references).
   • - Incorporate these changes into testing
   • - Create sequence diagram in accordance with the matroshka/babushka doll (i.e. clearly showing how and when everything is created and where they're injected in the lifecycle of Polykey)

[CMCDragonkai]

The current MR https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/213 shows how to use CreateDestroyStartStop for src/sessions/Session.ts.

Also in our meeting today we realised that we needed to make the networking related domains StartStop instead of CreateDestroyStartStop because of a loop occuring where the PolykeyAgent instance has to be accessed by the clientService.ts in order to stop/destroy the agent instance.

This can only be achieved by passing this from the start method of PolykeyAgent as the instance is not yet available when you call createX static methods of the dependencies of PolykeyAgent.

For classes using StartStop, they don't have a createX method, instead relying on the upstream start methods to propagate start calls.

This may apply to ForwardProxy, ReverseProxy, GRPCServer and more. This requires some prototyping to check what the case is.