TDM Tag "Flow" is unclear

[calcuttj]

I'm unsure how tags flow (and are intended to flow) through TDM-style graphs. This is an issue in general, but also particularly if I want to save as hdf5 in one shot after converting from TDM to Frame.

Here's what I see. I convert from TDM to IFrame, and try to save to HDF5

[11:16:16.142] D [  spng  ] <TdmToFrame:tpc0> call=0 output frame: frame: ident=100 time=-249898 tick=500 with 2560 traces.  frame tags:[ ] 1 tagged trace sets:[ "":2560 [0] ] cmm:[ ] 
[11:16:16.142] D [  hio   ] <HDF5FrameTap:test-asdf.h5> HDF5FrameTap: see frame #100 with 2560 traces with frame tags: and trace tags: "" looking for tags: "" 
[11:16:16.142] W [  hio   ] <HDF5FrameTap:test-asdf.h5> no traces for tag: ""

It looks like TdmToFrame sees (and extracts) the traces with tag "" (empty)

the HDF5FrameTap is configured to look for that same tag (it also 'sees' it in the frame as shown in the second line above). However, it doesn't actually find it (third line).

I chalk this up to some peculiarity in the way Aux::tagged_traces matches tags -- Looking at it now I can see there's a method called "untagged_traces". I can dig into that as well, but I still think the TDM tagging is an issue.

Anyways, I want to change the tags coming out of my SPNG-subgraph (into TdmToFrame) -- however this is what I can't figure out. Here's what I can figure out:

1. The TdmToFrame node in test_subgraphs.jsonnet, looks for traces tagged 'signal'. This is by default, but I've recently added a commit that makes it configurable. (Main subgraph: here)
2. Through applyroi_views that subgraph eventually configures Rebaseliner. Within that cfg is the only reference to "signal" as a tag that I see immediately.

I've stopped digging here. Does that line tag the tensor as 'signal'? It's hard to tell from within Rebaseline(r). There are outward calls to other methods regarding the cfg that I could follow, but I don't see those in other places where I might want to retag something I.e. in spng/Reduce.

[brettviren]

I've stopped digging here.

I think you almost got to the bottom of the hole. :)

In the TDM (spng/doc/datamodel.org), the meaning and treatment of the tag metadata was left intentionally a little vague as I had uncertainty. I now lean toward deprecating it, or at least minimizing its use.

As I developed the TDM nodes I began to see tensor metadata as an anti-pattern. It makes more sense in the context of a tensor set, which was the original TDM target. But most TDM nodes operate on individual tensors. And most of them are of a simple 1-to-1 filter type. And, those that are N-to-1 or 1-to-N get enough tensor identity from port order. Asking these nodes to care about metadata will only make them more complex and slow.

That said, metadata becomes needed again in TdmToFrame as that node is given a "rich" tensor set. TdmToFrame needs tensor metadata to know what tensors to turn into the various parts of the IFrame data model.

As you found, our current config relies on a "last minute" establishment of tag="signal" by the Rebaseliner node. TdmToFrame then seeks tensors with this tag to include as tagged traces of the same "signal" name.

This is hard-wired in current config and also we only configure TdmToFrame to look for a single tag, which it is capable of packing multiple tags. These config level things can be / should be made more general.

There are outward calls to other methods regarding the cfg that I could follow, but I don't see those in other places where I might want to retag something I.e. in spng/Reduce.

I have used class inheritance in the TDM nodes more than in mainline WCT in order to hoist common configuration and behavior in to a few base classes instead of expecting commonality to be kept across many final subclasses.

For example, Rebaseliner is a TensorFilter which handles the TensorFilterConfig that includes the tag and datapath_format parameters. In fact, every 1-to-1 TDM node IsA TensorFilter (assuming I didn't miss any) and so any of them can, as a side-effect of their main gig, set the tag tensor metadata.

OTOH, Reduce is a FaninBase. This also has a standard config but only with the multiplicity config parameter. You can not set the tag tensor metadata in a fan unless the leaf subclass happens to provide that behavior Reduce does not. An empty Transform node could be used to set the tag tensor metadata. Perhaps FaninBase could / should also implement tag and datapath_format config and behavior.

I know all this screams out for some documentation!