[lttng-dev] babletrace2 graph performance considerations

[Simon Marchi]

On 2020-03-30 12:10 a.m., Rocky Dunlap via lttng-dev wrote:
> A couple of questions on performance considerations when setting up bt2 processing graphs.
> 
> 1.  Do parts of the processing graph that can execute concurrently do so?  Does the user have any control over this, e.g., by creating threads in sink components?

The graph execution in itself is single-threaded at the moment.  We could imaging
a design where different parts of the graph execute concurrently, in different
threads, but it's the the case right now.

You could make your components spawn threads to do some processing on the side,
if that helps, but these other threads should not interact directly with the
graph.

> 2.  It looks like you cannot connect one output port to multiple inputs.  Is there a way to create a tee component?

Yes, we have discussed making a tee component, it is on the roadmap but not really
planned yet.  It should be possible, it's just not as trivial as it may sound.

One easy way to achieve it is to make each iterator that is created on the tee
component create and maintain its own upstream iterator.  If you have a tee with
two outputs, this will effectively make it so you have two graphs executing in
parallel.  If you have a src.ctf.fs source upstream of the tee, then there will
be two iterators created on that source, so the CTF trace will be open and decoded
twice.  We'd like to avoid that.

The other way of doing it is to make the tee buffer messages, and discard a
message once all downstream iterators have consumed it.  This has some more
difficult technical challenges, like what to do when one downstream iterator
consumes, but the other does not (we don't want to buffer an infinite amount
of data).  It also makes seeking a bit tricky.

We could go in more details, if you are interested in starting implementing it
yourself.

Simon