[lttng-dev] A question on protocols

[Jonathan Rajotte-Julien]

Hi Patrick,

On Fri, Nov 16, 2018 at 09:49:52AM +0100, mrx wrote:
> Hi,
> 
> I have a need to collect LTTng live-traces from systems with very limited
> RAM
> and flash resources. This tracing will be running continuesly for months
> monitoring our systems. The only way for me to transport those CTF records
> somewhere else is via HTTP proxy. LTTng doesn't seem to have support for
> sending
> over proxies at all. So I think I really have a challenge a head of me, if
> this
> is at all possible.

VPN through http proxy. Better alternative would be to speak with your sysadmin
and see what you can do. Keep in mind that the protocol between relayd and
consumer is in no way "secure".

> 
> The plan is to write my own relayd from which I can then stream the
> received CTF
> records + metadata to where I can analyze them. For this to work I need
> documentation on the protocol between consumerd and relayd. I cannot find
> the
> documentation for this, where can I find it?

The source code.

> 
> Do you think this is a viable solution?

Doubt it. But we never know.

> 
> Once I receive the data where I have the possiblity to analyze it. Then I'm
> not
> sure if I'm required to write everything to the file system to be able to
> analyze the data. How would I then rotate the logs on disc so I can clean
> up?

The 2.11 release will include a new feature for session rotation.

See this presentation [1] from Jérémie Galarneau explaining how the session rotation
can be used.

[1] https://events.linuxfoundation.org/wp-content/uploads/2017/12/Fine-grained-Distributed-Application-Monitoring-Using-LTTng-J%C3%A9r%C3%A9mie-Galarneau-EfficiOS.pdf


> The best for me would be if I didn't have to go via disc at all I think.
> 
> Are there any others working on similar solution, if so, how are they
> solving this?
> How would you recommend I solve this?

I would go for session rotation via relayd (not in live mode) with a daemon
watching for ready-to-consume chunks. You can adjust for the granularity
you need at the target level.

This could be done close to the target then compress the trace chunks and send them
over http(s) to the monitoring pipeline.

> 
> The reason the current relayd doesn't work for me is two-fold:
> 1. I cannot get relayd to not write down the trace to disc. Can you control
> this at all for live tracing?

What is the real reason for not writing to disk on the relayd side?

> 2. I cannot find the documentation for the relayd <-> viewer protocol.
> Where can I find it?

Source code. The initial design proposal is under doc/ in the lttng-tools tree.

> 
> It might be that storing the traces on disc is a pre-requisite for serving a
> viewer properly. Perhaps it's just something required by relayd based on
> how it works internally. I don't know.

You can look at relayd as a specialized ftp server. The user is responsible in
managing the lifetime of the traces generated. The live protocol simply allow a
viewer to see the trace as it get received/stored.

Nothing prevent you to output on a tmpfs (ramdisk) if hitting the disk is such a
problem.

Keep in mind that trace production is normally much more quicker than trace
reading/analysis. A buffering scheme is mostly always necessary.

You can also use the --trace-file-size and --trace-file-count to limit the disk
footprint of each live session. Make sure to have enough buffer for live
reading if still using live.

Cheers
-- 
Jonathan Rajotte-Julien
EfficiOS