[lttng-dev] Sessions are disconnected when disk is full on remote host

[Mikael Beckius]

________________________________________
Från: jeremie.galarneau at gmail.com [jeremie.galarneau at gmail.com] för Jérémie Galarneau [jeremie.galarneau at efficios.com]
Skickat: den 25 september 2015 21:50
Till: Mikael Beckius
Kopia: lttng-dev at lists.lttng.org
Ämne: Re: [lttng-dev] Sessions are disconnected when disk is full on remote host

On Wed, Sep 23, 2015 at 5:48 AM, Mikael Beckius
<mikael.beckius at ericsson.com> wrote:
> Hello!
>
> I am trying to verify the expected behaviour in the event a machine hosting
> an lttng relay daemon runs out of disk space.
>
> Currently it appears from testing and inspecting the source code that
> whenever the lttng relay daemon hits an error caused by the lack of disk
> space all failing sessions will be disconnected. It also appears that the
> only way to recover from this situation once disk space has been restored is
> to recreate all the disconnected sessions.
>
> I have also played around with different combinations of the below lttng
> enable-channel options:
> --discard
> --overwrite
> --tracefile-count <trace file count>
> --tracefile-size <trace file size>
>
> The initial channel configuration was using the options --tracefile-count
> --tracefile-size but both were abandoned as Babeltrace faced constant
> streaming errors during heavy live tracing and now the trace files are
> growing continuously until there is no available disk space.
>
> I assume that --overwrite would not make any difference compared to
> --discard with regards to a disk full scenario. Even if you specify
> --tracefile-count and --tracefile-size to limit the amount of trace data
> from a single session you may still face the same issue with disconnected
> sessions if you for some reason run out of disk space.
>
> Testing was carried out on the latest 2.6 releases and all sessions were
> live sessions.
>
> So my questions are:
> - Is there any way to keep sessions alive even if you run out of disk space?
> - How can you clean up old traces once you have collected them without
> recreating sessions?

Hi Mikael,

LTTng currently doesn't support these recovery modes.

However, the file rotation issues you mentioned should be resolved in
lttng-tools 2.6.1. Any chance you can give it a try?
The website will be updated later today. In the meantime, here's a
direct link to the release:

https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2

Regards,
Jérémie

Hello Jérémie!

Thanks for the hint. I will give the new 2.6.1 and 2.7.0 releases of lttng-tools a try and see if they work better than 2.6.0.

Micke

>
> Micke
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>



--
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com