[lttng-dev] "No space left on device" results in corrupt trace data set

[Julien Desfossez]

On 16/11/12 10:39 AM, Woegerer, Paul wrote:
> On 11/16/2012 04:30 PM, Mathieu Desnoyers wrote:
>> * David Goulet (dgoulet at efficios.com) wrote:
>>> Hi Paul,
>>>
>>> The problem here would be that metadata can grow arbitrarily over time
>>> when let say new events are enabled during tracing or a new lib is
>>> dynamically loaded in the application you are tracing. Same goes for the
>>> kernel for loaded module or CPU hotplug...
>>>
>>> To achieve a static size of metadata during the whole life of a tracing
>>> session, I guess we need somekind of flag or option set to tell the
>>> tracers to just deny new events once started... which from there, we
>>> need Mathieu's opinion :)
>>
>> live support will eventually allow to side-track this whole issue by
>> flushing metadata to disk periodically during tracing,
> 
> Does that mean that the live tracing feature that currently lives in the
> lttngtop-live branch will go into lttng-tools and babeltrace any time soon ?

The main change of lttng-tools 2.2 is the live tracing feature.
The way it is implemented in the lttngtop-live branch is more a
proof-of-concept, we will be starting the clean implementation
discussions here as soon as we have lttng-tools 2.1 released (which
should be around the end of this month).

In the mean time, to solve the problem you are currently experiencing,
you could manually add a flush on the metadata channel after the session
is started to make sure it is written on disk quickly (that's what is
done in lttngtop-live at the beginning). Of course like David said, if
you have new events/applications enabled during tracing, the metadata
needs to be flushed again.

Thanks,

Julien


> 
> Thanks
> Paul
> 
>>
>> Thanks,
>>
>> Mathieu
>>
>>>
>>> Cheers!
>>> David
>>>
>>> Woegerer, Paul:
>>>> Hi,
>>>>
>>>> embedded users sometimes have to trace to tmpfs (e.g. streaming in not
>>>> an option due to missing network connectivity).
>>>>
>>>> Ramdisks of 16M to 128M size are created and as long as the ram disk
>>>> does not get full, tracing works fine. Unfortunately if the disk gets
>>>> full things fall apart:
>>>>
>>>> PERROR: Error in file write: No space left on device [in
>>>> lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1367]
>>>>
>>>> The consumer daemon will not be able to write the metadata channel and
>>>> therefore the trace data set will be unusable (cannot be read with
>>>> babeltrace).
>>>>
>>>> -rwxrwxrwx 1 testuser users        0 Nov 16 11:22 metadata
>>>> -rwxrwxrwx 1 testuser users 15728640 Nov 16 11:22 myevents_0
>>>> -rwxrwxrwx 1 testuser users     4096 Nov 16 11:22 myevents_1
>>>>
>>>> Is it possible to somehow reserve the file size (fallocate) for metadata
>>>> so that in case the disk gets full at least the metadata gets written
>>>> and the resulting trace set is readable with babeltrace ?
>>>>
>>>> I wonder if that is possible at all. The consumer daemon doesn't know
>>>> upfront how much would need to be reserved for metadata, right ?
>>>>
>>>> Any ideas ?
>>>>
>>>> Thanks,
>>>> Paul
>>>>
>>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
> 
>