[lttng-dev] Trace files not getting generated

[Jérémie Galarneau]

On Wed, Oct 8, 2014 at 10:10 PM, Tommy Huynh <tom.huynh15 at yahoo.com> wrote:
> Thank you for your prompt response.
>
> Regarding your question:
>
>
> The lttng client seems to be starting a new session daemon at
> line 97 of t4-lttng-httperf.log. Did it crash in the meantime? Or
> perhaps these executions are unrelated?
>     These executions are probably unrelated. I did run the experiment
>     several times to make sure it's reproducible.
>
>
>
> It is expected if the IO capacities of the machine are exceeded by the
> event throughput. That wouldn't surprise me since you have 24 cores
>
> producing events with all kernel events enabled.
>     Are you suggesting a scalability issue with LTTng?

No. The problem is not that LTTng can't extract/capture the events.
The issue is that the machine's IO can't handle such a high,
sustained, throughput which forces LTTng to drop events.

>     What is the best way to address this?

Enabling fewer events or saving to a faster device will help a lot.
Otherwise, increasing the buffer sizes may help with coping with short
bursts of high tracing activity.

Regards,
Jérémie

>
>
> Regards,
> --
> Tom
>
>
> -------------------------------------------------------------------------
>
> On Wednesday, October 8, 2014 4:11 PM, Jérémie Galarneau <jeremie.galarneau at efficios.com> wrote:
> Good question... The consumer seems able to flush and I don't really
> see any errors...
> However, the lttng client seems to be starting a new session daemon at
> line 97 of t4-lttng-httperf.log. Did it crash in the meantime? Or
> perhaps these executions are unrelated?
>
> As for the messages in dmesg, the dropped events are not necessarily
> indicative of an issue; it is expected if the IO capacities of the
> machine are exceeded by the event throughput. That wouldn't surprise
> me since you have 24 cores producing events with all kernel events
> enabled.
>
> By the way, would you mind posting this on the lttng-dev mailing list?
> Perhaps someone can help us figure it out faster.
>
> Regards,
> Jérémie
>
>
> On Wed, Oct 8, 2014 at 4:14 PM, Tommy Huynh <tom.huynh15 at yahoo.com> wrote:
>> Hi Jérémie,
>>
>> LTTng doesn't generate the trace files and prints something like this:
>>     ring buffer relay-discard, cpu 0: records were lost. Caused by:
>>       [ 5626697 buffer full, 0 nest buffer wrap-around, 0 event too big ]
>> The behavior seems similar to this email thread:
>> http://lists.lttng.org/pipermail/lttng-dev/2013-December/022059.html,
>>
>> but my log doesn't show the problem with the consumer daemon.
>>
>> I'm running on a Freescale T4240qds development board with Linux 3.12.19
>> and LTTng 2.4.
>>
>> The run log is attached in this email along with the log file produced with:
>>
>>     lttng-sessiond -vvv --verbose-consumer > sessiond.log
>>
>> Could you let me know what's wrong?
>>
>> Regards
>> --
>> Tom Huynh
>
>
>
> --
> Jérémie Galarneau
> EfficiOS Inc.
> http://www.efficios.com



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com