[lttng-dev] delay between trigger and action

[Jérémie Galarneau]

Hi Yehuda,

The trigger mechanism is asynchroneous. The applications (or the kernel) won't block until the session daemon has processed the actions of the trigger.

As such, it is likely that you will lose events between the moment the trigger is triggered and when the session is started.

We have been considering implementing synchroneous start/stop actions, but no one has sponsored that work for the moment.

In your case, a work-around you could consider is to:

  *   setup a snapshot session with the same events
  *   set the trigger's actions to record a snapshot on that session, and start your existing session

In that setup, you would most likely see the events that led to the "start" (since they are captured in the in-memory buffers of the snapshot session), and you would capture the follow-up events in the on-disk session.

I hope that helps,
Jérémie

--
Jérémie Galarneau
EfficiOS Inc.
https://www.efficios.com
________________________________
From: lttng-dev <lttng-dev-bounces at lists.lttng.org> on behalf of Yitschak, Yehuda via lttng-dev <lttng-dev at lists.lttng.org>
Sent: May 23, 2023 08:17
To: lttng-dev at lists.lttng.org <lttng-dev at lists.lttng.org>
Subject: [lttng-dev] delay between trigger and action


Hi everyone

I am experimenting with enabling trace for a specific iteration of a loop in my application.
I created 2 trace points at the start and end of the loop which log the iteration number.
the loop itself creates thousands of trace points. It runs for ~5ms.

on lttng side I created 2 triggers to start and stop my session in case the loop tracepoint registers a specific iteration number.
the triggers seem to work since there is another session entry created under the session folder but there are very few tracepoints.

is it possible that the delay between the event happening and the session trace starting causes loss of events ?
If yes, is there a way around this  ?

here is my setup :

#lttng-status
Recording session session: [inactive]

    Trace output: /home/user/lttng-traces/session-20230523-113537



=== Domain: User space ===



Buffering scheme: per-user



Tracked process attributes

  Virtual process IDs:  all

  Virtual user IDs:     all

  Virtual group IDs:    all



Channels:

-------------

- user-channel: [enabled]



    Attributes:

      Event-loss mode:  discard

      Sub-buffer size:  16777216 bytes

      Sub-buffer count: 128

      Switch timer:     inactive

      Read timer:       inactive

      Monitor timer:    1000000 us

      Blocking timeout: 0 us

      Trace file count: 1 per stream

      Trace file size:  unlimited

      Output mode:      mmap



    Statistics:

      Discarded events: 0



    Recording event rules:

      trace_events* (type: tracepoint) [enabled]

      loop* (type: tracepoint) [enabled]

#lttng list-triggers

- name: iter-start

  owner uid: 1000

  condition: event rule matches

    rule: loop:iteration (type: user tracepoint, filter: iter==2 && start == 1)

    errors: none

  actions:

    start session `session`

      errors: none

  errors: none

- name: iter-stop

  owner uid: 1000

  condition: event rule matches

    rule: loop:iteration (type: user tracepoint, filter: iter==2 && start == 0)

    errors: none

  actions:

    stop session `session`

      errors: none

  errors: none


Thanks

Yehuda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20230523/febea9b7/attachment-0001.htm>