[lttng-dev] [lttng-tools GIT PULL] Session rotation
Julien Desfossez
jdesfossez at efficios.com
Tue Jan 16 19:43:38 UTC 2018
Hi,
Please pull from
https://github.com/jdesfossez/lttng-tools-dev.git rotate-squash
This branch implements the session rotation feature. For UST, no patches
are required, for kernel tracing it depends on this patch:
https://github.com/jdesfossez/lttng-modules-dev/commit/43336e7da5ac77ba06c24eb923cf94ccc5385ffc
The "lttng rotate" command can be called at any time while the session
is running. This flushes the current content of the ring-buffers memory
to disk or network, and create new tracefiles in a separate folder to
continue writing the active trace, a new metadata file is produced in
the new folder as well, so each invocation of the command effectively
creates new standalone traces. When the rotate command returns, it
outputs the path of the previous chunk of trace (the one that was active
up to the moment the rotation started), the user has then complete
access to this trace directory and can safely process and/or delete it.
The MI interface (lttng --mi xml rotate) also contains the path for
automated processing.
It is also possible to configure a session to rotate automatically based
on a timer or its total size (see the lttng enable-rotation command).
Rotations are supported locally and on the relay, the behavior is the
same. The only difference is the path output. Locally we output the
absolute path of the chunk, on the relay we output the relative path
under the relayd output directory.
The tests for this feature are in tests/regression/tools/rotation, the
complete documentation is currently being written.
On top of this branch is going to be added soon the feature the receive
notifications when a rotation starts and when it completes.
This is a big feature, that modifies a lot of the internals in
lttng-tools, so all testing and reviewing is appreciated !
Julien Desfossez (37):
Fix: use a free running channel key between sessiond and kernel
consumer
Fix: lttng logs nanoseconds
Fix: stream_per_chan_id_ht should allow duplicates
Make kernel tracer version global
Change trace_path to session_root_path and chunk_path
Keep the base directory of a relay session separate
Command to make a directory on the consumer or relay
Create the session and domain directories on start
Command to rename a folder
Fix: kernel snapshot handling of EAGAIN
Fix: keep the number of pipes used by poll in a variable
Keep read-only copies of fields from the channel to the stream
Dedicated function to wakeup the consumer metadata pipe
Common consumer functions to read current positions
Add ustctl_flush_buffer to the consumer API
Support to dump the kernel metadata cache for the beginning
Channel rotate pipe between sessiond and the consumers
Rotation on the relay
Rotate pending support on the relay
Consumer rotate stream
Consumer perform the rotation when extracting a packet
Consumer rotate a channel
Sessiond rotation thread
Sessiond timer thread
Rotate command
Relay rotate pending command
lttng rotate command
Rotate timer
Session consumed size notification
Size-based rotation
Save, restore and list the rotation parameters
Example client to use the session rotation API
Fix validate_trace_empty test check
Tests for the session rotation feature
Prevent manual session rotation when auto rotation is configured
Fix consumed pos for overwrite mode
Add the GMT offset in the rotated chunk path
Jérémie Galarneau (14):
ht utils are unnecessarily non-const
Bump minor notification protocol version
Typo fixes in notification thread comments
Separate session info from channel info in notification thread
Add likely/unlikely annotations on channel sample handling path
Fix: previous channel total is not updated
Remove unneeded forward declaration in condition headers
Docs: wrong enum value used in evaluation API description
Remove unneeded domain.h include
Fix: circular inclusion of lttng.h results in warning
Docs: typo in notification channel header
Fix: channel lock must be taken to check for pending notifications
Add lttng_notification_channel_has_pending_notification()
Check for pending notification on notification channel activity
configure.ac | 1 +
doc/examples/rotation/rotate-client-compress.sh | 14 ++
doc/examples/rotation/rotate-client-example.c | 345
++++++++++++++++++++++++++++++++
include/Makefile.am | 4 +
include/lttng/condition/buffer-usage.h | 11 +-
include/lttng/condition/condition.h | 3 +-
include/lttng/condition/session-consumed-size-internal.h | 65 ++++++
include/lttng/condition/session-consumed-size.h | 138
+++++++++++++
include/lttng/lttng-error.h | 7 +
include/lttng/lttng.h | 2 +
include/lttng/notification/channel-internal.h | 10 +-
include/lttng/notification/channel.h | 24 ++-
include/lttng/ref-internal.h | 62 ++++++
include/lttng/rotate-internal.h | 102 ++++++++++
include/lttng/rotate.h | 190
++++++++++++++++++
src/bin/lttng-relayd/index.c | 55 +++++
src/bin/lttng-relayd/index.h | 3 +
src/bin/lttng-relayd/main.c | 692
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
src/bin/lttng-relayd/stream.c | 1 +
src/bin/lttng-relayd/stream.h | 21 ++
src/bin/lttng-sessiond/Makefile.am | 6 +-
src/bin/lttng-sessiond/buffer-registry.c | 4 +-
src/bin/lttng-sessiond/cmd.c | 656
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
src/bin/lttng-sessiond/cmd.h | 15 +-
src/bin/lttng-sessiond/consumer.c | 232
++++++++++++++++++++-
src/bin/lttng-sessiond/consumer.h | 37 +++-
src/bin/lttng-sessiond/health-sessiond.h | 2 +
src/bin/lttng-sessiond/kernel-consumer.c | 34 +++-
src/bin/lttng-sessiond/kernel.c | 119 +++++++++--
src/bin/lttng-sessiond/kernel.h | 5 +-
src/bin/lttng-sessiond/main.c | 427
+++++++++++++++++++++++++++++++++++++--
src/bin/lttng-sessiond/notification-thread-commands.c | 14 +-
src/bin/lttng-sessiond/notification-thread-commands.h | 31 ++-
src/bin/lttng-sessiond/notification-thread-events.c | 558
+++++++++++++++++++++++++++++++++++++++------------
src/bin/lttng-sessiond/notification-thread-internal.h | 61 ++++++
src/bin/lttng-sessiond/notification-thread.c | 19 +-
src/bin/lttng-sessiond/notification-thread.h | 19 +-
src/bin/lttng-sessiond/rotate.c | 472
+++++++++++++++++++++++++++++++++++++++++++
src/bin/lttng-sessiond/rotate.h | 74 +++++++
src/bin/lttng-sessiond/rotation-thread.c | 819
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
src/bin/lttng-sessiond/rotation-thread.h | 76 +++++++
src/bin/lttng-sessiond/save.c | 26 ++-
src/bin/lttng-sessiond/session.c | 7 +-
src/bin/lttng-sessiond/session.h | 80 ++++++++
src/bin/lttng-sessiond/sessiond-timer.c | 478
++++++++++++++++++++++++++++++++++++++++++++
src/bin/lttng-sessiond/sessiond-timer.h | 67 +++++++
src/bin/lttng-sessiond/snapshot.c | 8 +-
src/bin/lttng-sessiond/trace-kernel.c | 2 +-
src/bin/lttng-sessiond/trace-kernel.h | 1 +
src/bin/lttng-sessiond/ust-app.c | 214
+++++++++++++++++++-
src/bin/lttng-sessiond/ust-app.h | 8 +
src/bin/lttng-sessiond/ust-consumer.c | 11 +-
src/bin/lttng-sessiond/ust-registry.c | 4 +-
src/bin/lttng-sessiond/utils.c | 19 ++
src/bin/lttng-sessiond/utils.h | 2 +
src/bin/lttng/Makefile.am | 3 +
src/bin/lttng/command.h | 3 +
src/bin/lttng/commands/destroy.c | 12 ++
src/bin/lttng/commands/disable_rotation.c | 261
++++++++++++++++++++++++
src/bin/lttng/commands/enable_rotation.c | 294
+++++++++++++++++++++++++++
src/bin/lttng/commands/list.c | 56 ++++++
src/bin/lttng/commands/rotate.c | 324
++++++++++++++++++++++++++++++
src/bin/lttng/commands/stop.c | 11 +
src/bin/lttng/lttng.c | 3 +
src/common/Makefile.am | 1 +
src/common/condition.c | 4 +
src/common/config/config-session-abi.h | 3 +
src/common/config/session-config.c | 128 +++++++++---
src/common/config/session.xsd | 12 +-
src/common/consumer/consumer-metadata-cache.c | 37 ++--
src/common/consumer/consumer-metadata-cache.h | 1 +
src/common/consumer/consumer-timer.c | 15 +-
src/common/consumer/consumer.c | 669
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
src/common/consumer/consumer.h | 70 +++++++
src/common/defaults.h | 6 +
src/common/error.c | 9 +-
src/common/error.h | 4 +-
src/common/evaluation.c | 9 +
src/common/hashtable/hashtable.h | 2 +-
src/common/hashtable/utils.c | 33 +--
src/common/hashtable/utils.h | 16 +-
src/common/kernel-consumer/kernel-consumer.c | 244
+++++++++++++++++++++--
src/common/kernel-consumer/kernel-consumer.h | 2 +-
src/common/kernel-ctl/kernel-ctl.c | 4 +
src/common/kernel-ctl/kernel-ctl.h | 1 +
src/common/kernel-ctl/kernel-ioctl.h | 5 +
src/common/mi-lttng-3.0.xsd | 33 +++
src/common/mi-lttng.c | 11 +
src/common/mi-lttng.h | 11 +
src/common/relayd/relayd.c | 202
+++++++++++++++++++
src/common/relayd/relayd.h | 7 +
src/common/session-consumed-size.c | 469
+++++++++++++++++++++++++++++++++++++++++++
src/common/sessiond-comm/relayd.h | 20 ++
src/common/sessiond-comm/sessiond-comm.h | 53 +++++
src/common/ust-consumer/ust-consumer.c | 233
++++++++++++++++++++--
src/common/ust-consumer/ust-consumer.h | 10 +-
src/common/utils.c | 72 ++++++-
src/common/utils.h | 1 +
src/lib/lttng-ctl/Makefile.am | 2 +-
src/lib/lttng-ctl/channel.c | 111 ++++++++++-
src/lib/lttng-ctl/lttng-ctl-health.c | 2 +
src/lib/lttng-ctl/rotate.c | 357
+++++++++++++++++++++++++++++++++
tests/regression/tools/notification/base_client.c | 1 +
tests/regression/tools/notification/notification.c | 1 +
tests/regression/tools/rotation/rotate_utils.sh | 129 ++++++++++++
tests/unit/test_ust_data.c | 12 ++
tests/utils/utils.sh | 86 +++++++-
107 files changed, 10019 insertions(+), 403 deletions(-)
create mode 100755 doc/examples/rotation/rotate-client-compress.sh
create mode 100644 doc/examples/rotation/rotate-client-example.c
create mode 100644 include/lttng/condition/session-consumed-size-internal.h
create mode 100644 include/lttng/condition/session-consumed-size.h
create mode 100644 include/lttng/ref-internal.h
create mode 100644 include/lttng/rotate-internal.h
create mode 100644 include/lttng/rotate.h
create mode 100644 src/bin/lttng-sessiond/notification-thread-internal.h
create mode 100644 src/bin/lttng-sessiond/rotate.c
create mode 100644 src/bin/lttng-sessiond/rotate.h
create mode 100644 src/bin/lttng-sessiond/rotation-thread.c
create mode 100644 src/bin/lttng-sessiond/rotation-thread.h
create mode 100644 src/bin/lttng-sessiond/sessiond-timer.c
create mode 100644 src/bin/lttng-sessiond/sessiond-timer.h
create mode 100644 src/bin/lttng/commands/disable_rotation.c
create mode 100644 src/bin/lttng/commands/enable_rotation.c
create mode 100644 src/bin/lttng/commands/rotate.c
create mode 100644 src/common/session-consumed-size.c
create mode 100644 src/lib/lttng-ctl/rotate.c
create mode 100644 tests/regression/tools/rotation/rotate_utils.sh
More information about the lttng-dev
mailing list