[lttng-dev] HugePages shared memory support in LLTng

[Yiteng Guo]

Hello,

I am wondering if there is any way for lttng-ust to create its shm on
hugepages. I noticed that there was an option `--shm-path` which can be
used to change the location of shm. However, if I specified the path to a
`hugetlbfs` such as /dev/hugepages, I would get errors in lttng-sessiond
and no trace data were generated.

The error I got was
```
PERROR - 17:54:56.740674 [8163/8168]: Error appending to metadata file:
Invalid argument (in lttng_metadata_printf() at ust-metadata.c:176)
Error: Failed to generate session metadata (errno = -1)
```
I took a look at lttng code base and found that lttng used `write` to
generate a metadata file under `--shm-path`. However, it looks like
`hugetlbfs` does not support `write` operation. I did a simple patch with
`mmap` to get around this problem. Then, I got another error:
```
Error: Error creating UST channel "my-channel" on the consumer daemon
```
This time, I could not locate the problem anymore :(. Do you have any idea
of how to get hugepages shm work in lttng?

To give you more context here, I was tracing a performance sensitive
program. I didn't want to suffer from the sub-buffer switch cost so I
created a very large sub-buffer (1MB). I did a benchmark on my tracepoint
and noticed that after running a certain number of tracepoints, I got a
noticeably larger overhead (1200ns larger than other) for every ~130
tracepoints. It turned out that this large overhead was due to a page
fault. The numbers were matched up (130 * 32 bytes = 4160 bytes, which is
approximately the size of a normal page 4kB) and I also used lttng perf
page fault counters to verify it. Therefore, I am looking for a solution to
have lttng create shm on hugepages.

Thank you very much! I look forward to hearing from you.

Best,
Yiteng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190712/e484150e/attachment.html>