[lttng-dev] [PATCH lttng-modules] Fix: Building the event list fails on fragmented memory
Martin Leisener
martin at leisener.de
Mon Jun 15 14:27:07 EDT 2015
On 06/13/2015 09:32 PM, Mathieu Desnoyers wrote:
> ----- On Jun 10, 2015, at 9:17 AM, Martin Leisener martin at leisener.de wrote:
>
>> Hi Mathieu,
>>
>> sorry for sending the broken patch, I fell for some "smart" formating of my
>> email client, I cannot utilize it myself the way I send it.
>> Hence I now send the patch again as it was original intended. Sent to myself, I
>> can apply it, so I hope it now works in general as well
>> I examined as recommended diff -urN, it seems the output can be utilized by
>> patch the same way as git diff.
>> Hope it is now received the way I intended.
>
> Hi Martin,
>
> I tried you patch on a x86-64 machine (my laptop), and although taking a
> single kernel trace worked fine, it appears to OOPS when I run
>
> (in lttng-tools)
> cd tests
> ./run.sh root_regression
>
> Not sure if it's a race within lttng-modules that gets triggered by using
> virtual mem rather than linear mapping, or if it's something else...
>
> Thoughts ?
>
Hi Mathieu,
thanks for testing my patch. I tried the root regression with patched and unpatched
lttng-modules, and found the following result (x86 Ubuntu 12.04 LTS):
martin at OEKAN42PVB:~/src/lttng-tools/tests$ ./run.sh root_regression
regression/kernel/test_all_events ....................... ok
regression/kernel/test_event_basic ...................... ok
regression/kernel/test_syscall .......................... ok
regression/tools/live/test_kernel ....................... No subtests run
regression/tools/live/test_lttng_kernel ................. ok
regression/tools/streaming/test_high_throughput_limits .. ok
regression/tools/streaming/test_kernel .................. ok
regression/tools/snapshots/test_kernel .................. ok
regression/tools/snapshots/test_kernel_streaming ........ ok
regression/tools/health/test_thread_ok .................. ok
Test Summary Report
-------------------
regression/tools/live/test_kernel (Wstat: 0 Tests: 0 Failed: 0)
Parse errors: No plan found in TAP output
Files=10, Tests=2402, 32 wallclock secs ( 0.16 usr 0.02 sys + 0.23 cusr 0.42 csys = 0.83 CPU)
Result: FAIL
Everything runs the same, but for the test_thread_ok, which executes 1-17 with different speeds
sometimes 1/17 takes longer, sometimes 4/17, but I couldn't make it fail.
Same behaviour with patched and unpatched modules.
It always fails (patched and unpatched) when I execute with sudo:
regression/kernel/test_all_events ....................... Dubious, test returned 6 (wstat 1536, 0x600)
...
Since it is not appearing on my system, but in yours it would point more into the direction
of a race.
If it is a race it would be interesting if it can be also triggered via a long running test,
also with the unpatched version (and the patched version on my system)
I will not be able to look into this for the next 3 weeks, unfortunately,
but I plan to setup some long running tests, once I find time again.
Regard Martin
> Thanks,
>
> Mathieu
>
>>
>> Regards Martin
>>
>>
>>
>> On a small arm imx6 solo with 256MB RAM it often happens that memory becomes
>> fragmented rather
>> quickly, so that kmalloc will not be able to get enough consecutive pages
>> (enocuntered for
>> example if you enable all kernel events: lttng enable-event -k syscall --all).
>>
>> This patch switches the allocation to vmalloc. Tested for x86 on Ubuntu 12.04
>> Lts
>> and on imx6 solo 256MB RAM
>>
>> If this patch is not applied, you can identify low and/or fragmented memory
>> failures by
>> looking at the kernel ring buffer (please ignore DMA, it is due to some memory
>> setup
>> misconfiguration, should read Normal):
>>
>> ...
>> [ 321.993820] lttng-sessiond: page allocation failure: order:4, mode:0x1040d0
>> ...
>> [ 321.994711] lowmem_reserve[]: 0 0 0
>> [ 321.994727] DMA: 801*4kB (UEMC) 424*8kB (EMC) 355*16kB (UEMC) 344*32kB (MC)
>> 340*64kB (C) 8*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB
>> 0*16384kB 0*32768kB = 46068kB
>>
>> Signed-off-by: Martin Leisener <martin at leisener.de>
>>
>> diff -urN a/lttng-events.c lttng-modules/lttng-events.c
>> --- a/lttng-events.c 2015-06-10 14:36:51.143083012 +0200
>> +++ lttng-modules/lttng-events.c 2015-06-10 14:48:05.846662088 +0200
>> @@ -39,6 +39,7 @@
>> #include "wrapper/file.h"
>> #include <linux/jhash.h>
>> #include <linux/uaccess.h>
>> +#include <linux/vmalloc.h>
>>
>> #include "wrapper/uuid.h"
>> #include "wrapper/vmalloc.h" /* for wrapper_vmalloc_sync_all() */
>> @@ -132,10 +133,10 @@
>> GFP_KERNEL);
>> if (!metadata_cache)
>> goto err_free_session;
>> - metadata_cache->data = kzalloc(METADATA_CACHE_DEFAULT_SIZE,
>> - GFP_KERNEL);
>> + metadata_cache->data = vmalloc(METADATA_CACHE_DEFAULT_SIZE);
>> if (!metadata_cache->data)
>> goto err_free_cache;
>> + memset(metadata_cache->data, 0, METADATA_CACHE_DEFAULT_SIZE);
>> metadata_cache->cache_alloc = METADATA_CACHE_DEFAULT_SIZE;
>> kref_init(&metadata_cache->refcount);
>> session->metadata_cache = metadata_cache;
>> @@ -162,7 +163,7 @@
>> {
>> struct lttng_metadata_cache *cache =
>> container_of(kref, struct lttng_metadata_cache, refcount);
>> - kfree(cache->data);
>> + vfree(cache->data);
>> kfree(cache);
>> }
>>
>> @@ -1523,10 +1524,15 @@
>> tmp_cache_alloc_size = max_t(unsigned int,
>> session->metadata_cache->cache_alloc + len,
>> session->metadata_cache->cache_alloc << 1);
>> - tmp_cache_realloc = krealloc(session->metadata_cache->data,
>> - tmp_cache_alloc_size, GFP_KERNEL);
>> + tmp_cache_realloc = vmalloc(tmp_cache_alloc_size);
>> if (!tmp_cache_realloc)
>> goto err;
>> + else if (session->metadata_cache->data) {
>> + memset(tmp_cache_realloc, 0, tmp_cache_alloc_size);
>> + memcpy(tmp_cache_realloc, session->metadata_cache->data,
>> session->metadata_cache->cache_alloc);
>> + vfree(session->metadata_cache->data);
>> + }
>> +
>> session->metadata_cache->cache_alloc = tmp_cache_alloc_size;
>> session->metadata_cache->data = tmp_cache_realloc;
>> }
>>
>>
>>
>>
>> On 06/10/2015 09:14 AM, Mathieu Desnoyers wrote:
>>> Hi Martin,
>>>
>>> Can you reformat your patch with diff -urN so I can apply it with patch ?
>>>
>>> Thanks!
>>>
>>> Mathieu
>>>
>>> ----- On Jun 8, 2015, at 11:19 AM, Martin Leisener martin at leisener.de wrote:
>>>
>>>> Hi Mathieu,
>>>>
>>>> thanks for merging my arm patch! I found one more issue during my lttng-modules
>>>> usages,
>>>> which can happen in low/fragmented memory situations. Please have a look, if you
>>>> find this
>>>> patch useful/mergable. Again it is tested on x86 Ubuntu 12.04 LTS as well as my
>>>> imx6 arm board.
>>>>
>>>> Regards Martin
>>>>
>>>>
>>>> On a small arm imx6 solo with 256MB RAM it often happens that memory becomes
>>>> fragmented rather
>>>> quickly, so that kmalloc will not be able to get enough consecutive pages
>>>> (enocuntered for
>>>> example if you enable all kernel events: lttng enable-event -k syscall --all).
>>>>
>>>> This patch switches the allocation to vmalloc. Tested for x86 on Ubuntu 12.04
>>>> Lts
>>>> and on imx6 solo 256MB RAM
>>>>
>>>> If this patch is not applied, you can identify low and/or fragmented memory
>>>> failures by
>>>> looking at the kernel ring buffer (please ignore DMA, it is due to some memory
>>>> setup
>>>> misconfiguration, should read Normal):
>>>>
>>>> ...
>>>> [ 321.993820] lttng-sessiond: page allocation failure: order:4, mode:0x1040d0
>>>> ...
>>>> [ 321.994711] lowmem_reserve[]: 0 0 0
>>>> [ 321.994727] DMA: 801*4kB (UEMC) 424*8kB (EMC) 355*16kB (UEMC) 344*32kB (MC)
>>>> 340*64kB (C) 8*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB
>>>> 0*16384kB 0*32768kB = 46068kB
>>>>
>>>> Signed-off-by: Martin Leisener <martin at leisener.de>
>>>> ---
>>>> lttng-events.c | 16 +++++++++++-----
>>>> 1 file changed, 11 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/lttng-events.c b/lttng-events.c
>>>> index 7eec04c..c861f71 100644
>>>> --- a/lttng-events.c
>>>> +++ b/lttng-events.c
>>>> @@ -39,6 +39,7 @@
>>>> #include "wrapper/file.h"
>>>> #include <linux/jhash.h>
>>>> #include <linux/uaccess.h>
>>>> +#include <linux/vmalloc.h>
>>>> #include "wrapper/uuid.h"
>>>> #include "wrapper/vmalloc.h" /* for wrapper_vmalloc_sync_all() */
>>>> @@ -132,10 +133,10 @@ struct lttng_session *lttng_session_create(void)
>>>> GFP_KERNEL);
>>>> if (!metadata_cache)
>>>> goto err_free_session;
>>>> - metadata_cache->data = kzalloc(METADATA_CACHE_DEFAULT_SIZE,
>>>> - GFP_KERNEL);
>>>> + metadata_cache->data = vmalloc(METADATA_CACHE_DEFAULT_SIZE);
>>>> if (!metadata_cache->data)
>>>> goto err_free_cache;
>>>> + memset(metadata_cache->data, 0, METADATA_CACHE_DEFAULT_SIZE);
>>>> metadata_cache->cache_alloc = METADATA_CACHE_DEFAULT_SIZE;
>>>> kref_init(&metadata_cache->refcount);
>>>> session->metadata_cache = metadata_cache;
>>>> @@ -162,7 +163,7 @@ void metadata_cache_destroy(struct kref *kref)
>>>> {
>>>> struct lttng_metadata_cache *cache =
>>>> container_of(kref, struct lttng_metadata_cache, refcount);
>>>> - kfree(cache->data);
>>>> + vfree(cache->data);
>>>> kfree(cache);
>>>> }
>>>> @@ -1523,10 +1524,15 @@ int lttng_metadata_printf(struct lttng_session *session,
>>>> tmp_cache_alloc_size = max_t(unsigned int,
>>>> session->metadata_cache->cache_alloc + len,
>>>> session->metadata_cache->cache_alloc << 1);
>>>> - tmp_cache_realloc = krealloc(session->metadata_cache->data,
>>>> - tmp_cache_alloc_size, GFP_KERNEL);
>>>> + tmp_cache_realloc = vmalloc(tmp_cache_alloc_size);
>>>> if (!tmp_cache_realloc)
>>>> goto err;
>>>> + else if (session->metadata_cache->data) {
>>>> + memset(tmp_cache_realloc, 0, tmp_cache_alloc_size);
>>>> + memcpy(tmp_cache_realloc, session->metadata_cache->data,
>>>> session->metadata_cache->cache_alloc);
>>>> + vfree(session->metadata_cache->data);
>>>> + }
>>>> +
>>>> session->metadata_cache->cache_alloc = tmp_cache_alloc_size;
>>>> session->metadata_cache->data = tmp_cache_realloc;
>>>> }
>>>> --
>>>> 1.7.9.5
>>>>
>>>>
>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev at lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
More information about the lttng-dev
mailing list