[lttng-dev] High memory consumption issue on RCU side

[Mathieu Desnoyers]

----- On Sep 24, 2016, at 11:22 AM, Paul E. McKenney paulmck at linux.vnet.ibm.com wrote:

> On Sat, Sep 24, 2016 at 10:42:24AM +0300, Evgeniy Ivanov wrote:
>> Hi Mathieu,
>> 
>> On Sat, Sep 24, 2016 at 12:59 AM, Mathieu Desnoyers
>> <mathieu.desnoyers at efficios.com> wrote:
>> > ----- On Sep 22, 2016, at 3:14 PM, Evgeniy Ivanov lolkaantimat at gmail.com wrote:
>> >
>> >> Hi all,
>> >>
>> >> I'm investigating high memory usage of my program: RSS varies between
>> >> executions in range 20-50 GB, though it should be determenistic. I've
>> >> found that all the memory is allocated in this stack:
>> >>
>> >> Allocated 17673781248 bytes in 556 allocations
>> >>        cds_lfht_alloc_bucket_table3     from liburcu-cds.so.2.0.0
>> >>        _do_cds_lfht_resize      from liburcu-cds.so.2.0.0
>> >>        do_resize_cb             from liburcu-cds.so.2.0.0
>> >>        call_rcu_thread          from liburcu-qsbr.so.2.0.0
>> >>        start_thread             from libpthread-2.12.so
>> >>        clone                    from libc-2.12.so
>> >>
>> >> According pstack it should be quiescent state.  Call thread waits on syscall:
>> >> syscall
>> >> call_rcu_thread
>> >> start_thread
>> >> clone
>> >>
>> >> We use urcu-0.8.7, only rculfhash (QSBR). Is it some kind of leak in
>> >> RCU or any chance I misuse it? What would you recommend to
>> >> troubleshoot the situation?
>> >
>> > urcu-qsbr is the fastest flavor of urcu, but it is rather tricky to use well.
>> > Make sure that:
>> >
>> > - Each registered thread periodically reach a quiescent state, by:
>> >   - Invoking rcu_quiescent_state periodically, and
>> >   - Making sure to surround any blocking for relatively large amount of
>> >     time by rcu_thread_offline()/rcu_thread_online().
>> >
>> > In urcu-qsbr, the "default" state of threads is to be within a RCU read-side.
>> > Therefore, if you omit any of the two advice above, you end up in a situation
>> > where grace periods never complete, and therefore no call_rcu() callbacks can
>> > be processed. This effectively acts like a big memory leak.
>> 
>> It was the original assumption, but in memory stacks I don't see such
>> allocations for my data. Instead huge allocations happen right in
>> call_rcu_thread. Memory footprint for my app is about 20 GB, erasing
>> RCU data is a rare operation, so almost 20 GB in rcu thread looks
>> suspecios. I'll try to not erase any RCU protected data and reproduce
>> the issue (complicated thing is that under memory tracer it happens
>> not so often).
> 
> Interesting.  Trying to figure out why your call_rcu_thread() would
> ever allocate memory.
> 
> Ah!  Do your RCU callbacks allocate memory?

In this case yes: urculfhash allocates memory within a call rcu worker
thread when a hash table resize is performed.

Thanks,

Mathieu

> 
>							Thanx, Paul
> 
>> > Hoping this helps,
>> >
>> > Thanks,
>> >
>> > Mathieu
>> >
>> >
>> > --
>> > Mathieu Desnoyers
>> > EfficiOS Inc.
>> > http://www.efficios.com
>> 
>> 
>> 
>> --
>> Cheers,
>> Evgeniy

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com