[lttng-dev] High memory consumption issue on RCU side

[Mathieu Desnoyers]

----- On Sep 22, 2016, at 3:14 PM, Evgeniy Ivanov lolkaantimat at gmail.com wrote:

> Hi all,
> 
> I'm investigating high memory usage of my program: RSS varies between
> executions in range 20-50 GB, though it should be determenistic. I've
> found that all the memory is allocated in this stack:
> 
> Allocated 17673781248 bytes in 556 allocations
>        cds_lfht_alloc_bucket_table3     from liburcu-cds.so.2.0.0
>        _do_cds_lfht_resize      from liburcu-cds.so.2.0.0
>        do_resize_cb             from liburcu-cds.so.2.0.0
>        call_rcu_thread          from liburcu-qsbr.so.2.0.0
>        start_thread             from libpthread-2.12.so
>        clone                    from libc-2.12.so
> 
> According pstack it should be quiescent state.  Call thread waits on syscall:
> syscall
> call_rcu_thread
> start_thread
> clone
> 
> We use urcu-0.8.7, only rculfhash (QSBR). Is it some kind of leak in
> RCU or any chance I misuse it? What would you recommend to
> troubleshoot the situation?

urcu-qsbr is the fastest flavor of urcu, but it is rather tricky to use well.
Make sure that:

- Each registered thread periodically reach a quiescent state, by:
  - Invoking rcu_quiescent_state periodically, and
  - Making sure to surround any blocking for relatively large amount of
    time by rcu_thread_offline()/rcu_thread_online().

In urcu-qsbr, the "default" state of threads is to be within a RCU read-side.
Therefore, if you omit any of the two advice above, you end up in a situation
where grace periods never complete, and therefore no call_rcu() callbacks can
be processed. This effectively acts like a big memory leak.

Hoping this helps,

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com