[lttng-dev] [PATCH lttng-tools] Fix: add missing rcu_barrier before daemon teardown

[Jonathan Rajotte Julien]

Tested it for teardown of lttng-sessiond.

No more core dump on SIGTERM teardown regarding :

#0  cds_lfht_lookup (ht=<optimized out>, hash=<optimized out>, 
match=match at entry=0x424220 <ht_match_reg_uid>, 
key=key at entry=0x7f050b7fba30, iter=iter at entry=0x7f050b7fba20)
at rculfhash.c:1553
#1  0x0000000000424a03 in buffer_reg_uid_find (session_id=<optimized 
out>, bits_per_long=<optimized out>, uid=<optimized out>) at 
buffer-registry.c:206
#2  0x000000000043291d in get_session_registry (ua_sess=<optimized out>) 
at ust-app.c:248
#3  0x0000000000437b0c in delete_ust_app_session (sock=sock at entry=120, 
ua_sess=0x7f04f00cd480, app=app at entry=0x7f04f00cc380) at ust-app.c:824
#4  0x0000000000437de5 in delete_ust_app (app=0x7f04f00cc380) at 
ust-app.c:904
#5  delete_ust_app_rcu (head=0x7f04f00cc430) at ust-app.c:948
#6  0x00007f051d0ef549 in call_rcu_thread (arg=0x7f04f00355b0) at 
urcu-call-rcu-impl.h:370
#7  0x00007f051c4ce6ba in start_thread (arg=0x7f050b7fe700) at 
pthread_create.c:333
#8  0x00007f051bdf782d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thanks


On 2016-12-21 05:59 PM, Mathieu Desnoyers wrote:
> When performing the "cleanup" of sessiond, consumerd, and relayd, we
> destroy data structures that may still be concurrently accessed by
> call_rcu worker thread.
>
> Ensure no more work is present in the call_rcu worker thread by issuing
> a rcu_barrier barrier. Note that this expects call_rcu handlers don't
> chain work to other call_rcu handlers.
>
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> ---
>   src/bin/lttng-consumerd/lttng-consumerd.c | 6 ++++++
>   src/bin/lttng-relayd/main.c               | 6 ++++++
>   src/bin/lttng-sessiond/main.c             | 6 ++++++
>   3 files changed, 18 insertions(+)
>
> diff --git a/src/bin/lttng-consumerd/lttng-consumerd.c b/src/bin/lttng-consumerd/lttng-consumerd.c
> index 1373a74..7f78c4e 100644
> --- a/src/bin/lttng-consumerd/lttng-consumerd.c
> +++ b/src/bin/lttng-consumerd/lttng-consumerd.c
> @@ -622,6 +622,12 @@ exit_init_data:
>   	tmp_ctx = ctx;
>   	ctx = NULL;
>   	cmm_barrier();	/* Clear ctx for signal handler. */
> +	/*
> +	 * Wait for all pending call_rcu work to complete before tearing
> +	 * down data structures. call_rcu worker may be trying to
> +	 * perform lookups in those structures.
> +	 */
> +	rcu_barrier();
>   	lttng_consumer_destroy(tmp_ctx);
>   	lttng_consumer_cleanup();
>   
> diff --git a/src/bin/lttng-relayd/main.c b/src/bin/lttng-relayd/main.c
> index dc19a69..9f26be5 100644
> --- a/src/bin/lttng-relayd/main.c
> +++ b/src/bin/lttng-relayd/main.c
> @@ -2932,6 +2932,12 @@ exit_init_data:
>   	health_app_destroy(health_relayd);
>   exit_health_app_create:
>   exit_options:
> +	/*
> +	 * Wait for all pending call_rcu work to complete before tearing
> +	 * down data structures. call_rcu worker may be trying to
> +	 * perform lookups in those structures.
> +	 */
> +	rcu_barrier();
>   	relayd_cleanup();
>   
>   	/* Ensure all prior call_rcu are done. */
> diff --git a/src/bin/lttng-sessiond/main.c b/src/bin/lttng-sessiond/main.c
> index 85a3518..40670dd 100644
> --- a/src/bin/lttng-sessiond/main.c
> +++ b/src/bin/lttng-sessiond/main.c
> @@ -6148,6 +6148,12 @@ exit_health:
>   
>   exit_init_data:
>   	/*
> +	 * Wait for all pending call_rcu work to complete before tearing
> +	 * down data structures. call_rcu worker may be trying to
> +	 * perform lookups in those structures.
> +	 */
> +	rcu_barrier();
> +	/*
>   	 * sessiond_cleanup() is called when no other thread is running, except
>   	 * the ht_cleanup thread, which is needed to destroy the hash tables.
>   	 */

-- 
Jonathan R. Julien
Efficios