[lttng-dev] Problem to get lttng work with a daemon application
Hans Nordebäck
hans.nordeback at ericsson.com
Fri Sep 14 02:36:35 EDT 2012
Hi Mathieu, the daemon is not closing file descriptors at this point, later it will schedule and daemonize applications,
by closing fd's up to SC_OPEN_MAX. An alternative to the getenv solution could be to add two new functions:
void lttng_ust_reinit(void)
{
lttng_ust_cleanup_for_reinit();
lttng_ust_init();
}
but the getenv solution seems ok.
/Thanks Hans
-----Original Message-----
From: Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com]
Sent: den 13 september 2012 15:38
To: Hans Nordebäck
Cc: lttng-dev at lists.lttng.org
Subject: Re: [lttng-dev] Problem to get lttng work with a daemon application
* Hans Nordebäck (hans.nordeback at ericsson.com) wrote:
> Hi Mathieu, I have installed lttng-ust 2.0.5 and lttng-tools 2.0.4, I
> run a gdb session and realized what the problem is. I noticed that
> lttng threads are created before main via __attribute__(constructor),
> and the problem with the daemon is that the lttng threads "disappears" after "daemonizing" the daemon, causing recvmsg in ustcomm_recv_unix_sock to hang. I removed the __attribute__(constructor) and called
> lttng_ust_init explicitly, it works, but I have to search for a more general solution./Thanks Hans
Ah! yes!
how do you deamonize your daemon ? Do you, for instance, close all file descriptors between 0 and 100 ? This will not behave well with lttng-ust. The following bug entry is keeping track of this issue:
http://bugs.lttng.org/issues/253
Indeed, initializing lttng-ust only after the file descriptors has been closed could work. Maybe we could do a work-around for this issue, where we could wrap the constructor within a getenv("LTTNG_UST_MANUAL_INIT") == "1" strcmp check, and, in that case, require that the application call some init API provided by lttng-ust. However, given that we want liblttng-ust to be a dependency that is not strictly required by applications (such as the demo-trace example in tests/demo), we'd need to look up the symbol of the initialization function with dlsym().
Thoughts ?
Thanks,
Mathieu
>
> -----Original Message-----
> From: Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com]
> Sent: den 12 september 2012 15:52
> To: Hans Nordebäck
> Cc: lttng-dev at lists.lttng.org
> Subject: Re: [lttng-dev] Problem to get lttng work with a daemon
> application
>
> * Hans Nordebäck (hans.nordeback at ericsson.com) wrote:
> > Hi Mathieu, I upgraded to version 2.0.4, but it did not solve the
> > problem./Thanks Hans
>
> Hi Hans,
>
> Please provide the version numbers you use for both lttng-tools and lttng-ust. The lttng.org website lists the latest lttng-ust as 2.0.5, which leads me to think you might have forgotten to upgrade it.
>
> Don't forget to restart your test application (just to be sure, rebuild it against the new lttng-ust). Also make sure you kill all lttng-sessiond that might still be active.
>
> Thanks,
>
> Mathieu
>
> >
> > -----Original Message-----
> > From: Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com]
> > Sent: den 12 september 2012 14:42
> > To: Hans Nordebäck
> > Cc: lttng-dev at lists.lttng.org
> > Subject: Re: [lttng-dev] Problem to get lttng work with a daemon
> > application
> >
> > * Hans Nordebäck (hans.nordeback at ericsson.com) wrote:
> > > Hi, I've just started using lttng and run it in Ubuntu 12.04 in VirtualBox.
> > >
> > > I have run the sample applications without any problem, but I have
> > > problems to get a daemon application work.
> > >
> > > I have added tracepoint support in the same way as for the sample
> > > applications and do the following:
> > >
> > > 1) The lttng-sessiond (version 2.0.1) is started (with
> >
> > 2.0.1 is old. Please upgrade to the latest versions of the 2.0.x series for both lttng-tools and lttng-ust and try again.
> >
> > Thanks,
> >
> > Mathieu
> >
> > > --verbose--verbose-consumer).
> > > 2) I start the daemon application with one tracepoint implemented.
> > > 3) I can see that the lttng-sessiond gets called regarding this tracepoint:
> > >
> > > DEBUG1: App registered with pid:18421 ppid:18419 uid:0 gid:0
> > > sock:18 name:osafamfnd (version 2.0) [in ust_app_register() at
> > > ust-app.c:1365]
> > >
> > > 4) I run, strace -f lttng -v list -u, and can see that it "hangs"
> > > in
> > > recvmsg:
> > > write(2, "DEBUG1: Getting UST tracing even"..., 81DEBUG1: Getting UST
> > > tracing events [in list_ust_events() at commands/list.c:253]
> > > ) = 81
> > > getuid() = 0
> > > socket(PF_FILE, SOCK_STREAM, 0) = 3
> > > connect(3, {sa_family=AF_FILE,
> > > path="/var/run/lttng/client-lttng-sessiond"}, 110) = 0
> > > geteuid() = 0
> > > getegid() = 0
> > > sendmsg(3, {msg_name(0)=NULL,
> > > msg_iov(1)=[{"\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
> > > 0\ 0\ 0\0\0\0\0"..., 8784}], msg_controllen=28, {cmsg_len=28,
> > > cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=28878, uid=0,
> > > gid=0}}, msg_flags=0}, 0) =
> > > 8784
> > > recvmsg(3,
> > >
> > > 5) the lttng-sessiond outputs only the following and hangs as long
> > > as the daemon application is running.
> > > DEBUG1: Processing client command 14 [in process_client_msg() at
> > > main.c:3309]
> > >
> > > 6) The only difference between sample app logs and daemon app logs
> > > is that the pid and ppid seen at DEBUG1: App registered. In the
> > > daemon case in this run it is pid 18421 ppid 18419 but when I
> > > print it in the function calling tracepoint it is pid 18430. I
> > > don't know if that is related to the problem. Any help on this would be appreciated.
> > >
> > > /Thanks Hans
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > lttng-dev mailing list
> > > lttng-dev at lists.lttng.org
> > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > --
> > Mathieu Desnoyers
> > Operating System Efficiency R&D Consultant EfficiOS Inc.
> > http://www.efficios.com
>
> --
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant EfficiOS Inc.
> http://www.efficios.com
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant EfficiOS Inc.
http://www.efficios.com
More information about the lttng-dev
mailing list