[lttng-dev] tracing of non-JNI java in IBM jvm
Aaron Spear
aspear at vmware.com
Wed Jan 23 14:45:08 EST 2013
Hi Erik,
I am happy to report that I already implemented what I mentioned before as a *prototype*. My solution has the following:
-JNI library and classes that use UST to log function entries-exits. So I end up with a CTF log that has function entries and exits.
-extension to in-trace agent library to use this library for function entries and exits. So, you can select a subset of classes to instrument dynamically.
-clone of the Control Flow View that understands this particular function entry and exit event schema
-I have threads as the parent, and then whatever functions the thread calls as children. If you collapse the thread, you still see intervals over which it is running which is nice.
The net result is that you can see Gantt charts of function calls in *any* java application. All you have to do is
1) launch the java app injecting the custom in-trace agent
2) start lttng ust tracing
3) connect to the java app's injected agent and select what classes you want instrumented
See the attached screenshot that shows a very simple java application that spawns 5 threads, and each thread calls 4 functions and then exits.
I think that the idea is pretty cool and have the intent to contribute it if there is interest. This implementation is not yet ready to show anyone as I hacked it together for a paper that I am writing. Here are a few of the caveats about it:
1) the current implementation is quite intrusive on the java side. Sloppy use of strings in particular. It needs some careful architecting. I have a number of ideas on this front, but have not yet had bandwidth to implement them.
2) There are lots of details to be worked out with the in-trace integration. The current collection UI really needs some help. I tried briefly to get the eclipse plugin to work with Eclipse 4, but only the RCP app seems to work for me. I think that the "Connections" view needs to have different collection agents plugged in and you should be able to connect to a java app and dynamically select classes to instrument and/or select from sets of classes/methods.
3) My vision is to create a viewer that could do arbitrary state over time, and use CTF extensions to feed this so that the implementation is entirely data driven. Basically any hierarchy of parent-child entries with state under them. This particular one is, like the LTTng Control Flow View, hard coded to the event schema right now, but I can see how we can make it more general. I should say that the guys that wrote the framework did a good job of making it useable for different things (nice work guys!)
Note that you could in fact use it for instrumenting native applications as well of course. The "DynamoRIO" project in particular does the same sort of thing as In Trace does (only more efficiently) so I think this is quite viable as a collection mechanism.
cheers!
Aaron Spear
----- Original Message -----
> From: "Erik Ostermueller" <Erik.Ostermueller at fisglobal.com>
> To: "Aaron Spear" <aspear at vmware.com>
> Cc: lttng-dev at lists.lttng.org, "Alexandre Montplaisir" <alexmonthy at voxpopuli.im>
> Sent: Wednesday, January 23, 2013 11:03:15 AM
> Subject: RE: [lttng-dev] tracing of non-JNI java in IBM jvm
>
> Aaron,
>
> Sorry to have disappeared from this discussion.
> What if we went is a slightly different direction.
>
> The InTrace tool (http://mchr3k.github.com/org.intrace/) will log
> start/stop times of dynamically instrumented methods to a text file
> (proprietary format).
> What if we simply wrote a process to convert that log to CTF, and
> view the trace with the lttng eclipse plug-in, unmodified.
>
> The idea would be to, at first, just support these two views:
> http://wiki.eclipse.org/Linux_Tools_Project/LTTng/User_Guide#Control_Flow_View
> http://wiki.eclipse.org/Linux_Tools_Project/LTTng/User_Guide#Statistics_View
>
> --Erik
>
>
>
> -----Original Message-----
> From: Aaron Spear [mailto:aspear at vmware.com]
> Sent: Friday, January 11, 2013 12:56 PM
> To: Ostermueller, Erik
> Cc: lttng-dev at lists.lttng.org; Alexandre Montplaisir
> Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
>
>
> ----- Original Message -----
> > From: "Erik Ostermueller" <Erik.Ostermueller at fisglobal.com>
> > To: "Alexandre Montplaisir" <alexmonthy at voxpopuli.im>
> > Cc: lttng-dev at lists.lttng.org
> > Sent: Friday, January 11, 2013 9:26:51 AM
> > Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> >
> > Thanks Alex for the quick reply....now I get it.
> > I'll jump in on that other thread you mentioned.
> >
> > There is a tool called InTrace that does some of this work.
> > Perhaps some design ideas or even code could be stolen from here:
> > http://mchr3k.github.com/org.intrace/
> > This project:
> > * implements the java.lang.instrumentation api, which can be used
> > to
> > plug a tracing tool into an already-running JVM.
> > * allows you to instrument/de-instrument various classes without
> > restarting the JVM.
> > * Uses http://asm.ow2.org/ to inject byte code for entry/exit
> > tracing
> > methods.
> > * stores thread id (good) but does nothing to sort/organize
> > activity
> > by thread id into a session(bad)
> > * no graphing/aggregating of performance metrics...just displays
> > entry/exit times, method parameters, etc...
>
> Hi Erik,
>
> I think that what you are mentioning is certainly one thing that is
> needed. To me I see a couple of different components here:
> 1) implementation of java.util.logging interfaces as well as SLF4J
> interfaces, as well as a dedicated LTTng java API for registering
> custom event typess. This would be used for static, compile time
> instrumentation of java apps.
> 2) an implementation that uses the tooling you mention above for
> dynamic instrumentation of any running java app. You basically
> would define trace and install trace points on the fly. This would
> be really, really powerful.
>
> The implementations of the above would have to be quite different
> from the way that things are done currently with UST. Currently UST
> uses macros in C files to define the various tracepoints statically.
> We would need to have API's that could be called dynamically to
> define tracepoints, so that the java code could do this. I have not
> actually looked under the hood with the current UST implementation
> to see what the implications are for this.
>
> It seems to me that we would want to have a Java API usage similar to
> what you can do with SLF4J:
>
> class MyClass {
> private static final Tracer tracer =
> LTTngFactory.getTracer("MyComponentName");
> private static final String MY_TRACEPOINT =
> "MyTracepointWithIntegerStringAndLong";
>
> static {
> // register a tracepoint. this goes through JNI and
> creates a tracepoint
> // in UST that has the given name and the specified objects
> tracer.registerTracepoint(MY_TRACEPOINT,
> new
> Object[]{Integer.class,String.class,Long.class});
> }
>
> public void someMethod() {
> Integer someInteger = 123456;
> String someString = "this is the string";
> Long someLong = 56789;
>
> tracer.trace(MY_TRACEPOINT,someInteger,someString,someLong);
> }
> }
>
> Then over the top of this you have implementations of SLF4J,
> java.util.logging, whatever. The infrastructure required in the JNI
> side for this would be the same implementation needed by the binding
> to the InTrace technology to dynamically create tracepoints.
>
> I need all of this right now, and am willing to spend time in the
> next few months to implement it.
>
> cheers,
> Aaron Spear, Staff Engineer, VMware Inc
>
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i)
> delete the message and all copies; (ii) do not disclose, distribute
> or use the message in any manner; and (iii) notify the sender
> immediately. In addition, please be aware that any message addressed
> to our domain is subject to archiving and review by persons other
> than the intended recipient. Thank you
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: execution-flow-view.png
Type: image/png
Size: 197827 bytes
Desc: not available
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20130123/176069c1/attachment-0001.png>
More information about the lttng-dev
mailing list