[lttng-dev] Tmf Ctf Parser Scalability issues (1/3)

[Mathieu Desnoyers]

----- Original Message -----
> From: "Matthew Khouzam" <matthew.khouzam at ericsson.com>
> To: lttng-dev at lists.lttng.org, "Linux Tools developer discussions" <linuxtools-dev at eclipse.org>
> Sent: Tuesday, November 26, 2013 11:44:46 AM
> Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (1/3)
> 
> Hello all,
> I was looking at the CTF parser in the TMF project of Linux Tools. I
> have come up with three points where scalability will be an issue. I am
> sending 3 emails, each one describing one of the issues so we can
> aggregate them more cohesively. First a primer, CTF is a file format
> that has traces written in packets, the packets are parts of files in
> streams, the streams are files in a directory.
> 
> Issue 1: Number of open files
> CTF can have an unlimited amount of stream files. The parser opens all
> of them simultaneously. This means if you have say over 1024 or whatever
> your file limit is, files opened simultaneously, you will hit a wall. An
> example of this would be the new Xeon Phi boards that have something
> like 255 cores, if you have more than 4 channels per core, you're going
> to hit this issue.
> 
> Proposed solution:
> I envision fixing them by setting up a pool of files and when the limit
> is hit, swapping files in and out, incurring a performance hit but still
> allowing the program to run. Any objections?

No objection. I would think a LRU list of most recently used files would
be a good data structure for this. I already described this in a bug
against babeltrace: http://bugs.lttng.org/issues/649

Thanks!

Mathieu

> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com