[lttng-dev] [-stable 3.8.1 performance regression] madvise POSIX_FADV_DONTNEED

[Mathieu Desnoyers]

* Dave Chinner (david at fromorbit.com) wrote:
> On Wed, Jul 03, 2013 at 08:31:03PM -0400, Mathieu Desnoyers wrote:
> > * Dave Chinner (david at fromorbit.com) wrote:
> > > On Wed, Jul 03, 2013 at 10:53:08AM -0400, Jeff Moyer wrote:
> > > > Mel Gorman <mgorman at suse.de> writes:
> > > > 
> > > > >> > I just tried replacing my sync_file_range()+fadvise() calls and instead
> > > > >> > pass the O_DIRECT flag to open(). Unfortunately, I must be doing
> > > > >> > something very wrong, because I get only 1/3rd of the throughput, and
> > > > >> > the page cache fills up. Any idea why ?
> > > > >> 
> > > > >> Since O_DIRECT does not seem to provide acceptable throughput, it may be
> > > > >> interesting to investigate other ways to lessen the latency impact of
> > > > >> the fadvise DONTNEED hint.
> > > > >> 
> > > > >
> > > > > There are cases where O_DIRECT falls back to buffered IO which is why you
> > > > > might have found that page cache was still filling up. There are a few
> > > > > reasons why this can happen but I would guess the common cause is that
> > > > > the range of pages being written was in the page cache already and could
> > > > > not be invalidated for some reason. I'm guessing this is the common case
> > > > > for page cache filling even with O_DIRECT but would not bet money on it
> > > > > as it's not a problem I investigated before.
> > > > 
> > > > Even when O_DIRECT falls back to buffered I/O for writes, it will
> > > > invalidate the page cache range described by the buffered I/O once it
> > > > completes.  For reads, the range is written out synchronously before the
> > > > direct I/O is issued.  Either way, you shouldn't see the page cache
> > > > filling up.
> > > 
> > > <sigh>
> > > 
> > > I keep forgetting that filesystems other than XFS have sub-optimal
> > > direct IO implementations. I wish that "silent fallback to buffered
> > > IO" idea had never seen the light of day, and that filesystems
> > > implemented direct IO properly.
> > > 
> > > > Switching to O_DIRECT often incurs a performance hit, especially if the
> > > > application does not submit more than one I/O at a time.  Remember,
> > > > you're not getting readahead, and you're not getting the benefit of the
> > > > writeback code submitting batches of I/O.
> > > 
> > > With the way IO is being done, there won't be any readahead (write
> > > only workload) and they are directly controlling writeback one chunk
> > > at a time, so there's not writeback caching to do batching, either.
> > > There's no obvious reason that direct IO should be any slower
> > > assuming that the application is actually doing 1MB sized and
> > > aligned IOs like was mentioned, because both methods are directly
> > > dispatching and then waiting for IO completion.
> > 
> > As a clarification, I use 256kB "chunks" (sub-buffers) in my tests, not
> > 1MB. Also, please note that since I'm using splice(), each individual
> > splice call is internally limited to 16 pages worth of data transfer
> > (64kB).
> 
> So you are doing 4 write() calls to a single sync_file_range(), not
> 1:1?

Yes, my test case did 4 splice() calls (each writing 64kB) for each
256kB sync_file_range()+fadvise().

> 
> > > What filesystem is in use here?
> > 
> > My test was performed on ext3 filesystem, that was itself sitting on
> > raid-1 software raid.
> 
> Yup, ext3 has limitations on direct IO that requires block
> allocation and will often fall back to buffered IO. ext4  and XFS
> should not have the same problems...

I'm not really keen on making good tracer performance so dependent on
which file system happens to be used on the system, unless there would
be a nice ABI (e.g. fcntl(), IOW for some arguable values of "nice" ;-) )
to ask the kernel if the file system being written to supports efficient
O_DIRECT or not.

If we had this kind of ABI, then I guess we could indeed use O_DIRECT as
an optimisation in some cases, but it unfortunately does not seem to
solve the latency regression we are seeing when using fadvise
POSIX_FADV_DONTNEED in the general case where we need to use buffered
I/O, but want to play nicely with the page cache. :-/

Thanks,

Mathieu

> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david at fromorbit.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com