[lttng-dev] Making 32-bit user-space events on a 64-bit Linux system

[Mathieu Desnoyers]

----- Original Message -----
> From: "Daniel Thibault" <Daniel.Thibault at drdc-rddc.gc.ca>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>, "Alexandre Montplaisir" <alexmonthy at voxpopuli.im>
> Cc: lttng-dev at lists.lttng.org
> Sent: Monday, October 7, 2013 10:40:28 AM
> Subject: RE: [lttng-dev] Making 32-bit user-space events on a 64-bit Linux system
> 
> -----Message d'origine-----
> De : Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com]
> Envoyé : 7 octobre 2013 08:57
> 
> ----- Original Message -----
> > From: "Alexandre Montplaisir" <alexmonthy at voxpopuli.im>
> > Sent: Thursday, October 3, 2013 11:54:15 AM
> > 
> > >    On a different note, why does LTTng have 32- and 64-bit consumers
> > >    anyway?  The only possible difference between a 32-bit tracer and
> > > a 64-bit tracer is that the latter could generate event record
> > > fields that are 64-bit longs and floats, while the former cannot
> > > (but will never have to, since the application it is tracing won't
> > > have such beasts anyway).
> >
> > No, this is not it at all. 32-bit apps can generate 64-bit integers too,
> > and double-precision floats.
> 
>    I realised that soon after having written the above, when I looked up the
>    C spec.  Clearly, the event records are precisely the same in 32- and
>    64-bit modes.  But if the event buffers don't have intrinsic bitness,
>    something else must be responsible for the requirement for 32-bit
>    consumers.  Does mmap have a bitness barrier?  Or do the event buffers
>    have intrinsic bitness in the sense that the buffer management fields
>    change?
> 
> > The difference between 32 and 64-bit consumers mainly comes for atomic
> > operations: on 64-bit,
> > consumer and application can interact through atomic operations on 64-bit
> > integers, which allow
> > detection of free-running counter overflow much larger than with 32-bit
> > counter.
> >
> > Mathieu
> 
>    If I understand you correctly, the 64-bit atomic operations needed are not
>    available in the 32-bit assembly code.  Makes sense.  One also sees that
>    the dynamic loader has 32- and 64-bit operating modes, due mostly to
>    different ELF formats.  But is there anything in the processor
>    architecture that makes it impossible (or undesirable because of, say,
>    some sort of context-switching) for a 64-bit processor running 32-bit
>    code to insert 64-bit instructions in the code flow?  Alternately, how
>    hard is it for 32-bit code to call a 64-bit library?  (In some ways, this
>    is similar to inserting an external call or a piece of assembly code in
>    high-level language code such as Object Pascal)  Taking yet another step
>    back, what is it about the event buffer lying in a 32-bit process's space
>    that makes it undesirable/difficult/impossible for a 64-bit consumer to
>    access it?
>    Obviously, when running on a 32-bit processor LTTng has no choice but to
>    be 32-bit wide throughout, and therefore it cannot use the desirable
>    64-bit integer atomic comparisons.  Running on a 64-bit processor should
>    be a wholly different kettle of fish, however.

Typically OSes don't allow 32-bit compatibility processes to use the upper register bits (those are cleared upon entry into the OS). So 32-bit processes typically never call a 64-bit library.

We chose to have a consumer daemon that match the traced application ABI bitness so the consumer can link to the liblttng-ust ring buffer implementation, so we don't have to duplicate code across multiple code-bases, and don't have to partially implement pieces of the ring buffer into different code-bases, compiled across different ABIs, talking to each other through shared data structures over shared memory (posix shm). In summary, this decision has been done for simplicity of implementation.

Having a 64-bit consumer that could deal with both 32-bit and 64-bit processes could eventually be done. The main reason why it has not been done originally was that we expected a large change in the UST consumer (it came in the 2.2 release with per-UID UST buffers), which was much easier to do if the ring buffer was sitting in its own library, used by the consumer daemon. We could revisit this eventually if there is sufficient interest for this and funding available.

Thanks,

Mathieu


> 
> Daniel U. Thibault
> Protection des systèmes et contremesures (PSC) | Systems Protection &
> Countermeasures (SPC)
> Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber
> Security (MCCS)
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D
> Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Québec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com