[lttng-dev] 回复:Re: 回复:Re: 回复:Re: Pros and Cons of LTTng

[Jonathan Rajotte-Julien]

Hi,

On Wed, Nov 06, 2019 at 11:59:41AM +0800, 杨海 wrote:
> Thanks Jonathan. Regarding to the CI MTTR/MTTF test results, it varies from time to time, and on master/stable branches. 
> 1. it monitors new Linux kernels, so the CI job may use newer kernel version than months ago?

Not sure of the question here but here more information on how to jobs works.

The job monitor all kernel tags that we deems pertinent for example for the
vanilla kernel (linux-stable) with the lttng-modules master branch [1]:

  09:13:52 Building the following kernel versions:
  09:13:52 v3.0.101
  09:13:52 v3.1.10
  09:13:52 v3.2.102
  09:13:52 v3.3.8
  09:13:52 v3.4.113
  09:13:52 v3.5.7
  09:13:52 v3.6.11
  09:13:52 v3.7.10
  09:13:52 v3.8.13
  09:13:52 v3.9.11
  09:13:52 v3.10.108
  09:13:52 v3.11.10
  09:13:52 v3.12.74
  09:13:52 v3.13.11
  09:13:52 v3.14.79
  09:13:52 v3.15.10
  09:13:52 v3.16.76
  09:13:52 v3.17.8
  09:13:52 v3.18.140
  09:13:52 v3.19.8
  09:13:52 v4.0.9
  09:13:52 v4.1.52
  09:13:52 v4.2.8
  09:13:52 v4.3.6
  09:13:52 v4.4.199
  09:13:52 v4.5.7
  09:13:52 v4.6.7
  09:13:52 v4.7.10
  09:13:52 v4.8.17
  09:13:52 v4.9.199
  09:13:52 v4.10.17
  09:13:52 v4.11.12
  09:13:52 v4.12.14
  09:13:52 v4.13.16
  09:13:52 v4.14.152
  09:13:52 v4.15.18
  09:13:52 v4.16.18
  09:13:52 v4.17.19
  09:13:52 v4.18.20
  09:13:52 v4.19.82
  09:13:52 v4.20.17
  09:13:52 v5.0.21
  09:13:52 v5.1.21
  09:13:52 v5.2.21
  09:13:52 v5.3.9
  09:13:52 v5.4-rc6

We always track the latest tag of each branch. This does not mean that we do not
support smaller tag of a branch, only that support is a best effort based on
user feedback. Tracking all tags would represent a monumental effort and would
require way more resource overall. Not something we are against, simply that we
do not have the resource for it.

[1] https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_master_build-vanilla/

Note the last rc kernel tag.

Note that we also perform this for some distros and their kernels:

https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_master_crossbuild-xenial/
https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_master_crossbuild-bionic/

I guess that it is true that we do not test against past kernel tag for each
branch. Given our history and process, most time that lttng-modules is broken is
that distro are taking liberty on patch backporting. These get fixed asap and
backported to all supported lttng-modules stable branches. 
> 2. What would be the criteria of MTBF here?

Note that these statistics have little to no value here due to the presence of the
testing against RC tags. These tags will breaks lttng-modules one way or another
and it is a good thing. This allows us to keep up to date with upstream.
The only pertinent value is the MTTR here. It shows our responsiveness to
external change.

A better representation of our MTTR would be the jobs following distros:

https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_master_build-xenial/

MTTR    Last 7 Days    0 ms
        Last 30 Days   0 ms
        All Time       15 hr
MTTF    Last 7 Days    0 ms
        Last 30 Days   0 ms
        All Time       4 mo 5 days

Here our MTTR is 15h. Which is quite quick considering the context of tracking
kernel outside of our control.

MTTR is from the moment the CI detect a problem and that we provide a fix for
it.

MTTF is not something we can have control over since the source is completely
external. Except for the rare event that the infrastructure is deficient.

TBH these statistic have little to no meaning in this context.

Cheers