Skip to content

Commit

Permalink
perf intel-pt: Parse VM Time Correlation options and set up decoding
Browse files Browse the repository at this point in the history 
Add parsing and validation of VM Time Correlation options, and pass
parameters to the decoder. Also update the Intel PT documentation
accordingly.

Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: https://lore.kernel.org/r/20210430070309.17624-13-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
  • Loading branch information
Adrian Hunter authored and Arnaldo Carvalho de Melo committed May 12, 2021
1 parent fa8f949 commit e3ff42b
Show file tree
Hide file tree
Showing 2 changed files with 202 additions and 4 deletions.
108 changes: 105 additions & 3 deletions tools/perf/Documentation/perf-intel-pt.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1155,8 +1155,9 @@ include::build-xed.txt[]
Tracing Virtual Machines
------------------------

Currently, only kernel tracing is supported and only with "timeless" decoding
i.e. no TSC timestamps
Currently, only kernel tracing is supported and only with either "timeless" decoding
(i.e. no TSC timestamps) or VM Time Correlation. VM Time Correlation is an extra step
using 'perf inject' and requires unchanging VMX TSC Offset and no VMX TSC Scaling.

Other limitations and caveats

Expand All @@ -1167,7 +1168,7 @@ Other limitations and caveats
Guest VCPU is unknown but may be able to be inferred from the host thread
Callchains are not supported

Example
Example using "timeless" decoding

Start VM

Expand Down Expand Up @@ -1231,6 +1232,107 @@ perf script can be used to provide an instruction trace
:1440 1440 ffffffffbb74603c clockevents_program_event+0x4c ([guest.kernel.kallsyms]) popq %rbx
:1440 1440 ffffffffbb74603d clockevents_program_event+0x4d ([guest.kernel.kallsyms]) popq %r12

Example using VM Time Correlation

Start VM

$ sudo virsh start kubuntu20.04
Domain kubuntu20.04 started

Mount the guest file system. Note sshfs needs -o direct_io to enable reading of proc files. root access is needed to read /proc/kcore.

$ mkdir -p vm0
$ sshfs -o direct_io root@vm0:/ vm0

Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore

$ perf buildid-cache -v --kcore vm0/proc/kcore
same kcore found in /home/user/.debug/[kernel.kcore]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777
$ KALLSYMS=/home/user/.debug/\[kernel.kcore\]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777/kallsyms

Find the VM process

$ ps -eLl | grep 'KVM\|PID'
F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
3 S 64055 16998 1 17005 13 80 0 - 1818189 - ? 00:00:16 CPU 0/KVM
3 S 64055 16998 1 17006 4 80 0 - 1818189 - ? 00:00:05 CPU 1/KVM
3 S 64055 16998 1 17007 3 80 0 - 1818189 - ? 00:00:04 CPU 2/KVM
3 S 64055 16998 1 17008 4 80 0 - 1818189 - ? 00:00:05 CPU 3/KVM

Start an open-ended perf record, tracing the VM process, do something on the VM, and then ctrl-C to stop.
IPC can be determined, hence cyc=1 can be added.
Only kernel decoding is supported, so 'k' must be specified.
Intel PT traces both the host and the guest so --guest and --host need to be specified.

$ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/cyc=1/k -p 16998
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 9.041 MB perf.data.kvm ]

Now 'perf inject' can be used to determine the VMX TCS Offset. Note, Intel PT TSC packets are
only 7-bytes, so the TSC Offset might differ from the actual value in the 8th byte. That will
have no effect i.e. the resulting timestamps will be correct anyway.

$ perf inject -i perf.data.kvm --vm-time-correlation=dry-run
ERROR: Unknown TSC Offset for VMCS 0x1bff6a
VMCS: 0x1bff6a TSC Offset 0xffffe42722c64c41
ERROR: Unknown TSC Offset for VMCS 0x1cbc08
VMCS: 0x1cbc08 TSC Offset 0xffffe42722c64c41
ERROR: Unknown TSC Offset for VMCS 0x1c3ce8
VMCS: 0x1c3ce8 TSC Offset 0xffffe42722c64c41
ERROR: Unknown TSC Offset for VMCS 0x1cbce9
VMCS: 0x1cbce9 TSC Offset 0xffffe42722c64c41

Each virtual CPU has a different Virtual Machine Control Structure (VMCS)
shown above with the calculated TSC Offset. For an unchanging TSC Offset
they should all be the same for the same virtual machine.

Now that the TSC Offset is known, it can be provided to 'perf inject'

$ perf inject -i perf.data.kvm --vm-time-correlation="dry-run 0xffffe42722c64c41"

Note the options for 'perf inject' --vm-time-correlation are:

[ dry-run ] [ <TSC Offset> [ : <VMCS> [ , <VMCS> ]... ] ]...

So it is possible to specify different TSC Offsets for different VMCS.
The option "dry-run" will cause the file to be processed but without updating it.
Note it is also possible to get a intel_pt.log file by adding option --itrace=d

There were no errors so, do it for real

$ perf inject -i perf.data.kvm --vm-time-correlation=0xffffe42722c64c41 --force

'perf script' can be used to see if there are any decoder errors

$ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --itrace=e-o

There were none.

'perf script' can be used to provide an instruction trace showing timestamps

$ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
CPU 1/KVM 17006 [001] 11500.262866075: ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.05 (40/769)
:17006 17006 [001] 11500.262869216: ffffffff82200cb0 asm_sysvec_apic_timer_interrupt+0x0 ([guest.kernel.kallsyms]) clac
:17006 17006 [001] 11500.262869216: ffffffff82200cb3 asm_sysvec_apic_timer_interrupt+0x3 ([guest.kernel.kallsyms]) pushq $0xffffffffffffffff
:17006 17006 [001] 11500.262869216: ffffffff82200cb5 asm_sysvec_apic_timer_interrupt+0x5 ([guest.kernel.kallsyms]) callq 0xffffffff82201160
:17006 17006 [001] 11500.262869216: ffffffff82201160 error_entry+0x0 ([guest.kernel.kallsyms]) cld
:17006 17006 [001] 11500.262869216: ffffffff82201161 error_entry+0x1 ([guest.kernel.kallsyms]) pushq %rsi
:17006 17006 [001] 11500.262869216: ffffffff82201162 error_entry+0x2 ([guest.kernel.kallsyms]) movq 0x8(%rsp), %rsi
:17006 17006 [001] 11500.262869216: ffffffff82201167 error_entry+0x7 ([guest.kernel.kallsyms]) movq %rdi, 0x8(%rsp)
:17006 17006 [001] 11500.262869216: ffffffff8220116c error_entry+0xc ([guest.kernel.kallsyms]) pushq %rdx
:17006 17006 [001] 11500.262869216: ffffffff8220116d error_entry+0xd ([guest.kernel.kallsyms]) pushq %rcx
:17006 17006 [001] 11500.262869216: ffffffff8220116e error_entry+0xe ([guest.kernel.kallsyms]) pushq %rax



SEE ALSO
Expand Down
98 changes: 97 additions & 1 deletion tools/perf/util/intel-pt.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1018,6 +1018,19 @@ static bool intel_pt_have_tsc(struct intel_pt *pt)
return have_tsc;
}

static bool intel_pt_have_mtc(struct intel_pt *pt)
{
struct evsel *evsel;
u64 config;

evlist__for_each_entry(pt->session->evlist, evsel) {
if (intel_pt_get_config(pt, &evsel->core.attr, &config) &&
(config & pt->mtc_bit))
return true;
}
return false;
}

static bool intel_pt_sampling_mode(struct intel_pt *pt)
{
struct evsel *evsel;
Expand Down Expand Up @@ -1182,6 +1195,8 @@ static struct intel_pt_queue *intel_pt_alloc_queue(struct intel_pt *pt,
params.tsc_ctc_ratio_n = pt->tsc_ctc_ratio_n;
params.tsc_ctc_ratio_d = pt->tsc_ctc_ratio_d;
params.quick = pt->synth_opts.quick;
params.vm_time_correlation = pt->synth_opts.vm_time_correlation;
params.vm_tm_corr_dry_run = pt->synth_opts.vm_tm_corr_dry_run;
params.first_timestamp = pt->first_timestamp;

if (pt->filts.cnt > 0)
Expand Down Expand Up @@ -2465,7 +2480,7 @@ static int intel_pt_run_decoder(struct intel_pt_queue *ptq, u64 *timestamp)
if (pt->per_cpu_mmaps &&
(pt->have_sched_switch == 1 || pt->have_sched_switch == 3) &&
!pt->timeless_decoding && intel_pt_tracing_kernel(pt) &&
!pt->sampling_mode) {
!pt->sampling_mode && !pt->synth_opts.vm_time_correlation) {
pt->switch_ip = intel_pt_switch_ip(pt, &pt->ptss_ip);
if (pt->switch_ip) {
intel_pt_log("switch_ip: %"PRIx64" ptss_ip: %"PRIx64"\n",
Expand Down Expand Up @@ -3496,6 +3511,65 @@ static int intel_pt_setup_time_ranges(struct intel_pt *pt,
return 0;
}

static int intel_pt_parse_vm_tm_corr_arg(struct intel_pt *pt, char **args)
{
struct intel_pt_vmcs_info *vmcs_info;
u64 tsc_offset, vmcs;
char *p = *args;

errno = 0;

p = skip_spaces(p);
if (!*p)
return 1;

tsc_offset = strtoull(p, &p, 0);
if (errno)
return -errno;
p = skip_spaces(p);
if (*p != ':') {
pt->dflt_tsc_offset = tsc_offset;
*args = p;
return 0;
}
while (1) {
vmcs = strtoull(p, &p, 0);
if (errno)
return -errno;
if (!vmcs)
return -EINVAL;
vmcs_info = intel_pt_findnew_vmcs(&pt->vmcs_info, vmcs, tsc_offset);
if (!vmcs_info)
return -ENOMEM;
p = skip_spaces(p);
if (*p != ',')
break;
p += 1;
}
*args = p;
return 0;
}

static int intel_pt_parse_vm_tm_corr_args(struct intel_pt *pt)
{
char *args = pt->synth_opts.vm_tm_corr_args;
int ret;

if (!args)
return 0;

do {
ret = intel_pt_parse_vm_tm_corr_arg(pt, &args);
} while (!ret);

if (ret < 0) {
pr_err("Failed to parse VM Time Correlation options\n");
return ret;
}

return 0;
}

static const char * const intel_pt_info_fmts[] = {
[INTEL_PT_PMU_TYPE] = " PMU Type %"PRId64"\n",
[INTEL_PT_TIME_SHIFT] = " Time Shift %"PRIu64"\n",
Expand Down Expand Up @@ -3667,6 +3741,28 @@ int intel_pt_process_auxtrace_info(union perf_event *event,
pt->sampling_mode = intel_pt_sampling_mode(pt);
pt->est_tsc = !pt->timeless_decoding;

if (pt->synth_opts.vm_time_correlation) {
if (pt->timeless_decoding) {
pr_err("Intel PT has no time information for VM Time Correlation\n");
err = -EINVAL;
goto err_free_queues;
}
if (session->itrace_synth_opts->ptime_range) {
pr_err("Time ranges cannot be specified with VM Time Correlation\n");
err = -EINVAL;
goto err_free_queues;
}
/* Currently TSC Offset is calculated using MTC packets */
if (!intel_pt_have_mtc(pt)) {
pr_err("MTC packets must have been enabled for VM Time Correlation\n");
err = -EINVAL;
goto err_free_queues;
}
err = intel_pt_parse_vm_tm_corr_args(pt);
if (err)
goto err_free_queues;
}

pt->unknown_thread = thread__new(999999999, 999999999);
if (!pt->unknown_thread) {
err = -ENOMEM;
Expand Down

0 comments on commit e3ff42b

Please sign in to comment.