Skip to content

Commit

Permalink
x86/entry/32: Add PTI CR3 switches to NMI handler code
Browse files Browse the repository at this point in the history
The NMI handler is special, as it needs to leave with the same CR3 as it
was entered with. This is required because the NMI can happen within kernel
context but with user CR3 already loaded, i.e. after switching to user CR3
but before returning to user space.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Pavel Machek <pavel@ucw.cz>
Cc: "H . Peter Anvin" <hpa@zytor.com>
Cc: linux-mm@kvack.org
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Jiri Kosina <jkosina@suse.cz>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Waiman Long <llong@redhat.com>
Cc: "David H . Gutteridge" <dhgutteridge@sympatico.ca>
Cc: joro@8bytes.org
Link: https://lkml.kernel.org/r/1531906876-13451-14-git-send-email-joro@8bytes.org
  • Loading branch information
Joerg Roedel authored and Thomas Gleixner committed Jul 19, 2018
1 parent e464fb9 commit b65bef4
Showing 1 changed file with 33 additions and 6 deletions.
39 changes: 33 additions & 6 deletions arch/x86/entry/entry_32.S
Original file line number Diff line number Diff line change
Expand Up @@ -210,8 +210,19 @@

.endm

.macro SAVE_ALL_NMI
.macro SAVE_ALL_NMI cr3_reg:req
SAVE_ALL

/*
* Now switch the CR3 when PTI is enabled.
*
* We can enter with either user or kernel cr3, the code will
* store the old cr3 in \cr3_reg and switches to the kernel cr3
* if necessary.
*/
SWITCH_TO_KERNEL_CR3 scratch_reg=\cr3_reg

.Lend_\@:
.endm
/*
* This is a sneaky trick to help the unwinder find pt_regs on the stack. The
Expand Down Expand Up @@ -259,7 +270,23 @@
POP_GS_EX
.endm

.macro RESTORE_ALL_NMI pop=0
.macro RESTORE_ALL_NMI cr3_reg:req pop=0
/*
* Now switch the CR3 when PTI is enabled.
*
* We enter with kernel cr3 and switch the cr3 to the value
* stored on \cr3_reg, which is either a user or a kernel cr3.
*/
ALTERNATIVE "jmp .Lswitched_\@", "", X86_FEATURE_PTI

testl $PTI_SWITCH_MASK, \cr3_reg
jz .Lswitched_\@

/* User cr3 in \cr3_reg - write it to hardware cr3 */
movl \cr3_reg, %cr3

.Lswitched_\@:

RESTORE_REGS pop=\pop
.endm

Expand Down Expand Up @@ -1331,7 +1358,7 @@ ENTRY(nmi)
#endif

pushl %eax # pt_regs->orig_ax
SAVE_ALL_NMI
SAVE_ALL_NMI cr3_reg=%edi
ENCODE_FRAME_POINTER
xorl %edx, %edx # zero error code
movl %esp, %eax # pt_regs pointer
Expand Down Expand Up @@ -1359,7 +1386,7 @@ ENTRY(nmi)

.Lnmi_return:
CHECK_AND_APPLY_ESPFIX
RESTORE_ALL_NMI pop=4
RESTORE_ALL_NMI cr3_reg=%edi pop=4
jmp .Lirq_return

#ifdef CONFIG_X86_ESPFIX32
Expand All @@ -1375,12 +1402,12 @@ ENTRY(nmi)
pushl 16(%esp)
.endr
pushl %eax
SAVE_ALL_NMI
SAVE_ALL_NMI cr3_reg=%edi
ENCODE_FRAME_POINTER
FIXUP_ESPFIX_STACK # %eax == %esp
xorl %edx, %edx # zero error code
call do_nmi
RESTORE_ALL_NMI
RESTORE_ALL_NMI cr3_reg=%edi
lss 12+4(%esp), %esp # back to espfix stack
jmp .Lirq_return
#endif
Expand Down

0 comments on commit b65bef4

Please sign in to comment.