Skip to content

Commit

Permalink
docs: fault-injection: convert docs to ReST and rename to *.rst
Browse files Browse the repository at this point in the history
The conversion is actually:
  - add blank lines and identation in order to identify paragraphs;
  - fix tables markups;
  - add some lists markups;
  - mark literal blocks;
  - adjust title markups.

At its new index.rst, let's add a :orphan: while this is not linked to
the main index.rst file, in order to avoid build warnings.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Acked-by: Federico Vaga <federico.vaga@vaga.pv.it>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
  • Loading branch information
Mauro Carvalho Chehab authored and Jonathan Corbet committed Jun 14, 2019
1 parent f0ba437 commit 10ffebb
Show file tree
Hide file tree
Showing 14 changed files with 352 additions and 303 deletions.

Large diffs are not rendered by default.

20 changes: 20 additions & 0 deletions Documentation/fault-injection/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
:orphan:

===============
fault-injection
===============

.. toctree::
:maxdepth: 1

fault-injection
notifier-error-inject
nvme-fault-injection
provoke-crashes

.. only:: subproject and html

Indices
=======

* :ref:`genindex`
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,16 @@ modules that can be used to test the following notifiers.
PM notifier error injection module
----------------------------------
This feature is controlled through debugfs interface
/sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error

/sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error

Possible PM notifier events to be failed are:

* PM_HIBERNATION_PREPARE
* PM_SUSPEND_PREPARE
* PM_RESTORE_PREPARE

Example: Inject PM suspend error (-12 = -ENOMEM)
Example: Inject PM suspend error (-12 = -ENOMEM)::

# cd /sys/kernel/debug/notifier-error-inject/pm/
# echo -12 > actions/PM_SUSPEND_PREPARE/error
Expand All @@ -32,14 +33,15 @@ Example: Inject PM suspend error (-12 = -ENOMEM)
Memory hotplug notifier error injection module
----------------------------------------------
This feature is controlled through debugfs interface
/sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error

/sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error

Possible memory notifier events to be failed are:

* MEM_GOING_ONLINE
* MEM_GOING_OFFLINE

Example: Inject memory hotplug offline error (-12 == -ENOMEM)
Example: Inject memory hotplug offline error (-12 == -ENOMEM)::

# cd /sys/kernel/debug/notifier-error-inject/memory
# echo -12 > actions/MEM_GOING_OFFLINE/error
Expand All @@ -49,7 +51,8 @@ Example: Inject memory hotplug offline error (-12 == -ENOMEM)
powerpc pSeries reconfig notifier error injection module
--------------------------------------------------------
This feature is controlled through debugfs interface
/sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error

/sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error

Possible pSeries reconfig notifier events to be failed are:

Expand All @@ -61,7 +64,8 @@ Possible pSeries reconfig notifier events to be failed are:
Netdevice notifier error injection module
----------------------------------------------
This feature is controlled through debugfs interface
/sys/kernel/debug/notifier-error-inject/netdev/actions/<notifier event>/error

/sys/kernel/debug/notifier-error-inject/netdev/actions/<notifier event>/error

Netdevice notifier events which can be failed are:

Expand All @@ -75,7 +79,7 @@ Netdevice notifier events which can be failed are:
* NETDEV_PRECHANGEUPPER
* NETDEV_CHANGEUPPER

Example: Inject netdevice mtu change error (-22 == -EINVAL)
Example: Inject netdevice mtu change error (-22 == -EINVAL)::

# cd /sys/kernel/debug/notifier-error-inject/netdev
# echo -22 > actions/NETDEV_CHANGEMTU/error
Expand Down
120 changes: 120 additions & 0 deletions Documentation/fault-injection/nvme-fault-injection.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
NVMe Fault Injection
====================
Linux's fault injection framework provides a systematic way to support
error injection via debugfs in the /sys/kernel/debug directory. When
enabled, the default NVME_SC_INVALID_OPCODE with no retry will be
injected into the nvme_end_request. Users can change the default status
code and no retry flag via the debugfs. The list of Generic Command
Status can be found in include/linux/nvme.h

Following examples show how to inject an error into the nvme.

First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config,
recompile the kernel. After booting up the kernel, do the
following.

Example 1: Inject default status code with no retry
---------------------------------------------------

::

mount /dev/nvme0n1 /mnt
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times
echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability
cp a.file /mnt

Expected Result::

cp: cannot stat ‘/mnt/a.file’: Input/output error

Message from dmesg::

FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0, times 1
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc8+ #2
Hardware name: innotek GmbH VirtualBox/VirtualBox,
BIOS VirtualBox 12/01/2006
Call Trace:
<IRQ>
dump_stack+0x5c/0x7d
should_fail+0x148/0x170
nvme_should_fail+0x2f/0x50 [nvme_core]
nvme_process_cq+0xe7/0x1d0 [nvme]
nvme_irq+0x1e/0x40 [nvme]
__handle_irq_event_percpu+0x3a/0x190
handle_irq_event_percpu+0x30/0x70
handle_irq_event+0x36/0x60
handle_fasteoi_irq+0x78/0x120
handle_irq+0xa7/0x130
? tick_irq_enter+0xa8/0xc0
do_IRQ+0x43/0xc0
common_interrupt+0xa2/0xa2
</IRQ>
RIP: 0010:native_safe_halt+0x2/0x10
RSP: 0018:ffffffff82003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd
RAX: ffffffff817a10c0 RBX: ffffffff82012480 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 000000008e38ce64 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82012480
R13: ffffffff82012480 R14: 0000000000000000 R15: 0000000000000000
? __sched_text_end+0x4/0x4
default_idle+0x18/0xf0
do_idle+0x150/0x1d0
cpu_startup_entry+0x6f/0x80
start_kernel+0x4c4/0x4e4
? set_init_arg+0x55/0x55
secondary_startup_64+0xa5/0xb0
print_req_error: I/O error, dev nvme0n1, sector 9240
EXT4-fs error (device nvme0n1): ext4_find_entry:1436:
inode #2: comm cp: reading directory lblock 0

Example 2: Inject default status code with retry
------------------------------------------------

::

mount /dev/nvme0n1 /mnt
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times
echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status
echo 0 > /sys/kernel/debug/nvme0n1/fault_inject/dont_retry

cp a.file /mnt

Expected Result::

command success without error

Message from dmesg::

FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0, times 1
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc8+ #4
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Call Trace:
<IRQ>
dump_stack+0x5c/0x7d
should_fail+0x148/0x170
nvme_should_fail+0x30/0x60 [nvme_core]
nvme_loop_queue_response+0x84/0x110 [nvme_loop]
nvmet_req_complete+0x11/0x40 [nvmet]
nvmet_bio_done+0x28/0x40 [nvmet]
blk_update_request+0xb0/0x310
blk_mq_end_request+0x18/0x60
flush_smp_call_function_queue+0x3d/0xf0
smp_call_function_single_interrupt+0x2c/0xc0
call_function_single_interrupt+0xa2/0xb0
</IRQ>
RIP: 0010:native_safe_halt+0x2/0x10
RSP: 0018:ffffc9000068bec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
RAX: ffffffff817a10c0 RBX: ffff88011a3c9680 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000001 R08: 000000008e38c131 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88011a3c9680
R13: ffff88011a3c9680 R14: 0000000000000000 R15: 0000000000000000
? __sched_text_end+0x4/0x4
default_idle+0x18/0xf0
do_idle+0x150/0x1d0
cpu_startup_entry+0x6f/0x80
start_secondary+0x187/0x1e0
secondary_startup_64+0xa5/0xb0
116 changes: 0 additions & 116 deletions Documentation/fault-injection/nvme-fault-injection.txt

This file was deleted.

48 changes: 48 additions & 0 deletions Documentation/fault-injection/provoke-crashes.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
===============
Provoke crashes
===============

The lkdtm module provides an interface to crash or injure the kernel at
predefined crashpoints to evaluate the reliability of crash dumps obtained
using different dumping solutions. The module uses KPROBEs to instrument
crashing points, but can also crash the kernel directly without KRPOBE
support.


You can provide the way either through module arguments when inserting
the module, or through a debugfs interface.

Usage::

insmod lkdtm.ko [recur_count={>0}] cpoint_name=<> cpoint_type=<>
[cpoint_count={>0}]

recur_count
Recursion level for the stack overflow test. Default is 10.

cpoint_name
Crash point where the kernel is to be crashed. It can be
one of INT_HARDWARE_ENTRY, INT_HW_IRQ_EN, INT_TASKLET_ENTRY,
FS_DEVRW, MEM_SWAPOUT, TIMERADD, SCSI_DISPATCH_CMD,
IDE_CORE_CP, DIRECT

cpoint_type
Indicates the action to be taken on hitting the crash point.
It can be one of PANIC, BUG, EXCEPTION, LOOP, OVERFLOW,
CORRUPT_STACK, UNALIGNED_LOAD_STORE_WRITE, OVERWRITE_ALLOCATION,
WRITE_AFTER_FREE,

cpoint_count
Indicates the number of times the crash point is to be hit
to trigger an action. The default is 10.

You can also induce failures by mounting debugfs and writing the type to
<mountpoint>/provoke-crash/<crashpoint>. E.g.::

mount -t debugfs debugfs /mnt
echo EXCEPTION > /mnt/provoke-crash/INT_HARDWARE_ENTRY


A special file is `DIRECT` which will induce the crash directly without
KPROBE instrumentation. This mode is the only one available when the module
is built on a kernel without KPROBEs support.
Loading

0 comments on commit 10ffebb

Please sign in to comment.