From 32337c0a28242f725c2c499c15100d67a4133050 Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Sat, 26 Aug 2023 13:08:43 -0700 Subject: [PATCH 01/15] bpf: Prevent inlining of bpf_fentry_test7() With latest clang18, I hit test_progs failures for the following test: #13/2 bpf_cookie/multi_kprobe_link_api:FAIL #13/3 bpf_cookie/multi_kprobe_attach_api:FAIL #13 bpf_cookie:FAIL #75 fentry_fexit:FAIL #76/1 fentry_test/fentry:FAIL #76 fentry_test:FAIL #80/1 fexit_test/fexit:FAIL #80 fexit_test:FAIL #110/1 kprobe_multi_test/skel_api:FAIL #110/2 kprobe_multi_test/link_api_addrs:FAIL #110/3 kprobe_multi_test/link_api_syms:FAIL #110/4 kprobe_multi_test/attach_api_pattern:FAIL #110/5 kprobe_multi_test/attach_api_addrs:FAIL #110/6 kprobe_multi_test/attach_api_syms:FAIL #110 kprobe_multi_test:FAIL For example, for #13/2, the error messages are: [...] kprobe_multi_test_run:FAIL:kprobe_test7_result unexpected kprobe_test7_result: actual 0 != expected 1 [...] kprobe_multi_test_run:FAIL:kretprobe_test7_result unexpected kretprobe_test7_result: actual 0 != expected 1 clang17 does not have this issue. Further investigation shows that kernel func bpf_fentry_test7(), used in the above tests, is inlined by the compiler although it is marked as noinline. int noinline bpf_fentry_test7(struct bpf_fentry_test_t *arg) { return (long)arg; } It is known that for simple functions like the above (e.g. just returning a constant or an input argument), the clang compiler may still do inlining for a noinline function. Adding 'asm volatile ("")' in the beginning of the bpf_fentry_test7() can prevent inlining. Signed-off-by: Yonghong Song Signed-off-by: Daniel Borkmann Tested-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20230826200843.2210074-1-yonghong.song@linux.dev --- net/bpf/test_run.c | 1 + 1 file changed, 1 insertion(+) diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 57a7a64b84ede..0841f8d824198 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -543,6 +543,7 @@ struct bpf_fentry_test_t { int noinline bpf_fentry_test7(struct bpf_fentry_test_t *arg) { + asm volatile (""); return (long)arg; } From 6a8faf10709161e7138202a8cf052b070971239f Mon Sep 17 00:00:00 2001 From: Yafang Shao Date: Wed, 30 Aug 2023 03:03:25 +0000 Subject: [PATCH 02/15] bpftool: Fix build warnings with -Wtype-limits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Quentin reported build warnings when building bpftool : link.c: In function ‘perf_config_hw_cache_str’: link.c:86:18: warning: comparison of unsigned expression in ‘>= 0’ is always true [-Wtype-limits] 86 | if ((id) >= 0 && (id) < ARRAY_SIZE(array)) \ | ^~ link.c:320:20: note: in expansion of macro ‘perf_event_name’ 320 | hw_cache = perf_event_name(evsel__hw_cache, config & 0xff); | ^~~~~~~~~~~~~~~ [... more of the same for the other calls to perf_event_name ...] He also pointed out the reason and the solution: We're always passing unsigned, so it should be safe to drop the check on (id) >= 0. Fixes: 62b57e3ddd64 ("bpftool: Add perf event names") Reported-by: Quentin Monnet Suggested-by: Quentin Monnet Signed-off-by: Yafang Shao Signed-off-by: Daniel Borkmann Acked-by: Quentin Monnet Closes: https://lore.kernel.org/bpf/a35d9a2d-54a0-49ec-9ed1-8fcf1369d3cc@isovalent.com Link: https://lore.kernel.org/bpf/20230830030325.3786-1-laoar.shao@gmail.com --- tools/bpf/bpftool/link.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/bpf/bpftool/link.c b/tools/bpf/bpftool/link.c index 0b214f6ab5c83..2e5c231e08ac7 100644 --- a/tools/bpf/bpftool/link.c +++ b/tools/bpf/bpftool/link.c @@ -83,7 +83,7 @@ const char *evsel__hw_cache_result[PERF_COUNT_HW_CACHE_RESULT_MAX] = { #define perf_event_name(array, id) ({ \ const char *event_str = NULL; \ \ - if ((id) >= 0 && (id) < ARRAY_SIZE(array)) \ + if ((id) < ARRAY_SIZE(array)) \ event_str = array[id]; \ event_str; \ }) From 9d0a67b9d42c630d5013ef81587335d975a7a4a9 Mon Sep 17 00:00:00 2001 From: Tirthendu Sarkar Date: Wed, 23 Aug 2023 20:17:13 +0530 Subject: [PATCH 03/15] xsk: Fix xsk_build_skb() error: 'skb' dereferencing possible ERR_PTR() Currently, xsk_build_skb() is a function that builds skb in two possible ways and then is ended with common error handling. We can distinguish four possible error paths and handling in xsk_build_skb(): 1. sock_alloc_send_skb fails: Retry (skb is NULL). 2. skb_store_bits fails : Free skb and retry. 3. MAX_SKB_FRAGS exceeded: Free skb, cleanup and drop packet. 4. alloc_page fails for frag: Retry page allocation w/o freeing skb 1] and 3] can happen in xsk_build_skb_zerocopy(), which is one of the two code paths responsible for building skb. Common error path in xsk_build_skb() assumes that in case errno != -EAGAIN, skb is a valid pointer, which is wrong as kernel test robot reports that in xsk_build_skb_zerocopy() other errno values are returned for skb being NULL. To fix this, set -EOVERFLOW as error when MAX_SKB_FRAGS are exceeded and packet needs to be dropped in both xsk_build_skb() and xsk_build_skb_zerocopy() and use this to distinguish against all other error cases. Also, add explicit kfree_skb() for 3] so that handling of 1], 2], and 3] becomes identical where allocation needs to be retried. Fixes: cf24f5a5feea ("xsk: add support for AF_XDP multi-buffer on Tx path") Reported-by: kernel test robot Reported-by: Dan Carpenter Signed-off-by: Tirthendu Sarkar Signed-off-by: Daniel Borkmann Acked-by: Magnus Karlsson Closes: https://lore.kernel.org/r/202307210434.OjgqFcbB-lkp@intel.com Link: https://lore.kernel.org/bpf/20230823144713.2231808-1-tirthendu.sarkar@intel.com --- net/xdp/xsk.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index fcfc8472f73da..55f8b9b0e06d1 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -602,7 +602,7 @@ static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs, for (copied = 0, i = skb_shinfo(skb)->nr_frags; copied < len; i++) { if (unlikely(i >= MAX_SKB_FRAGS)) - return ERR_PTR(-EFAULT); + return ERR_PTR(-EOVERFLOW); page = pool->umem->pgs[addr >> PAGE_SHIFT]; get_page(page); @@ -655,15 +655,17 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, skb_put(skb, len); err = skb_store_bits(skb, 0, buffer, len); - if (unlikely(err)) + if (unlikely(err)) { + kfree_skb(skb); goto free_err; + } } else { int nr_frags = skb_shinfo(skb)->nr_frags; struct page *page; u8 *vaddr; if (unlikely(nr_frags == (MAX_SKB_FRAGS - 1) && xp_mb_desc(desc))) { - err = -EFAULT; + err = -EOVERFLOW; goto free_err; } @@ -690,12 +692,14 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, return skb; free_err: - if (err == -EAGAIN) { - xsk_cq_cancel_locked(xs, 1); - } else { - xsk_set_destructor_arg(skb); - xsk_drop_skb(skb); + if (err == -EOVERFLOW) { + /* Drop the packet */ + xsk_set_destructor_arg(xs->skb); + xsk_drop_skb(xs->skb); xskq_cons_release(xs->tx); + } else { + /* Let application retry */ + xsk_cq_cancel_locked(xs, 1); } return ERR_PTR(err); @@ -738,7 +742,7 @@ static int __xsk_generic_xmit(struct sock *sk) skb = xsk_build_skb(xs, &desc); if (IS_ERR(skb)) { err = PTR_ERR(skb); - if (err == -EAGAIN) + if (err != -EOVERFLOW) goto out; err = 0; continue; From 5439cfa7fe612e7d02d5a1234feda3fa6e483ba7 Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Sun, 27 Aug 2023 08:05:51 -0700 Subject: [PATCH 04/15] selftests/bpf: Fix flaky cgroup_iter_sleepable subtest Occasionally, with './test_progs -j' on my vm, I will hit the following failure: test_cgrp_local_storage:PASS:join_cgroup /cgrp_local_storage 0 nsec test_cgroup_iter_sleepable:PASS:skel_open 0 nsec test_cgroup_iter_sleepable:PASS:skel_load 0 nsec test_cgroup_iter_sleepable:PASS:attach_iter 0 nsec test_cgroup_iter_sleepable:PASS:iter_create 0 nsec test_cgroup_iter_sleepable:FAIL:cgroup_id unexpected cgroup_id: actual 1 != expected 2812 #48/5 cgrp_local_storage/cgroup_iter_sleepable:FAIL #48 cgrp_local_storage:FAIL Finally, I decided to do some investigation since the test is introduced by myself. It turns out the reason is due to cgroup_fd with value 0. In cgroup_iter, a cgroup_fd of value 0 means the root cgroup. /* from cgroup_iter.c */ if (fd) cgrp = cgroup_v1v2_get_from_fd(fd); else if (id) cgrp = cgroup_get_from_id(id); else /* walk the entire hierarchy by default. */ cgrp = cgroup_get_from_path("/"); That is why we got cgroup_id 1 instead of expected 2812. Why we got a cgroup_fd 0? Nobody should really touch 'stdin' (fd 0) in test_progs. I traced 'close' syscall with stack trace and found the root cause, which is a bug in bpf_obj_pinning.c. Basically, the code closed fd 0 although it should not. Fixing the bug in bpf_obj_pinning.c also resolved the above cgroup_iter_sleepable subtest failure. Fixes: 3b22f98e5a05 ("selftests/bpf: Add path_fd-based BPF_OBJ_PIN and BPF_OBJ_GET tests") Signed-off-by: Yonghong Song Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230827150551.1743497-1-yonghong.song@linux.dev --- tools/testing/selftests/bpf/prog_tests/bpf_obj_pinning.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_obj_pinning.c b/tools/testing/selftests/bpf/prog_tests/bpf_obj_pinning.c index 31f1e815f6719..ee0458a5ce789 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_obj_pinning.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_obj_pinning.c @@ -8,6 +8,7 @@ #include #include #include +#include "bpf/libbpf_internal.h" static inline int sys_fsopen(const char *fsname, unsigned flags) { @@ -155,7 +156,7 @@ static void validate_pin(int map_fd, const char *map_name, int src_value, ASSERT_OK(err, "obj_pin"); /* cleanup */ - if (pin_opts.path_fd >= 0) + if (path_kind == PATH_FD_REL && pin_opts.path_fd >= 0) close(pin_opts.path_fd); if (old_cwd[0]) ASSERT_OK(chdir(old_cwd), "restore_cwd"); @@ -220,7 +221,7 @@ static void validate_get(int map_fd, const char *map_name, int src_value, goto cleanup; /* cleanup */ - if (get_opts.path_fd >= 0) + if (path_kind == PATH_FD_REL && get_opts.path_fd >= 0) close(get_opts.path_fd); if (old_cwd[0]) ASSERT_OK(chdir(old_cwd), "restore_cwd"); From 2d71a90f7e0fa3cd348602a36f6eb1237ab7cebb Mon Sep 17 00:00:00 2001 From: Will Hawkins Date: Sat, 26 Aug 2023 01:32:54 -0400 Subject: [PATCH 05/15] bpf, docs: Correct source of offset for program-local call The offset to use when calculating the target of a program-local call is in the instruction's imm field, not its offset field. Signed-off-by: Will Hawkins Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Acked-by: David Vernet Link: https://lore.kernel.org/bpf/20230826053258.1860167-1-hawkinsw@obs.cr --- Documentation/bpf/standardization/instruction-set.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/bpf/standardization/instruction-set.rst b/Documentation/bpf/standardization/instruction-set.rst index 4f73e9dc8d9ee..c5b0b2011f162 100644 --- a/Documentation/bpf/standardization/instruction-set.rst +++ b/Documentation/bpf/standardization/instruction-set.rst @@ -373,7 +373,7 @@ BPF_JNE 0x5 any PC += offset if dst != src BPF_JSGT 0x6 any PC += offset if dst > src signed BPF_JSGE 0x7 any PC += offset if dst >= src signed BPF_CALL 0x8 0x0 call helper function by address see `Helper functions`_ -BPF_CALL 0x8 0x1 call PC += offset see `Program-local functions`_ +BPF_CALL 0x8 0x1 call PC += imm see `Program-local functions`_ BPF_CALL 0x8 0x2 call helper function by BTF ID see `Helper functions`_ BPF_EXIT 0x9 0x0 return BPF_JMP only BPF_JLT 0xa any PC += offset if dst < src unsigned @@ -424,8 +424,8 @@ Program-local functions ~~~~~~~~~~~~~~~~~~~~~~~ Program-local functions are functions exposed by the same BPF program as the caller, and are referenced by offset from the call instruction, similar to -``BPF_JA``. A ``BPF_EXIT`` within the program-local function will return to -the caller. +``BPF_JA``. The offset is encoded in the imm field of the call instruction. +A ``BPF_EXIT`` within the program-local function will return to the caller. Load and store instructions =========================== From be4033d36070e44fba766a21ef2d0c24fa04c377 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Sun, 27 Aug 2023 01:29:12 +0300 Subject: [PATCH 06/15] docs/bpf: Add description for CO-RE relocations Add a section on CO-RE relocations to llvm_relo.rst. Describe relevant .BTF.ext structure, `enum bpf_core_relo_kind` and `struct bpf_core_relo` in some detail. Description is based on doc-strings from: - include/uapi/linux/bpf.h:struct bpf_core_relo - tools/lib/bpf/relo_core.c:__bpf_core_types_match() Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Yonghong Song Link: https://lore.kernel.org/bpf/20230826222912.2560865-2-eddyz87@gmail.com --- Documentation/bpf/btf.rst | 31 +++- Documentation/bpf/llvm_reloc.rst | 304 +++++++++++++++++++++++++++++++ 2 files changed, 329 insertions(+), 6 deletions(-) diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst index f32db1f44ae90..ffc11afee5691 100644 --- a/Documentation/bpf/btf.rst +++ b/Documentation/bpf/btf.rst @@ -726,8 +726,8 @@ same as the one describe in :ref:`BTF_Type_String`. 4.2 .BTF.ext section -------------------- -The .BTF.ext section encodes func_info and line_info which needs loader -manipulation before loading into the kernel. +The .BTF.ext section encodes func_info, line_info and CO-RE relocations +which needs loader manipulation before loading into the kernel. The specification for .BTF.ext section is defined at ``tools/lib/bpf/btf.h`` and ``tools/lib/bpf/btf.c``. @@ -745,15 +745,20 @@ The current header of .BTF.ext section:: __u32 func_info_len; __u32 line_info_off; __u32 line_info_len; + + /* optional part of .BTF.ext header */ + __u32 core_relo_off; + __u32 core_relo_len; }; It is very similar to .BTF section. Instead of type/string section, it -contains func_info and line_info section. See :ref:`BPF_Prog_Load` for details -about func_info and line_info record format. +contains func_info, line_info and core_relo sub-sections. +See :ref:`BPF_Prog_Load` for details about func_info and line_info +record format. The func_info is organized as below.:: - func_info_rec_size + func_info_rec_size /* __u32 value */ btf_ext_info_sec for section #1 /* func_info for section #1 */ btf_ext_info_sec for section #2 /* func_info for section #2 */ ... @@ -773,7 +778,7 @@ Here, num_info must be greater than 0. The line_info is organized as below.:: - line_info_rec_size + line_info_rec_size /* __u32 value */ btf_ext_info_sec for section #1 /* line_info for section #1 */ btf_ext_info_sec for section #2 /* line_info for section #2 */ ... @@ -787,6 +792,20 @@ kernel API, the ``insn_off`` is the instruction offset in the unit of ``struct bpf_insn``. For ELF API, the ``insn_off`` is the byte offset from the beginning of section (``btf_ext_info_sec->sec_name_off``). +The core_relo is organized as below.:: + + core_relo_rec_size /* __u32 value */ + btf_ext_info_sec for section #1 /* core_relo for section #1 */ + btf_ext_info_sec for section #2 /* core_relo for section #2 */ + +``core_relo_rec_size`` specifies the size of ``bpf_core_relo`` +structure when .BTF.ext is generated. All ``bpf_core_relo`` structures +within a single ``btf_ext_info_sec`` describe relocations applied to +section named by ``btf_ext_info_sec->sec_name_off``. + +See :ref:`Documentation/bpf/llvm_reloc ` +for more information on CO-RE relocations. + 4.2 .BTF_ids section -------------------- diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst index 450e6403fe3de..73bf805000f23 100644 --- a/Documentation/bpf/llvm_reloc.rst +++ b/Documentation/bpf/llvm_reloc.rst @@ -240,3 +240,307 @@ The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations:: Offset Info Type Symbol's Value Symbol's Name 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text + +.. _btf-co-re-relocations: + +================= +CO-RE Relocations +================= + +From object file point of view CO-RE mechanism is implemented as a set +of CO-RE specific relocation records. These relocation records are not +related to ELF relocations and are encoded in .BTF.ext section. +See :ref:`Documentation/bpf/btf ` for more +information on .BTF.ext structure. + +CO-RE relocations are applied to BPF instructions to update immediate +or offset fields of the instruction at load time with information +relevant for target kernel. + +Field to patch is selected basing on the instruction class: + +* For BPF_ALU, BPF_ALU64, BPF_LD `immediate` field is patched; +* For BPF_LDX, BPF_STX, BPF_ST `offset` field is patched; +* BPF_JMP, BPF_JMP32 instructions **should not** be patched. + +Relocation kinds +================ + +There are several kinds of CO-RE relocations that could be split in +three groups: + +* Field-based - patch instruction with field related information, e.g. + change offset field of the BPF_LDX instruction to reflect offset + of a specific structure field in the target kernel. + +* Type-based - patch instruction with type related information, e.g. + change immediate field of the BPF_ALU move instruction to 0 or 1 to + reflect if specific type is present in the target kernel. + +* Enum-based - patch instruction with enum related information, e.g. + change immediate field of the BPF_LD_IMM64 instruction to reflect + value of a specific enum literal in the target kernel. + +The complete list of relocation kinds is represented by the following enum: + +.. code-block:: c + + enum bpf_core_relo_kind { + BPF_CORE_FIELD_BYTE_OFFSET = 0, /* field byte offset */ + BPF_CORE_FIELD_BYTE_SIZE = 1, /* field size in bytes */ + BPF_CORE_FIELD_EXISTS = 2, /* field existence in target kernel */ + BPF_CORE_FIELD_SIGNED = 3, /* field signedness (0 - unsigned, 1 - signed) */ + BPF_CORE_FIELD_LSHIFT_U64 = 4, /* bitfield-specific left bitshift */ + BPF_CORE_FIELD_RSHIFT_U64 = 5, /* bitfield-specific right bitshift */ + BPF_CORE_TYPE_ID_LOCAL = 6, /* type ID in local BPF object */ + BPF_CORE_TYPE_ID_TARGET = 7, /* type ID in target kernel */ + BPF_CORE_TYPE_EXISTS = 8, /* type existence in target kernel */ + BPF_CORE_TYPE_SIZE = 9, /* type size in bytes */ + BPF_CORE_ENUMVAL_EXISTS = 10, /* enum value existence in target kernel */ + BPF_CORE_ENUMVAL_VALUE = 11, /* enum value integer value */ + BPF_CORE_TYPE_MATCHES = 12, /* type match in target kernel */ + }; + +Notes: + +* ``BPF_CORE_FIELD_LSHIFT_U64`` and ``BPF_CORE_FIELD_RSHIFT_U64`` are + supposed to be used to read bitfield values using the following + algorithm: + + .. code-block:: c + + // To read bitfield ``f`` from ``struct s`` + is_signed = relo(s->f, BPF_CORE_FIELD_SIGNED) + off = relo(s->f, BPF_CORE_FIELD_BYTE_OFFSET) + sz = relo(s->f, BPF_CORE_FIELD_BYTE_SIZE) + l = relo(s->f, BPF_CORE_FIELD_LSHIFT_U64) + r = relo(s->f, BPF_CORE_FIELD_RSHIFT_U64) + // define ``v`` as signed or unsigned integer of size ``sz`` + v = *({s|u} *)((void *)s + off) + v <<= l + v >>= r + +* The ``BPF_CORE_TYPE_MATCHES`` queries matching relation, defined as + follows: + + * for integers: types match if size and signedness match; + * for arrays & pointers: target types are recursively matched; + * for structs & unions: + + * local members need to exist in target with the same name; + + * for each member we recursively check match unless it is already behind a + pointer, in which case we only check matching names and compatible kind; + + * for enums: + + * local variants have to have a match in target by symbolic name (but not + numeric value); + + * size has to match (but enum may match enum64 and vice versa); + + * for function pointers: + + * number and position of arguments in local type has to match target; + * for each argument and the return value we recursively check match. + +CO-RE Relocation Record +======================= + +Relocation record is encoded as the following structure: + +.. code-block:: c + + struct bpf_core_relo { + __u32 insn_off; + __u32 type_id; + __u32 access_str_off; + enum bpf_core_relo_kind kind; + }; + +* ``insn_off`` - instruction offset (in bytes) within a code section + associated with this relocation; + +* ``type_id`` - BTF type ID of the "root" (containing) entity of a + relocatable type or field; + +* ``access_str_off`` - offset into corresponding .BTF string section. + String interpretation depends on specific relocation kind: + + * for field-based relocations, string encodes an accessed field using + a sequence of field and array indices, separated by colon (:). It's + conceptually very close to LLVM's `getelementptr `_ instruction's + arguments for identifying offset to a field. For example, consider the + following C code: + + .. code-block:: c + + struct sample { + int a; + int b; + struct { int c[10]; }; + } __attribute__((preserve_access_index)); + struct sample *s; + + * Access to ``s[0].a`` would be encoded as ``0:0``: + + * ``0``: first element of ``s`` (as if ``s`` is an array); + * ``0``: index of field ``a`` in ``struct sample``. + + * Access to ``s->a`` would be encoded as ``0:0`` as well. + * Access to ``s->b`` would be encoded as ``0:1``: + + * ``0``: first element of ``s``; + * ``1``: index of field ``b`` in ``struct sample``. + + * Access to ``s[1].c[5]`` would be encoded as ``1:2:0:5``: + + * ``1``: second element of ``s``; + * ``2``: index of anonymous structure field in ``struct sample``; + * ``0``: index of field ``c`` in anonymous structure; + * ``5``: access to array element #5. + + * for type-based relocations, string is expected to be just "0"; + + * for enum value-based relocations, string contains an index of enum + value within its enum type; + +* ``kind`` - one of ``enum bpf_core_relo_kind``. + +.. _GEP: https://llvm.org/docs/LangRef.html#getelementptr-instruction + +.. _btf_co_re_relocation_examples: + +CO-RE Relocation Examples +========================= + +For the following C code: + +.. code-block:: c + + struct foo { + int a; + int b; + unsigned c:15; + } __attribute__((preserve_access_index)); + + enum bar { U, V }; + +With the following BTF definitions: + +.. code-block:: + + ... + [2] STRUCT 'foo' size=8 vlen=2 + 'a' type_id=3 bits_offset=0 + 'b' type_id=3 bits_offset=32 + 'c' type_id=4 bits_offset=64 bitfield_size=15 + [3] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED + [4] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none) + ... + [16] ENUM 'bar' encoding=UNSIGNED size=4 vlen=2 + 'U' val=0 + 'V' val=1 + +Field offset relocations are generated automatically when +``__attribute__((preserve_access_index))`` is used, for example: + +.. code-block:: c + + void alpha(struct foo *s, volatile unsigned long *g) { + *g = s->a; + s->a = 1; + } + + 00 : + 0: r3 = *(s32 *)(r1 + 0x0) + 00: CO-RE [2] struct foo::a (0:0) + 1: *(u64 *)(r2 + 0x0) = r3 + 2: *(u32 *)(r1 + 0x0) = 0x1 + 10: CO-RE [2] struct foo::a (0:0) + 3: exit + + +All relocation kinds could be requested via built-in functions. +E.g. field-based relocations: + +.. code-block:: c + + void bravo(struct foo *s, volatile unsigned long *g) { + *g = __builtin_preserve_field_info(s->b, 0 /* field byte offset */); + *g = __builtin_preserve_field_info(s->b, 1 /* field byte size */); + *g = __builtin_preserve_field_info(s->b, 2 /* field existence */); + *g = __builtin_preserve_field_info(s->b, 3 /* field signedness */); + *g = __builtin_preserve_field_info(s->c, 4 /* bitfield left shift */); + *g = __builtin_preserve_field_info(s->c, 5 /* bitfield right shift */); + } + + 20 : + 4: r1 = 0x4 + 20: CO-RE [2] struct foo::b (0:1) + 5: *(u64 *)(r2 + 0x0) = r1 + 6: r1 = 0x4 + 30: CO-RE [2] struct foo::b (0:1) + 7: *(u64 *)(r2 + 0x0) = r1 + 8: r1 = 0x1 + 40: CO-RE [2] struct foo::b (0:1) + 9: *(u64 *)(r2 + 0x0) = r1 + 10: r1 = 0x1 + 50: CO-RE [2] struct foo::b (0:1) + 11: *(u64 *)(r2 + 0x0) = r1 + 12: r1 = 0x31 + 60: CO-RE [2] struct foo::c (0:2) + 13: *(u64 *)(r2 + 0x0) = r1 + 14: r1 = 0x31 + 70: CO-RE [2] struct foo::c (0:2) + 15: *(u64 *)(r2 + 0x0) = r1 + 16: exit + + +Type-based relocations: + +.. code-block:: c + + void charlie(struct foo *s, volatile unsigned long *g) { + *g = __builtin_preserve_type_info(*s, 0 /* type existence */); + *g = __builtin_preserve_type_info(*s, 1 /* type size */); + *g = __builtin_preserve_type_info(*s, 2 /* type matches */); + *g = __builtin_btf_type_id(*s, 0 /* type id in this object file */); + *g = __builtin_btf_type_id(*s, 1 /* type id in target kernel */); + } + + 88 : + 17: r1 = 0x1 + 88: CO-RE [2] struct foo + 18: *(u64 *)(r2 + 0x0) = r1 + 19: r1 = 0xc + 98: CO-RE [2] struct foo + 20: *(u64 *)(r2 + 0x0) = r1 + 21: r1 = 0x1 + a8: CO-RE [2] struct foo + 22: *(u64 *)(r2 + 0x0) = r1 + 23: r1 = 0x2 ll + b8: CO-RE [2] struct foo + 25: *(u64 *)(r2 + 0x0) = r1 + 26: r1 = 0x2 ll + d0: CO-RE [2] struct foo + 28: *(u64 *)(r2 + 0x0) = r1 + 29: exit + +Enum-based relocations: + +.. code-block:: c + + void delta(struct foo *s, volatile unsigned long *g) { + *g = __builtin_preserve_enum_value(*(enum bar *)U, 0 /* enum literal existence */); + *g = __builtin_preserve_enum_value(*(enum bar *)V, 1 /* enum literal value */); + } + + f0 : + 30: r1 = 0x1 ll + f0: CO-RE [16] enum bar::U = 0 + 32: *(u64 *)(r2 + 0x0) = r1 + 33: r1 = 0x1 ll + 108: CO-RE [16] enum bar::V = 1 + 35: *(u64 *)(r2 + 0x0) = r1 + 36: exit From 35d2b7ffffc1d9b3dc6c761010aa3338da49165b Mon Sep 17 00:00:00 2001 From: John Fastabend Date: Tue, 29 Aug 2023 22:35:17 -0700 Subject: [PATCH 07/15] bpf, sockmap: Fix preempt_rt splat when using raw_spin_lock_t Sockmap and sockhash maps are a collection of psocks that are objects representing a socket plus a set of metadata needed to manage the BPF programs associated with the socket. These maps use the stab->lock to protect from concurrent operations on the maps, e.g. trying to insert to objects into the array at the same time in the same slot. Additionally, a sockhash map has a bucket lock to protect iteration and insert/delete into the hash entry. Each psock has a psock->link which is a linked list of all the maps that a psock is attached to. This allows a psock (socket) to be included in multiple sockmap and sockhash maps. This linked list is protected the psock->link_lock. They _must_ be nested correctly to avoid deadlock: lock(stab->lock) : do BPF map operations and psock insert/delete lock(psock->link_lock) : add map to psock linked list of maps unlock(psock->link_lock) unlock(stab->lock) For non PREEMPT_RT kernels both raw_spin_lock_t and spin_lock_t are guaranteed to not sleep. But, with PREEMPT_RT kernels the spin_lock_t variants may sleep. In the current code we have many patterns like this: rcu_critical_section: raw_spin_lock(stab->lock) spin_lock(psock->link_lock) <- may sleep ouch spin_unlock(psock->link_lock) raw_spin_unlock(stab->lock) rcu_critical_section Nesting spin_lock() inside a raw_spin_lock() violates locking rules for PREEMPT_RT kernels. And additionally we do alloc(GFP_ATOMICS) inside the stab->lock, but those might sleep on PREEMPT_RT kernels. The result is splats like this: ./test_progs -t sockmap_basic [ 33.344330] bpf_testmod: loading out-of-tree module taints kernel. [ 33.441933] [ 33.442089] ============================= [ 33.442421] [ BUG: Invalid wait context ] [ 33.442763] 6.5.0-rc5-01731-gec0ded2e0282 #4958 Tainted: G O [ 33.443320] ----------------------------- [ 33.443624] test_progs/2073 is trying to lock: [ 33.443960] ffff888102a1c290 (&psock->link_lock){....}-{3:3}, at: sock_map_update_common+0x2c2/0x3d0 [ 33.444636] other info that might help us debug this: [ 33.444991] context-{5:5} [ 33.445183] 3 locks held by test_progs/2073: [ 33.445498] #0: ffff88811a208d30 (sk_lock-AF_INET){+.+.}-{0:0}, at: sock_map_update_elem_sys+0xff/0x330 [ 33.446159] #1: ffffffff842539e0 (rcu_read_lock){....}-{1:3}, at: sock_map_update_elem_sys+0xf5/0x330 [ 33.446809] #2: ffff88810d687240 (&stab->lock){+...}-{2:2}, at: sock_map_update_common+0x177/0x3d0 [ 33.447445] stack backtrace: [ 33.447655] CPU: 10 PID To fix observe we can't readily remove the allocations (for that we would need to use/create something similar to bpf_map_alloc). So convert raw_spin_lock_t to spin_lock_t. We note that sock_map_update that would trigger the allocate and potential sleep is only allowed through sys_bpf ops and via sock_ops which precludes hw interrupts and low level atomic sections in RT preempt kernel. On non RT preempt kernel there are no changes here and spin locks sections and alloc(GFP_ATOMIC) are still not sleepable. Signed-off-by: John Fastabend Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230830053517.166611-1-john.fastabend@gmail.com --- net/core/sock_map.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 8f07fea39d9ea..cb11750b1df5f 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -18,7 +18,7 @@ struct bpf_stab { struct bpf_map map; struct sock **sks; struct sk_psock_progs progs; - raw_spinlock_t lock; + spinlock_t lock; }; #define SOCK_CREATE_FLAG_MASK \ @@ -44,7 +44,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&stab->map, attr); - raw_spin_lock_init(&stab->lock); + spin_lock_init(&stab->lock); stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * sizeof(struct sock *), @@ -411,7 +411,7 @@ static int __sock_map_delete(struct bpf_stab *stab, struct sock *sk_test, struct sock *sk; int err = 0; - raw_spin_lock_bh(&stab->lock); + spin_lock_bh(&stab->lock); sk = *psk; if (!sk_test || sk_test == sk) sk = xchg(psk, NULL); @@ -421,7 +421,7 @@ static int __sock_map_delete(struct bpf_stab *stab, struct sock *sk_test, else err = -EINVAL; - raw_spin_unlock_bh(&stab->lock); + spin_unlock_bh(&stab->lock); return err; } @@ -487,7 +487,7 @@ static int sock_map_update_common(struct bpf_map *map, u32 idx, psock = sk_psock(sk); WARN_ON_ONCE(!psock); - raw_spin_lock_bh(&stab->lock); + spin_lock_bh(&stab->lock); osk = stab->sks[idx]; if (osk && flags == BPF_NOEXIST) { ret = -EEXIST; @@ -501,10 +501,10 @@ static int sock_map_update_common(struct bpf_map *map, u32 idx, stab->sks[idx] = sk; if (osk) sock_map_unref(osk, &stab->sks[idx]); - raw_spin_unlock_bh(&stab->lock); + spin_unlock_bh(&stab->lock); return 0; out_unlock: - raw_spin_unlock_bh(&stab->lock); + spin_unlock_bh(&stab->lock); if (psock) sk_psock_put(sk, psock); out_free: @@ -835,7 +835,7 @@ struct bpf_shtab_elem { struct bpf_shtab_bucket { struct hlist_head head; - raw_spinlock_t lock; + spinlock_t lock; }; struct bpf_shtab { @@ -910,7 +910,7 @@ static void sock_hash_delete_from_link(struct bpf_map *map, struct sock *sk, * is okay since it's going away only after RCU grace period. * However, we need to check whether it's still present. */ - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); elem_probe = sock_hash_lookup_elem_raw(&bucket->head, elem->hash, elem->key, map->key_size); if (elem_probe && elem_probe == elem) { @@ -918,7 +918,7 @@ static void sock_hash_delete_from_link(struct bpf_map *map, struct sock *sk, sock_map_unref(elem->sk, elem); sock_hash_free_elem(htab, elem); } - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); } static long sock_hash_delete_elem(struct bpf_map *map, void *key) @@ -932,7 +932,7 @@ static long sock_hash_delete_elem(struct bpf_map *map, void *key) hash = sock_hash_bucket_hash(key, key_size); bucket = sock_hash_select_bucket(htab, hash); - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); elem = sock_hash_lookup_elem_raw(&bucket->head, hash, key, key_size); if (elem) { hlist_del_rcu(&elem->node); @@ -940,7 +940,7 @@ static long sock_hash_delete_elem(struct bpf_map *map, void *key) sock_hash_free_elem(htab, elem); ret = 0; } - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); return ret; } @@ -1000,7 +1000,7 @@ static int sock_hash_update_common(struct bpf_map *map, void *key, hash = sock_hash_bucket_hash(key, key_size); bucket = sock_hash_select_bucket(htab, hash); - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); elem = sock_hash_lookup_elem_raw(&bucket->head, hash, key, key_size); if (elem && flags == BPF_NOEXIST) { ret = -EEXIST; @@ -1026,10 +1026,10 @@ static int sock_hash_update_common(struct bpf_map *map, void *key, sock_map_unref(elem->sk, elem); sock_hash_free_elem(htab, elem); } - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); return 0; out_unlock: - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); sk_psock_put(sk, psock); out_free: sk_psock_free_link(link); @@ -1115,7 +1115,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) for (i = 0; i < htab->buckets_num; i++) { INIT_HLIST_HEAD(&htab->buckets[i].head); - raw_spin_lock_init(&htab->buckets[i].lock); + spin_lock_init(&htab->buckets[i].lock); } return &htab->map; @@ -1147,11 +1147,11 @@ static void sock_hash_free(struct bpf_map *map) * exists, psock exists and holds a ref to socket. That * lets us to grab a socket ref too. */ - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); hlist_for_each_entry(elem, &bucket->head, node) sock_hold(elem->sk); hlist_move_list(&bucket->head, &unlink_list); - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); /* Process removed entries out of atomic context to * block for socket lock before deleting the psock's From aee1720eeb87a3adc242eb07e5d4f7ba3eb8c736 Mon Sep 17 00:00:00 2001 From: David Vernet Date: Mon, 28 Aug 2023 10:59:46 -0500 Subject: [PATCH 08/15] bpf, docs: Move linux-notes.rst to root bpf docs tree In commit 4d496be9ca05 ("bpf,docs: Create new standardization subdirectory"), I added a standardization/ directory to the BPF documentation, which will contain the docs that will be standardized as part of the effort with the IETF. I included linux-notes.rst in that directory, but I shouldn't have. It doesn't contain anything that will be standardized. Let's move it back to Documentation/bpf. Signed-off-by: David Vernet Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230828155948.123405-2-void@manifault.com --- Documentation/bpf/index.rst | 1 + Documentation/bpf/{standardization => }/linux-notes.rst | 0 Documentation/bpf/standardization/index.rst | 1 - 3 files changed, 1 insertion(+), 1 deletion(-) rename Documentation/bpf/{standardization => }/linux-notes.rst (100%) diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst index 1ff177b89d66b..aeaeb35e6d4a7 100644 --- a/Documentation/bpf/index.rst +++ b/Documentation/bpf/index.rst @@ -29,6 +29,7 @@ that goes into great technical depth about the BPF Architecture. bpf_licensing test_debug clang-notes + linux-notes other redirect diff --git a/Documentation/bpf/standardization/linux-notes.rst b/Documentation/bpf/linux-notes.rst similarity index 100% rename from Documentation/bpf/standardization/linux-notes.rst rename to Documentation/bpf/linux-notes.rst diff --git a/Documentation/bpf/standardization/index.rst b/Documentation/bpf/standardization/index.rst index 09c6ba055fd70..d7b946f71261f 100644 --- a/Documentation/bpf/standardization/index.rst +++ b/Documentation/bpf/standardization/index.rst @@ -12,7 +12,6 @@ for the working group charter, documents, and more. :maxdepth: 1 instruction-set - linux-notes .. Links: .. _IETF BPF Working Group: https://datatracker.ietf.org/wg/bpf/about/ From deb88407254621bf926658cff49a7ba01b59dec6 Mon Sep 17 00:00:00 2001 From: David Vernet Date: Mon, 28 Aug 2023 10:59:47 -0500 Subject: [PATCH 09/15] bpf, docs: Add abi.rst document to standardization subdirectory As specified in the IETF BPF charter, the BPF working group has plans to add one or more informational documents that recommend conventions and guidelines for producing portable BPF program binaries. The instruction-set.rst document currently contains a "Registers and calling convention" subsection which dictates a calling convention that belongs in an ABI document, rather than an instruction set document. Let's move it to a new abi.rst document so we can clean it up. The abi.rst document will of course be significantly changed and expanded upon over time. For now, it's really just a placeholder which will contain ABI-specific language that doesn't belong in other documents. Signed-off-by: David Vernet Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230828155948.123405-3-void@manifault.com --- Documentation/bpf/standardization/abi.rst | 25 +++++++++++++++++++ Documentation/bpf/standardization/index.rst | 1 + .../bpf/standardization/instruction-set.rst | 16 ------------ 3 files changed, 26 insertions(+), 16 deletions(-) create mode 100644 Documentation/bpf/standardization/abi.rst diff --git a/Documentation/bpf/standardization/abi.rst b/Documentation/bpf/standardization/abi.rst new file mode 100644 index 0000000000000..0c2e10eeb89a0 --- /dev/null +++ b/Documentation/bpf/standardization/abi.rst @@ -0,0 +1,25 @@ +.. contents:: +.. sectnum:: + +=================================================== +BPF ABI Recommended Conventions and Guidelines v1.0 +=================================================== + +This is version 1.0 of an informational document containing recommended +conventions and guidelines for producing portable BPF program binaries. + +Registers and calling convention +================================ + +BPF has 10 general purpose registers and a read-only frame pointer register, +all of which are 64-bits wide. + +The BPF calling convention is defined as: + +* R0: return value from function calls, and exit value for BPF programs +* R1 - R5: arguments for function calls +* R6 - R9: callee saved registers that function calls will preserve +* R10: read-only frame pointer to access stack + +R0 - R5 are scratch registers and BPF programs needs to spill/fill them if +necessary across calls. diff --git a/Documentation/bpf/standardization/index.rst b/Documentation/bpf/standardization/index.rst index d7b946f71261f..a50c3baf6345a 100644 --- a/Documentation/bpf/standardization/index.rst +++ b/Documentation/bpf/standardization/index.rst @@ -12,6 +12,7 @@ for the working group charter, documents, and more. :maxdepth: 1 instruction-set + abi .. Links: .. _IETF BPF Working Group: https://datatracker.ietf.org/wg/bpf/about/ diff --git a/Documentation/bpf/standardization/instruction-set.rst b/Documentation/bpf/standardization/instruction-set.rst index c5b0b2011f162..83583d735e38f 100644 --- a/Documentation/bpf/standardization/instruction-set.rst +++ b/Documentation/bpf/standardization/instruction-set.rst @@ -97,22 +97,6 @@ Definitions A: 10000110 B: 11111111 10000110 -Registers and calling convention -================================ - -eBPF has 10 general purpose registers and a read-only frame pointer register, -all of which are 64-bits wide. - -The eBPF calling convention is defined as: - -* R0: return value from function calls, and exit value for eBPF programs -* R1 - R5: arguments for function calls -* R6 - R9: callee saved registers that function calls will preserve -* R10: read-only frame pointer to access stack - -R0 - R5 are scratch registers and eBPF programs needs to spill/fill them if -necessary across calls. - Instruction encoding ==================== From 7d35eb1a184a3f0759ad9e9cde4669b5c55b2063 Mon Sep 17 00:00:00 2001 From: David Vernet Date: Mon, 28 Aug 2023 10:59:48 -0500 Subject: [PATCH 10/15] bpf, docs: s/eBPF/BPF in standards documents There isn't really anything other than just "BPF" at this point, so referring to it as "eBPF" in our standards document just causes unnecessary confusion. Let's just be consistent and use "BPF". Suggested-by: Will Hawkins Signed-off-by: David Vernet Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230828155948.123405-4-void@manifault.com --- .../bpf/standardization/instruction-set.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/Documentation/bpf/standardization/instruction-set.rst b/Documentation/bpf/standardization/instruction-set.rst index 83583d735e38f..c5d53a6e8c79f 100644 --- a/Documentation/bpf/standardization/instruction-set.rst +++ b/Documentation/bpf/standardization/instruction-set.rst @@ -1,11 +1,11 @@ .. contents:: .. sectnum:: -======================================== -eBPF Instruction Set Specification, v1.0 -======================================== +======================================= +BPF Instruction Set Specification, v1.0 +======================================= -This document specifies version 1.0 of the eBPF instruction set. +This document specifies version 1.0 of the BPF instruction set. Documentation conventions ========================= @@ -100,7 +100,7 @@ Definitions Instruction encoding ==================== -eBPF has two instruction encodings: +BPF has two instruction encodings: * the basic instruction encoding, which uses 64 bits to encode an instruction * the wide instruction encoding, which appends a second 64-bit immediate (i.e., @@ -244,7 +244,7 @@ BPF_END 0xd0 0 byte swap operations (see `Byte swap instructions`_ b ========= ===== ======= ========================================================== Underflow and overflow are allowed during arithmetic operations, meaning -the 64-bit or 32-bit value will wrap. If eBPF program execution would +the 64-bit or 32-bit value will wrap. If BPF program execution would result in division by zero, the destination register is instead set to zero. If execution would result in modulo by zero, for ``BPF_ALU64`` the value of the destination register is unchanged whereas for ``BPF_ALU`` the upper @@ -366,7 +366,7 @@ BPF_JSLT 0xc any PC += offset if dst < src signed BPF_JSLE 0xd any PC += offset if dst <= src signed ======== ===== === =========================================== ========================================= -The eBPF program needs to store the return value into register R0 before doing a +The BPF program needs to store the return value into register R0 before doing a ``BPF_EXIT``. Example: @@ -486,9 +486,9 @@ Atomic operations Atomic operations are operations that operate on memory and can not be interrupted or corrupted by other access to the same memory region -by other eBPF programs or means outside of this specification. +by other BPF programs or means outside of this specification. -All atomic operations supported by eBPF are encoded as store operations +All atomic operations supported by BPF are encoded as store operations that use the ``BPF_ATOMIC`` mode modifier as follows: * ``BPF_ATOMIC | BPF_W | BPF_STX`` for 32-bit operations @@ -578,7 +578,7 @@ where Maps ~~~~ -Maps are shared memory regions accessible by eBPF programs on some platforms. +Maps are shared memory regions accessible by BPF programs on some platforms. A map can have various semantics as defined in a separate document, and may or may not have a single contiguous memory region, but the 'map_val(map)' is currently only defined for maps that do have a single contiguous memory region. @@ -600,6 +600,6 @@ identified by the given id. Legacy BPF Packet access instructions ------------------------------------- -eBPF previously introduced special instructions for access to packet data that were +BPF previously introduced special instructions for access to packet data that were carried over from classic BPF. However, these instructions are deprecated and should no longer be used. From 3e019d8a05a38abb5c85d4f1e85fda964610aa14 Mon Sep 17 00:00:00 2001 From: Magnus Karlsson Date: Thu, 31 Aug 2023 12:01:17 +0200 Subject: [PATCH 11/15] xsk: Fix xsk_diag use-after-free error during socket cleanup Fix a use-after-free error that is possible if the xsk_diag interface is used after the socket has been unbound from the device. This can happen either due to the socket being closed or the device disappearing. In the early days of AF_XDP, the way we tested that a socket was not bound to a device was to simply check if the netdevice pointer in the xsk socket structure was NULL. Later, a better system was introduced by having an explicit state variable in the xsk socket struct. For example, the state of a socket that is on the way to being closed and has been unbound from the device is XSK_UNBOUND. The commit in the Fixes tag below deleted the old way of signalling that a socket is unbound, setting dev to NULL. This in the belief that all code using the old way had been exterminated. That was unfortunately not true as the xsk diagnostics code was still using the old way and thus does not work as intended when a socket is going down. Fix this by introducing a test against the state variable. If the socket is in the state XSK_UNBOUND, simply abort the diagnostic's netlink operation. Fixes: 18b1ab7aa76b ("xsk: Fix race at socket teardown") Reported-by: syzbot+822d1359297e2694f873@syzkaller.appspotmail.com Signed-off-by: Magnus Karlsson Signed-off-by: Daniel Borkmann Tested-by: syzbot+822d1359297e2694f873@syzkaller.appspotmail.com Tested-by: Maciej Fijalkowski Reviewed-by: Maciej Fijalkowski Link: https://lore.kernel.org/bpf/20230831100119.17408-1-magnus.karlsson@gmail.com --- net/xdp/xsk_diag.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/net/xdp/xsk_diag.c b/net/xdp/xsk_diag.c index c014217f5fa7d..22b36c8143cfd 100644 --- a/net/xdp/xsk_diag.c +++ b/net/xdp/xsk_diag.c @@ -111,6 +111,9 @@ static int xsk_diag_fill(struct sock *sk, struct sk_buff *nlskb, sock_diag_save_cookie(sk, msg->xdiag_cookie); mutex_lock(&xs->mutex); + if (READ_ONCE(xs->state) == XSK_UNBOUND) + goto out_nlmsg_trim; + if ((req->xdiag_show & XDP_SHOW_INFO) && xsk_diag_put_info(xs, nlskb)) goto out_nlmsg_trim; From 121fd33bf2d99007f8fe2a155c291a30baca3f52 Mon Sep 17 00:00:00 2001 From: Vishal Chourasia Date: Tue, 29 Aug 2023 13:19:31 +0530 Subject: [PATCH 12/15] bpf, docs: Fix invalid escape sequence warnings in bpf_doc.py The script bpf_doc.py generates multiple SyntaxWarnings related to invalid escape sequences when executed with Python 3.12. These warnings do not appear in Python 3.10 and 3.11 and do not affect the kernel build, which completes successfully. This patch resolves these SyntaxWarnings by converting the relevant string literals to raw strings or by escaping backslashes. This ensures that backslashes are interpreted as literal characters, eliminating the warnings. Reported-by: Srikar Dronamraju Signed-off-by: Vishal Chourasia Signed-off-by: Daniel Borkmann Tested-by: Quentin Monnet Link: https://lore.kernel.org/bpf/20230829074931.2511204-1-vishalc@linux.ibm.com --- scripts/bpf_doc.py | 56 +++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/scripts/bpf_doc.py b/scripts/bpf_doc.py index eaae2ce783811..61b7dddedc461 100755 --- a/scripts/bpf_doc.py +++ b/scripts/bpf_doc.py @@ -59,9 +59,9 @@ def proto_break_down(self): Break down helper function protocol into smaller chunks: return type, name, distincts arguments. """ - arg_re = re.compile('((\w+ )*?(\w+|...))( (\**)(\w+))?$') + arg_re = re.compile(r'((\w+ )*?(\w+|...))( (\**)(\w+))?$') res = {} - proto_re = re.compile('(.+) (\**)(\w+)\(((([^,]+)(, )?){1,5})\)$') + proto_re = re.compile(r'(.+) (\**)(\w+)\(((([^,]+)(, )?){1,5})\)$') capture = proto_re.match(self.proto) res['ret_type'] = capture.group(1) @@ -114,11 +114,11 @@ def parse_helper(self): return Helper(proto=proto, desc=desc, ret=ret) def parse_symbol(self): - p = re.compile(' \* ?(BPF\w+)$') + p = re.compile(r' \* ?(BPF\w+)$') capture = p.match(self.line) if not capture: raise NoSyscallCommandFound - end_re = re.compile(' \* ?NOTES$') + end_re = re.compile(r' \* ?NOTES$') end = end_re.match(self.line) if end: raise NoSyscallCommandFound @@ -133,7 +133,7 @@ def parse_proto(self): # - Same as above, with "const" and/or "struct" in front of type # - "..." (undefined number of arguments, for bpf_trace_printk()) # There is at least one term ("void"), and at most five arguments. - p = re.compile(' \* ?((.+) \**\w+\((((const )?(struct )?(\w+|\.\.\.)( \**\w+)?)(, )?){1,5}\))$') + p = re.compile(r' \* ?((.+) \**\w+\((((const )?(struct )?(\w+|\.\.\.)( \**\w+)?)(, )?){1,5}\))$') capture = p.match(self.line) if not capture: raise NoHelperFound @@ -141,7 +141,7 @@ def parse_proto(self): return capture.group(1) def parse_desc(self, proto): - p = re.compile(' \* ?(?:\t| {5,8})Description$') + p = re.compile(r' \* ?(?:\t| {5,8})Description$') capture = p.match(self.line) if not capture: raise Exception("No description section found for " + proto) @@ -154,7 +154,7 @@ def parse_desc(self, proto): if self.line == ' *\n': desc += '\n' else: - p = re.compile(' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') + p = re.compile(r' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') capture = p.match(self.line) if capture: desc_present = True @@ -167,7 +167,7 @@ def parse_desc(self, proto): return desc def parse_ret(self, proto): - p = re.compile(' \* ?(?:\t| {5,8})Return$') + p = re.compile(r' \* ?(?:\t| {5,8})Return$') capture = p.match(self.line) if not capture: raise Exception("No return section found for " + proto) @@ -180,7 +180,7 @@ def parse_ret(self, proto): if self.line == ' *\n': ret += '\n' else: - p = re.compile(' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') + p = re.compile(r' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') capture = p.match(self.line) if capture: ret_present = True @@ -219,12 +219,12 @@ def parse_enum_syscall(self): self.seek_to('enum bpf_cmd {', 'Could not find start of bpf_cmd enum', 0) # Searches for either one or more BPF\w+ enums - bpf_p = re.compile('\s*(BPF\w+)+') + bpf_p = re.compile(r'\s*(BPF\w+)+') # Searches for an enum entry assigned to another entry, # for e.g. BPF_PROG_RUN = BPF_PROG_TEST_RUN, which is # not documented hence should be skipped in check to # determine if the right number of syscalls are documented - assign_p = re.compile('\s*(BPF\w+)\s*=\s*(BPF\w+)') + assign_p = re.compile(r'\s*(BPF\w+)\s*=\s*(BPF\w+)') bpf_cmd_str = '' while True: capture = assign_p.match(self.line) @@ -239,7 +239,7 @@ def parse_enum_syscall(self): break self.line = self.reader.readline() # Find the number of occurences of BPF\w+ - self.enum_syscalls = re.findall('(BPF\w+)+', bpf_cmd_str) + self.enum_syscalls = re.findall(r'(BPF\w+)+', bpf_cmd_str) def parse_desc_helpers(self): self.seek_to(helpersDocStart, @@ -263,7 +263,7 @@ def parse_define_helpers(self): self.seek_to('#define ___BPF_FUNC_MAPPER(FN, ctx...)', 'Could not find start of eBPF helper definition list') # Searches for one FN(\w+) define or a backslash for newline - p = re.compile('\s*FN\((\w+), (\d+), ##ctx\)|\\\\') + p = re.compile(r'\s*FN\((\w+), (\d+), ##ctx\)|\\\\') fn_defines_str = '' i = 0 while True: @@ -278,7 +278,7 @@ def parse_define_helpers(self): break self.line = self.reader.readline() # Find the number of occurences of FN(\w+) - self.define_unique_helpers = re.findall('FN\(\w+, \d+, ##ctx\)', fn_defines_str) + self.define_unique_helpers = re.findall(r'FN\(\w+, \d+, ##ctx\)', fn_defines_str) def validate_helpers(self): last_helper = '' @@ -425,7 +425,7 @@ def get_last_doc_update(self, delimiter): try: cmd = ['git', 'log', '-1', '--pretty=format:%cs', '--no-patch', '-L', - '/{}/,/\*\//:include/uapi/linux/bpf.h'.format(delimiter)] + '/{}/,/\\*\\//:include/uapi/linux/bpf.h'.format(delimiter)] date = subprocess.run(cmd, cwd=linuxRoot, capture_output=True, check=True) return date.stdout.decode().rstrip() @@ -516,7 +516,7 @@ def print_footer(self): programs that are compatible with the GNU Privacy License (GPL). In order to use such helpers, the eBPF program must be loaded with the correct -license string passed (via **attr**) to the **bpf**\ () system call, and this +license string passed (via **attr**) to the **bpf**\\ () system call, and this generally translates into the C source code of the program containing a line similar to the following: @@ -550,7 +550,7 @@ def print_footer(self): * The bpftool utility can be used to probe the availability of helper functions on the system (as well as supported program and map types, and a number of other parameters). To do so, run **bpftool feature probe** (see - **bpftool-feature**\ (8) for details). Add the **unprivileged** keyword to + **bpftool-feature**\\ (8) for details). Add the **unprivileged** keyword to list features available to unprivileged users. Compatibility between helper functions and program types can generally be found @@ -562,23 +562,23 @@ def print_footer(self): requirement for GPL license is also in those **struct bpf_func_proto**. Compatibility between helper functions and map types can be found in the -**check_map_func_compatibility**\ () function in file *kernel/bpf/verifier.c*. +**check_map_func_compatibility**\\ () function in file *kernel/bpf/verifier.c*. Helper functions that invalidate the checks on **data** and **data_end** pointers for network processing are listed in function -**bpf_helper_changes_pkt_data**\ () in file *net/core/filter.c*. +**bpf_helper_changes_pkt_data**\\ () in file *net/core/filter.c*. SEE ALSO ======== -**bpf**\ (2), -**bpftool**\ (8), -**cgroups**\ (7), -**ip**\ (8), -**perf_event_open**\ (2), -**sendmsg**\ (2), -**socket**\ (7), -**tc-bpf**\ (8)''' +**bpf**\\ (2), +**bpftool**\\ (8), +**cgroups**\\ (7), +**ip**\\ (8), +**perf_event_open**\\ (2), +**sendmsg**\\ (2), +**socket**\\ (7), +**tc-bpf**\\ (8)''' print(footer) def print_proto(self, helper): @@ -598,7 +598,7 @@ def print_proto(self, helper): one_arg = '{}{}'.format(comma, a['type']) if a['name']: if a['star']: - one_arg += ' {}**\ '.format(a['star'].replace('*', '\\*')) + one_arg += ' {}**\\ '.format(a['star'].replace('*', '\\*')) else: one_arg += '** ' one_arg += '*{}*\\ **'.format(a['name']) From d11ae1b16b0a57fac524cad8e277a20ec62600d1 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Thu, 31 Aug 2023 16:11:03 +0200 Subject: [PATCH 13/15] selftests/bpf: Fix d_path test Recent commit [1] broke d_path test, because now filp_close is not called directly from sys_close, but eventually later when the file is finally released. As suggested by Hou Tao we don't need to re-hook the bpf program, but just instead we can use sys_close_range to trigger filp_close synchronously. [1] 021a160abf62 ("fs: use __fput_sync in close(2)") Suggested-by: Hou Tao Signed-off-by: Jiri Olsa Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230831141103.359810-1-jolsa@kernel.org --- .../testing/selftests/bpf/prog_tests/d_path.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/prog_tests/d_path.c b/tools/testing/selftests/bpf/prog_tests/d_path.c index 911345c526e6a..ccc768592e66a 100644 --- a/tools/testing/selftests/bpf/prog_tests/d_path.c +++ b/tools/testing/selftests/bpf/prog_tests/d_path.c @@ -12,6 +12,17 @@ #include "test_d_path_check_rdonly_mem.skel.h" #include "test_d_path_check_types.skel.h" +/* sys_close_range is not around for long time, so let's + * make sure we can call it on systems with older glibc + */ +#ifndef __NR_close_range +#ifdef __alpha__ +#define __NR_close_range 546 +#else +#define __NR_close_range 436 +#endif +#endif + static int duration; static struct { @@ -90,7 +101,11 @@ static int trigger_fstat_events(pid_t pid) fstat(indicatorfd, &fileStat); out_close: - /* triggers filp_close */ + /* sys_close no longer triggers filp_close, but we can + * call sys_close_range instead which still does + */ +#define close(fd) syscall(__NR_close_range, fd, fd, 0) + close(pipefd[0]); close(pipefd[1]); close(sockfd); @@ -98,6 +113,8 @@ static int trigger_fstat_events(pid_t pid) close(devfd); close(localfd); close(indicatorfd); + +#undef close return ret; } From 6a86b5b5cd76d2734304a0173f5f01aa8aa2025e Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 29 Aug 2023 22:53:52 +0200 Subject: [PATCH 14/15] bpf: Annotate bpf_long_memcpy with data_race syzbot reported a data race splat between two processes trying to update the same BPF map value via syscall on different CPUs: BUG: KCSAN: data-race in bpf_percpu_array_update / bpf_percpu_array_update write to 0xffffe8fffe7425d8 of 8 bytes by task 8257 on cpu 1: bpf_long_memcpy include/linux/bpf.h:428 [inline] bpf_obj_memcpy include/linux/bpf.h:441 [inline] copy_map_value_long include/linux/bpf.h:464 [inline] bpf_percpu_array_update+0x3bb/0x500 kernel/bpf/arraymap.c:380 bpf_map_update_value+0x190/0x370 kernel/bpf/syscall.c:175 generic_map_update_batch+0x3ae/0x4f0 kernel/bpf/syscall.c:1749 bpf_map_do_batch+0x2df/0x3d0 kernel/bpf/syscall.c:4648 __sys_bpf+0x28a/0x780 __do_sys_bpf kernel/bpf/syscall.c:5241 [inline] __se_sys_bpf kernel/bpf/syscall.c:5239 [inline] __x64_sys_bpf+0x43/0x50 kernel/bpf/syscall.c:5239 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd write to 0xffffe8fffe7425d8 of 8 bytes by task 8268 on cpu 0: bpf_long_memcpy include/linux/bpf.h:428 [inline] bpf_obj_memcpy include/linux/bpf.h:441 [inline] copy_map_value_long include/linux/bpf.h:464 [inline] bpf_percpu_array_update+0x3bb/0x500 kernel/bpf/arraymap.c:380 bpf_map_update_value+0x190/0x370 kernel/bpf/syscall.c:175 generic_map_update_batch+0x3ae/0x4f0 kernel/bpf/syscall.c:1749 bpf_map_do_batch+0x2df/0x3d0 kernel/bpf/syscall.c:4648 __sys_bpf+0x28a/0x780 __do_sys_bpf kernel/bpf/syscall.c:5241 [inline] __se_sys_bpf kernel/bpf/syscall.c:5239 [inline] __x64_sys_bpf+0x43/0x50 kernel/bpf/syscall.c:5239 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd value changed: 0x0000000000000000 -> 0xfffffff000002788 The bpf_long_memcpy is used with 8-byte aligned pointers, power-of-8 size and forced to use long read/writes to try to atomically copy long counters. It is best-effort only and no barriers are here since it _will_ race with concurrent updates from BPF programs. The bpf_long_memcpy() is called from bpf(2) syscall. Marco suggested that the best way to make this known to KCSAN would be to use data_race() annotation. Reported-by: syzbot+97522333291430dd277f@syzkaller.appspotmail.com Suggested-by: Marco Elver Signed-off-by: Daniel Borkmann Acked-by: Marco Elver Link: https://lore.kernel.org/bpf/000000000000d87a7f06040c970c@google.com Link: https://lore.kernel.org/bpf/57628f7a15e20d502247c3b55fceb1cb2b31f266.1693342186.git.daniel@iogearbox.net --- include/linux/bpf.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 12596af59c004..024e8b28c34b8 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -438,7 +438,7 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size) size /= sizeof(long); while (size--) - *ldst++ = *lsrc++; + data_race(*ldst++ = *lsrc++); } /* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */ From be8e754cbfac698d6304bb8382c8d18ac74424d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= Date: Thu, 31 Aug 2023 18:29:54 +0200 Subject: [PATCH 15/15] selftests/bpf: Include build flavors for install target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When using the "install" or targets depending on install, e.g. "gen_tar", the BPF machine flavors weren't included. A command like: | make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- O=/workspace/kbuild \ | HOSTCC=gcc FORMAT= SKIP_TARGETS="arm64 ia64 powerpc sparc64 x86 sgx" \ | -C tools/testing/selftests gen_tar would not include bpf/no_alu32, bpf/cpuv4, or bpf/bpf-gcc. Include the BPF machine flavors for "install" make target. Signed-off-by: Björn Töpel Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20230831162954.111485-1-bjorn@kernel.org --- tools/testing/selftests/bpf/Makefile | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index edef49fcd23e3..caede9b574cb1 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -50,14 +50,17 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test test_cgroup_storage \ test_tcpnotify_user test_sysctl \ test_progs-no_alu32 +TEST_INST_SUBDIRS := no_alu32 # Also test bpf-gcc, if present ifneq ($(BPF_GCC),) TEST_GEN_PROGS += test_progs-bpf_gcc +TEST_INST_SUBDIRS += bpf_gcc endif ifneq ($(CLANG_CPUV4),) TEST_GEN_PROGS += test_progs-cpuv4 +TEST_INST_SUBDIRS += cpuv4 endif TEST_GEN_FILES = test_lwt_ip_encap.bpf.o test_tc_edt.bpf.o @@ -714,3 +717,12 @@ EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \ # Delete partially updated (corrupted) files on error .DELETE_ON_ERROR: + +DEFAULT_INSTALL_RULE := $(INSTALL_RULE) +override define INSTALL_RULE + $(DEFAULT_INSTALL_RULE) + @for DIR in $(TEST_INST_SUBDIRS); do \ + mkdir -p $(INSTALL_PATH)/$$DIR; \ + rsync -a $(OUTPUT)/$$DIR/*.bpf.o $(INSTALL_PATH)/$$DIR;\ + done +endef