- C 98.2%
- Assembly 1%
- Makefile 0.3%
- Shell 0.2%
- Python 0.1%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
This commit serves two things:
1) it optimizes BPF prologue/epilogue generation
2) it makes possible to have tailcalls within BPF subprogram
Both points are related to each other since without 1), 2) could not be
achieved.
In [1], Alexei says:
"The prologue will look like:
nop5
xor eax,eax // two new bytes if bpf_tail_call() is used in this
// function
push rbp
mov rbp, rsp
sub rsp, rounded_stack_depth
push rax // zero init tail_call counter
variable number of push rbx,r13,r14,r15
Then bpf_tail_call will pop variable number rbx,..
and final 'pop rax'
Then 'add rsp, size_of_current_stack_frame'
jmp to next function and skip over 'nop5; xor eax,eax; push rpb; mov
rbp, rsp'
This way new function will set its own stack size and will init tail
call
counter with whatever value the parent had.
If next function doesn't use bpf_tail_call it won't have 'xor eax,eax'.
Instead it would need to have 'nop2' in there."
Implement that suggestion.
Since the layout of stack is changed, tail call counter handling can not
rely anymore on popping it to rbx just like it have been handled for
constant prologue case and later overwrite of rbx with actual value of
rbx pushed to stack. Therefore, let's use one of the register (%rcx) that
is considered to be volatile/caller-saved and pop the value of tail call
counter in there in the epilogue.
Drop the BUILD_BUG_ON in emit_prologue and in
emit_bpf_tail_call_indirect where instruction layout is not constant
anymore.
Introduce new poke target, 'tailcall_bypass' to poke descriptor that is
dedicated for skipping the register pops and stack unwind that are
generated right before the actual jump to target program.
For case when the target program is not present, BPF program will skip
the pop instructions and nop5 dedicated for jmpq $target. An example of
such state when only R6 of callee saved registers is used by program:
ffffffffc0513aa1: e9 0e 00 00 00 jmpq 0xffffffffc0513ab4
ffffffffc0513aa6: 5b pop %rbx
ffffffffc0513aa7: 58 pop %rax
ffffffffc0513aa8: 48 81 c4 00 00 00 00 add $0x0,%rsp
ffffffffc0513aaf: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
ffffffffc0513ab4: 48 89 df mov %rbx,%rdi
When target program is inserted, the jump that was there to skip
pops/nop5 will become the nop5, so CPU will go over pops and do the
actual tailcall.
One might ask why there simply can not be pushes after the nop5?
In the following example snippet:
ffffffffc037030c: 48 89 fb mov %rdi,%rbx
(...)
ffffffffc0370332: 5b pop %rbx
ffffffffc0370333: 58 pop %rax
ffffffffc0370334: 48 81 c4 00 00 00 00 add $0x0,%rsp
ffffffffc037033b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
ffffffffc0370340: 48 81 ec 00 00 00 00 sub $0x0,%rsp
ffffffffc0370347: 50 push %rax
ffffffffc0370348: 53 push %rbx
ffffffffc0370349: 48 89 df mov %rbx,%rdi
ffffffffc037034c: e8 f7 21 00 00 callq 0xffffffffc0372548
There is the bpf2bpf call (at ffffffffc037034c) right after the tailcall
and jump target is not present. ctx is in %rbx register and BPF
subprogram that we will call into on ffffffffc037034c is relying on it,
e.g. it will pick ctx from there. Such code layout is therefore broken
as we would overwrite the content of %rbx with the value that was pushed
on the prologue. That is the reason for the 'bypass' approach.
Special care needs to be taken during the install/update/remove of
tailcall target. In case when target program is not present, the CPU
must not execute the pop instructions that precede the tailcall.
To address that, the following states can be defined:
A nop, unwind, nop
B nop, unwind, tail
C skip, unwind, nop
D skip, unwind, tail
A is forbidden (lead to incorrectness). The state transitions between
tailcall install/update/remove will work as follows:
First install tail call f: C->D->B(f)
* poke the tailcall, after that get rid of the skip
Update tail call f to f': B(f)->B(f')
* poke the tailcall (poke->tailcall_target) and do NOT touch the
poke->tailcall_bypass
Remove tail call: B(f')->C(f')
* poke->tailcall_bypass is poked back to jump, then we wait the RCU
grace period so that other programs will finish its execution and
after that we are safe to remove the poke->tailcall_target
Install new tail call (f''): C(f')->D(f'')->B(f'').
* same as first step
This way CPU can never be exposed to "unwind, tail" state.
Last but not least, when tailcalls get mixed with bpf2bpf calls, it
would be possible to encounter the endless loop due to clearing the
tailcall counter if for example we would use the tailcall3-like from BPF
selftests program that would be subprogram-based, meaning the tailcall
would be present within the BPF subprogram.
This test, broken down to particular steps, would do:
entry -> set tailcall counter to 0, bump it by 1, tailcall to func0
func0 -> call subprog_tail
(we are NOT skipping the first 11 bytes of prologue and this subprogram
has a tailcall, therefore we clear the counter...)
subprog -> do the same thing as entry
and then loop forever.
To address this, the idea is to go through the call chain of bpf2bpf progs
and look for a tailcall presence throughout whole chain. If we saw a single
tail call then each node in this call chain needs to be marked as a subprog
that can reach the tailcall. We would later feed the JIT with this info
and:
- set eax to 0 only when tailcall is reachable and this is the entry prog
- if tailcall is reachable but there's no tailcall in insns of currently
JITed prog then push rax anyway, so that it will be possible to
propagate further down the call chain
- finally if tailcall is reachable, then we need to precede the 'call'
insn with mov rax, [rbp - (stack_depth + 8)]
Tail call related cases from test_verifier kselftest are also working
fine. Sample BPF programs that utilize tail calls (sockex3, tracex5)
work properly as well.
[1]: https://lore.kernel.org/bpf/20200517043227.2gpq22ifoq37ogst@ast-mbp.dhcp.thefacebook.com/
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Change-Id: Ia394b1cf1a1a99ed9810395b588c7c2cbe8790e7
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
||
| android | ||
| arch | ||
| block | ||
| certs | ||
| crypto | ||
| Documentation | ||
| drivers | ||
| fs | ||
| include | ||
| init | ||
| ipc | ||
| kernel | ||
| lib | ||
| LICENSES | ||
| mm | ||
| net | ||
| samples | ||
| scripts | ||
| security | ||
| sound | ||
| techpack | ||
| tools | ||
| usr | ||
| virt | ||
| .clang-format | ||
| .cocciconfig | ||
| .get_maintainer.ignore | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| Android.mk | ||
| Androidbp | ||
| build.config.aarch64 | ||
| build.config.allmodconfig | ||
| build.config.allmodconfig.aarch64 | ||
| build.config.allmodconfig.arm | ||
| build.config.allmodconfig.x86_64 | ||
| build.config.arm | ||
| build.config.common | ||
| build.config.db845c | ||
| build.config.gki | ||
| build.config.gki-debug.aarch64 | ||
| build.config.gki-debug.x86_64 | ||
| build.config.gki.aarch64 | ||
| build.config.gki.x86_64 | ||
| build.config.gki_kasan | ||
| build.config.gki_kasan.aarch64 | ||
| build.config.gki_kasan.x86_64 | ||
| build.config.gki_kprobes | ||
| build.config.gki_kprobes.aarch64 | ||
| build.config.gki_kprobes.x86_64 | ||
| build.config.hikey960 | ||
| build.config.msm.common | ||
| build.config.msm.gki | ||
| build.config.msm.lahaina | ||
| build.config.x86_64 | ||
| build.targets | ||
| COPYING | ||
| CREDITS | ||
| gen_headers_arm.bp | ||
| gen_headers_arm64.bp | ||
| gen_headers_arm64_auto.bp | ||
| gen_headers_arm_auto.bp | ||
| Kbuild | ||
| Kconfig | ||
| kernel_headers.py | ||
| LICENSE | ||
| MAINTAINERS | ||
| Makefile | ||
| modules.list.msm.lahaina | ||
| OWNERS | ||
| README | ||
| README.md | ||
How do I submit patches to Android Common Kernels
-
BEST: Make all of your changes to upstream Linux. If appropriate, backport to the stable releases. These patches will be merged automatically in the corresponding common kernels. If the patch is already in upstream Linux, post a backport of the patch that conforms to the patch requirements below.
-
LESS GOOD: Develop your patches out-of-tree (from an upstream Linux point-of-view). Unless these are fixing an Android-specific bug, these are very unlikely to be accepted unless they have been coordinated with kernel-team@android.com. If you want to proceed, post a patch that conforms to the patch requirements below.
Common Kernel patch requirements
- All patches must conform to the Linux kernel coding standards and pass
script/checkpatch.pl - Patches shall not break gki_defconfig or allmodconfig builds for arm, arm64, x86, x86_64 architectures (see https://source.android.com/setup/build/building-kernels)
- If the patch is not merged from an upstream branch, the subject must be tagged with the type of patch:
UPSTREAM:,BACKPORT:,FROMGIT:,FROMLIST:, orANDROID:. - All patches must have a
Change-Id:tag (see https://gerrit-review.googlesource.com/Documentation/user-changeid.html) - If an Android bug has been assigned, there must be a
Bug:tag. - All patches must have a
Signed-off-by:tag by the author and the submitter
Additional requirements are listed below based on patch type
Requirements for backports from mainline Linux: UPSTREAM:, BACKPORT:
- If the patch is a cherry-pick from Linux mainline with no changes at all
- tag the patch subject with
UPSTREAM:. - add upstream commit information with a
(cherry-picked from ...)line - Example:
- if the upstream commit message is
- tag the patch subject with
important patch from upstream
This is the detailed description of the important patch
Signed-off-by: Fred Jones <fred.jones@foo.org>
- then Joe Smith would upload the patch for the common kernel as
UPSTREAM: important patch from upstream
This is the detailed description of the important patch
Signed-off-by: Fred Jones <fred.jones@foo.org>
Bug: 135791357
Change-Id: I4caaaa566ea080fa148c5e768bb1a0b6f7201c01
(cherry-picked from c31e73121f4c1ec41143423ac6ce3ce6dafdcec1)
Signed-off-by: Joe Smith <joe.smith@foo.org>
- If the patch requires any changes from the upstream version, tag the patch with
BACKPORT:instead ofUPSTREAM:.- use the same tags as
UPSTREAM: - add comments about the changes under the
(cherry-picked from ...)line - Example:
- use the same tags as
BACKPORT: important patch from upstream
This is the detailed description of the important patch
Signed-off-by: Fred Jones <fred.jones@foo.org>
Bug: 135791357
Change-Id: I4caaaa566ea080fa148c5e768bb1a0b6f7201c01
(cherry-picked from c31e73121f4c1ec41143423ac6ce3ce6dafdcec1)
[ Resolved minor conflict in drivers/foo/bar.c ]
Signed-off-by: Joe Smith <joe.smith@foo.org>
Requirements for other backports: FROMGIT:, FROMLIST:,
- If the patch has been merged into an upstream maintainer tree, but has not yet
been merged into Linux mainline
- tag the patch subject with
FROMGIT: - add info on where the patch came from as
(cherry picked from commit <sha1> <repo> <branch>). This must be a stable maintainer branch (not rebased, so don't uselinux-nextfor example). - if changes were required, use
BACKPORT: FROMGIT: - Example:
- if the commit message in the maintainer tree is
- tag the patch subject with
important patch from upstream
This is the detailed description of the important patch
Signed-off-by: Fred Jones <fred.jones@foo.org>
- then Joe Smith would upload the patch for the common kernel as
FROMGIT: important patch from upstream
This is the detailed description of the important patch
Signed-off-by: Fred Jones <fred.jones@foo.org>
Bug: 135791357
(cherry picked from commit 878a2fd9de10b03d11d2f622250285c7e63deace
https://git.kernel.org/pub/scm/linux/kernel/git/foo/bar.git test-branch)
Change-Id: I4caaaa566ea080fa148c5e768bb1a0b6f7201c01
Signed-off-by: Joe Smith <joe.smith@foo.org>
- If the patch has been submitted to LKML, but not accepted into any maintainer tree
- tag the patch subject with
FROMLIST: - add a
Link:tag with a link to the submittal on lore.kernel.org - if changes were required, use
BACKPORT: FROMLIST: - Example:
- tag the patch subject with
FROMLIST: important patch from upstream
This is the detailed description of the important patch
Signed-off-by: Fred Jones <fred.jones@foo.org>
Bug: 135791357
Link: https://lore.kernel.org/lkml/20190619171517.GA17557@someone.com/
Change-Id: I4caaaa566ea080fa148c5e768bb1a0b6f7201c01
Signed-off-by: Joe Smith <joe.smith@foo.org>
Requirements for Android-specific patches: ANDROID:
- If the patch is fixing a bug to Android-specific code
- tag the patch subject with
ANDROID: - add a
Fixes:tag that cites the patch with the bug - Example:
- tag the patch subject with
ANDROID: fix android-specific bug in foobar.c
This is the detailed description of the important fix
Fixes: 1234abcd2468 ("foobar: add cool feature")
Change-Id: I4caaaa566ea080fa148c5e768bb1a0b6f7201c01
Signed-off-by: Joe Smith <joe.smith@foo.org>
- If the patch is a new feature
- tag the patch subject with
ANDROID: - add a
Bug:tag with the Android bug (required for android-specific features)
- tag the patch subject with
Vibrator driver for HHG device
How to merge the driver into kernel source tree
-
Copy ${this_project}/drivers/hid/hid-aksys.c into ${your_kernel_root}/drivers/hid/
-
Compare and merge ${this_project}/drivers/hid/hid-ids.h into ${your_kernel_root}/drivers/hid/hid-ids.h : Add the following code before the last line of this file
#define USB_VENDER_ID_QUALCOMM 0x0a12 #define USB_VENDER_ID_TEMP_HHG_AKSY 0x1234 #define USB_PRODUCT_ID_AKSYS_HHG 0x1000 -
Merge ${this_project}/drivers/hid/Kconfig into ${your_kernel_root}/drivers/hid/Kconfig : Add the following code before the last line of this file
config HID_AKSYS_QRD tristate "AKSys gamepad USB adapter support" depends on HID ---help--- Support for AKSys gamepad USB adapter config AKSYS_QRD_FF bool "AKSys gamepad USB adapter force feedback support" depends on HID_AKSYS_QRD select INPUT_FF_MEMLESS ---help--- Say Y here if you have a AKSys gamepad USB adapter and want to enable force feedback support for it. -
Merge ${this_project}/drivers/hid/Makefile into ${your_kernel_root}/drivers/hid/Makefile : Add the following code at the end of this file
obj-$(CONFIG_HID_AKSYS_QRD) += hid-aksys.o -
Modify your kernel's default build configuration file. Add the following two lines:
CONFIG_HID_AKSYS_QRD=m CONFIG_AKSYS_QRD_FF=y