Jann Horn
48b00ceb54
mm/khugepaged: fix GUP-fast interaction by sending IPI
...
commit 2ba99c5e08812494bc57f319fb562f527d9bacd8 upstream.
Since commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP
collapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t to
ensure that the page table was not removed by khugepaged in between.
However, lockless_pages_from_mm() still requires that the page table is
not concurrently freed. Fix it by sending IPIs (if the architecture uses
semi-RCU-style page table freeing) before freeing/reusing page tables.
Link: https://lkml.kernel.org/r/20221129154730.2274278-2-jannh@google.com
Link: https://lkml.kernel.org/r/20221128180252.1684965-2-jannh@google.com
Link: https://lkml.kernel.org/r/20221125213714.4115729-2-jannh@google.com
Fixes: ba76149f47 ("thp: khugepaged")
Signed-off-by: Jann Horn <jannh@google.com>
Reviewed-by: Yang Shi <shy828301@gmail.com>
Acked-by: David Hildenbrand <david@redhat.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[manual backport: two of the three places in khugepaged that can free
ptes were refactored into a common helper between 5.15 and 6.0;
TLB flushing was refactored between 5.4 and 5.10]
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2022-12-14 11:30:42 +01:00
..
bitops
locking/atomic: Make test_and_*_bit() ordered on failure
2022-08-25 11:18:31 +02:00
vdso
lib/vdso: Make __arch_update_vdso_data() logic understandable
2020-03-05 16:43:49 +01:00
4level-fixup.h
5level-fixup.h
include/asm-generic/5level-fixup.h: fix variable 'p4d' set but not used
2019-08-13 16:06:52 -07:00
asm-offsets.h
asm-prototypes.h
atomic-instrumented.h
atomic-long.h
atomic.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
atomic64.h
locking/atomic: Use s64 for atomic64
2019-06-03 12:32:56 +02:00
audit_change_attr.h
audit_dir_write.h
audit_read.h
audit_signal.h
audit_write.h
barrier.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
bitops-instrumented.h
asm-generic, x86: add bitops instrumentation for KASAN
2019-07-12 11:05:42 -07:00
bitops.h
bitsperlong.h
bug.h
bug: move WARN_ON() "cut here" into exception handler
2019-09-25 17:51:41 -07:00
bugs.h
cache.h
cacheflush.h
asm-generic/nds32: don't redefine cacheflush primitives
2020-01-17 19:48:43 +01:00
checksum.h
cmpxchg-local.h
cmpxchg.h
compat.h
current.h
delay.h
device.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 428
2019-06-05 17:37:16 +02:00
div64.h
asm-generic changes for v5.4
2019-09-17 14:30:30 -07:00
dma-contiguous.h
dma-mapping.h
dma.h
early_ioremap.h
emergency-restart.h
error-injection.h
error-injection: Consolidate override function definition
2019-08-07 13:52:43 +01:00
exec.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
export.h
module: Remove leftover '#undef' from export header
2019-09-12 15:29:46 +02:00
extable.h
fb.h
fixmap.h
flat.h
binfmt_flat: remove the persistent argument from flat_get_addr_from_rp
2019-06-24 09:16:47 +10:00
ftrace.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500
2019-06-19 17:09:55 +02:00
futex.h
futex: Cleanup generic SMP variant of arch_futex_atomic_op_inuser()
2019-07-22 11:20:10 +02:00
getorder.h
asm-generic: fix -Wtype-limits compiler warnings
2019-08-03 07:02:01 -07:00
gpio.h
hardirq.h
hugetlb.h
hugetlb: allow to free gigantic pages regardless of the configuration
2019-05-14 09:47:47 -07:00
hw_irq.h
ide_iops.h
int-ll64.h
io.h
asm-generic: don't provide __ioremap
2019-08-30 21:46:27 +02:00
ioctl.h
iomap.h
irq.h
irq_regs.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152
2019-05-30 11:26:32 -07:00
irq_work.h
irqflags.h
Kbuild
asm-generic: make simd.h a mandatory include/asm header
2019-07-31 21:29:24 +10:00
kdebug.h
kmap_types.h
kprobes.h
kvm_para.h
linkage.h
local.h
local64.h
mcs_spinlock.h
memory_model.h
mm-arch-hooks.h
mm_hooks.h
x86/mpx, mm/core: Fix recursive munmap() corruption
2019-05-09 10:37:17 +02:00
mmiowb.h
asm-generic/mmiowb: Allow mmiowb_set_pending() when preemptible()
2020-07-29 10:18:40 +02:00
mmiowb_types.h
mmu.h
mmu_context.h
module.h
mshyperv.h
x86/Hyper-V: Report crash data in die() when panic_on_oops is set
2020-04-23 10:36:24 +02:00
msi.h
page.h
param.h
parport.h
pci.h
pci_iomap.h
percpu.h
pgalloc.h
mm: treewide: clarify pgtable_page_{ctor,dtor}() naming
2019-09-26 10:10:44 -07:00
pgtable-nop4d-hack.h
pgtable-nop4d.h
pgtable-nopmd.h
pgtable-nopud.h
pgtable.h
arch: pgtable: define MAX_POSSIBLE_PHYSMEM_BITS where needed
2020-12-02 08:49:50 +01:00
preempt.h
sched/preempt: Use CONFIG_PREEMPTION where appropriate
2019-07-31 19:03:34 +02:00
qrwlock.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
qrwlock_types.h
qspinlock.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
qspinlock_types.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
resource.h
seccomp.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500
2019-06-19 17:09:55 +02:00
sections.h
asm-generic: sections: refactor memory_intersects
2022-09-05 10:27:43 +02:00
serial.h
set_memory.h
shmparam.h
treewide: remove SPDX "WITH Linux-syscall-note" from kernel-space headers
2019-05-14 19:52:48 -07:00
signal.h
simd.h
spinlock.h
statfs.h
string.h
switch_to.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
syscall.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 193
2019-05-30 11:29:21 -07:00
syscalls.h
termios-base.h
termios.h
timex.h
tlb.h
mm/khugepaged: fix GUP-fast interaction by sending IPI
2022-12-14 11:30:42 +01:00
tlbflush.h
topology.h
include/asm-generic/topology.h: guard cpumask_of_node() macro argument
2020-06-03 08:21:27 +02:00
trace_clock.h
uaccess.h
unaligned.h
user.h
vga.h
vmlinux.lds.h
vmlinux.lds.h: Fix placement of '.data..decrypted' section
2022-11-25 17:42:09 +01:00
vtime.h
word-at-a-time.h
xor.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 47
2019-05-24 17:27:13 +02:00