Mathieu Desnoyers
227a4aadc7
sched/membarrier: Fix p->mm->membarrier_state racy load
...
The membarrier_state field is located within the mm_struct, which
is not guaranteed to exist when used from runqueue-lock-free iteration
on runqueues by the membarrier system call.
Copy the membarrier_state from the mm_struct into the scheduler runqueue
when the scheduler switches between mm.
When registering membarrier for mm, after setting the registration bit
in the mm membarrier state, issue a synchronize_rcu() to ensure the
scheduler observes the change. In order to take care of the case
where a runqueue keeps executing the target mm without swapping to
other mm, iterate over each runqueue and issue an IPI to copy the
membarrier_state from the mm_struct into each runqueue which have the
same mm which state has just been modified.
Move the mm membarrier_state field closer to pgd in mm_struct to use
a cache line already touched by the scheduler switch_mm.
The membarrier_execve() (now membarrier_exec_mmap) hook now needs to
clear the runqueue's membarrier state in addition to clear the mm
membarrier state, so move its implementation into the scheduler
membarrier code so it can access the runqueue structure.
Add memory barrier in membarrier_exec_mmap() prior to clearing
the membarrier state, ensuring memory accesses executed prior to exec
are not reordered with the stores clearing the membarrier state.
As suggested by Linus, move all membarrier.c RCU read-side locks outside
of the for each cpu loops.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Chris Metcalf <cmetcalf@ezchip.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Eric W. Biederman <ebiederm@xmission.com>
Cc: Kirill Tkhai <tkhai@yandex.ru>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Paul E. McKenney <paulmck@linux.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Russell King - ARM Linux admin <linux@armlinux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lkml.kernel.org/r/20190919173705.2181-5-mathieu.desnoyers@efficios.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-09-25 17:42:30 +02:00
..
bpf
bpf: fix precision tracking of stack slots
2019-09-05 14:06:58 +02:00
cgroup
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
configs
debug
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
dma
Merge branches 'arm/omap', 'arm/exynos', 'arm/smmu', 'arm/mediatek', 'arm/qcom', 'arm/renesas', 'x86/amd', 'x86/vt-d' and 'core' into next
2019-09-11 12:39:19 +02:00
events
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
gcov
gcov: no need to check return value of debugfs_create functions
2019-06-03 16:18:12 +02:00
irq
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
livepatch
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching
2019-07-11 15:30:05 -07:00
locking
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
power
pci-v5.3-changes
2019-07-15 20:44:49 -07:00
printk
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 13
2019-05-21 11:28:45 +02:00
rcu
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
sched
sched/membarrier: Fix p->mm->membarrier_state racy load
2019-09-25 17:42:30 +02:00
time
timekeeping/vsyscall: Prevent math overflow in BOOTTIME update
2019-08-23 02:12:11 +02:00
trace
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
.gitignore
Provide in-kernel headers to make extending kernel easier
2019-04-29 16:48:03 +02:00
acct.c
acct_on(): don't mess with freeze protection
2019-04-04 21:04:13 -04:00
async.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
audit.c
audit/stable-5.3 PR 20190702
2019-07-08 18:55:42 -07:00
audit.h
audit/stable-5.3 PR 20190702
2019-07-08 18:55:42 -07:00
audit_fsnotify.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
audit_tree.c
fsnotify: switch send_to_group() and ->handle_event to const struct qstr *
2019-04-26 13:51:03 -04:00
audit_watch.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
auditfilter.c
audit/stable-5.3 PR 20190702
2019-07-08 18:55:42 -07:00
auditsc.c
audit: enforce op for string fields
2019-05-28 17:46:43 -04:00
backtracetest.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
bounds.c
capability.c
LSM: add SafeSetID module that gates setid calls
2019-01-25 11:22:43 -08:00
compat.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500
2019-06-19 17:09:55 +02:00
configs.c
kernel/configs: Replace GPL boilerplate code with SPDX identifier
2019-07-30 18:34:15 +02:00
context_tracking.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
cpu.c
Merge branch 'smp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-07-08 10:39:56 -07:00
cpu_pm.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 282
2019-06-05 17:36:37 +02:00
crash_core.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 230
2019-06-19 17:09:06 +02:00
crash_dump.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
cred.c
Merge branch 'access-creds'
2019-07-25 08:36:29 -07:00
delayacct.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 25
2019-05-21 11:52:39 +02:00
dma.c
elfcore.c
exec_domain.c
exit.c
tasks, sched/core: With a grace period after finish_task_switch(), remove unnecessary code
2019-09-25 17:42:29 +02:00
extable.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
fail_function.c
fail_function: no need to check return value of debugfs_create functions
2019-06-03 15:49:06 +02:00
fork.c
tasks, sched/core: Ensure tasks are available for a grace period after leaving the runqueue
2019-09-25 17:42:29 +02:00
freezer.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
futex.c
Linux 5.2-rc3
2019-06-03 11:50:18 +02:00
gen_kheaders.sh
kheaders: include only headers into kheaders_data.tar.xz
2019-07-09 10:10:52 +09:00
groups.c
hung_task.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
iomem.c
mm/nvdimm: add is_ioremap_addr and use that to check ioremap address
2019-07-12 11:05:40 -07:00
irq_work.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
jump_label.c
jump_label: Don't warn on __exit jump entries
2019-08-29 15:10:10 +01:00
kallsyms.c
kallsyms: Don't let kallsyms_lookup_size_offset() fail on retrieving the first symbol
2019-08-27 16:19:56 +01:00
kcmp.c
Kconfig.freezer
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
Kconfig.hz
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
Kconfig.locks
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
Kconfig.preempt
sched/rt, Kconfig: Unbreak def/oldconfig with CONFIG_PREEMPT=y
2019-07-22 18:05:11 +02:00
kcov.c
kcov: convert kcov.refcount to refcount_t
2019-03-07 18:32:02 -08:00
kexec.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 230
2019-06-19 17:09:06 +02:00
kexec_core.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 230
2019-06-19 17:09:06 +02:00
kexec_elf.c
kexec_elf: support 32 bit ELF files
2019-09-06 23:58:44 +02:00
kexec_file.c
Merge branch 'next-integrity' of git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity
2019-07-08 20:28:59 -07:00
kexec_internal.h
kheaders.c
kheaders: Move from proc to sysfs
2019-05-24 20:16:01 +02:00
kmod.c
kprobes.c
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 17:25:49 -07:00
ksysfs.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 170
2019-05-30 11:26:39 -07:00
kthread.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
latencytop.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
Makefile
Merge branch 'parisc-5.4-1' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux
2019-09-16 15:38:31 -07:00
module-internal.h
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
module.c
modules: page-align module section allocations only for arches supporting strict module rwx
2019-08-21 10:43:56 +02:00
module_signing.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
notifier.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
nsproxy.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
padata.c
padata: use smp_mb in padata_reorder to avoid orphaned padata jobs
2019-07-18 13:39:54 +08:00
panic.c
docs: admin-guide: move sysctl directory to it
2019-07-15 11:03:01 -03:00
params.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
pid.c
kernel/pid.c: convert struct pid count to refcount_t
2019-07-16 19:23:24 -07:00
pid_namespace.c
proc/sysctl: add shared variables for range check
2019-07-18 17:08:07 -07:00
profile.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
ptrace.c
ptrace: add PTRACE_GET_SYSCALL_INFO request
2019-07-16 19:23:24 -07:00
range.c
reboot.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
relay.c
Merge branch 'work.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-03-12 13:27:20 -07:00
resource.c
resource: avoid unnecessary lookups in find_next_iomem_res()
2019-07-18 17:08:06 -07:00
rseq.c
signal: Remove task parameter from force_sig
2019-05-27 09:36:28 -05:00
seccomp.c
signal: Remove the signal number and task parameters from force_sig_info
2019-05-29 09:31:44 -05:00
signal.c
core-process-v5.4
2019-09-16 09:28:19 -07:00
smp.c
smp: Warn on function calls from softirq context
2019-07-20 11:27:16 +02:00
smpboot.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
smpboot.h
softirq.c
Merge branch 'irq-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-07-08 11:01:13 -07:00
stackleak.c
stacktrace.c
stacktrace: Constify 'entries' arguments
2019-07-25 15:43:26 +02:00
stop_machine.c
stop_machine: Fix stop_cpus_in_progress ordering
2019-08-08 09:09:30 +02:00
sys.c
Merge branch 'x86-cpu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-16 18:47:53 -07:00
sys_ni.c
arch: handle arches who do not yet define clone3
2019-06-21 01:54:53 +02:00
sysctl.c
proc/sysctl: add shared variables for range check
2019-07-18 17:08:07 -07:00
sysctl_binary.c
kernel/sysctl: add panic_print into sysctl
2019-01-04 13:13:47 -08:00
task_work.c
taskstats.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
test_kprobes.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 25
2019-05-21 11:52:39 +02:00
torture.c
torture: Remove exporting of internal functions
2019-08-01 14:30:22 -07:00
tracepoint.c
The main changes in this release include:
2019-07-18 11:51:00 -07:00
tsacct.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
ucount.c
proc/sysctl: add shared variables for range check
2019-07-18 17:08:07 -07:00
uid16.c
uid16.h
umh.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
up.c
smp: Remove smp_call_function() and on_each_cpu() return values
2019-06-23 14:26:26 +02:00
user-return-notifier.c
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
user.c
Keyrings namespacing
2019-07-08 19:36:47 -07:00
user_namespace.c
Keyrings namespacing
2019-07-08 19:36:47 -07:00
utsname.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
utsname_sysctl.c
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
watchdog.c
watchdog: Fix typo in comment
2019-04-18 14:05:51 +02:00
watchdog_hld.c
kernel/watchdog_hld.c: hard lockup message should end with a newline
2019-04-19 09:46:05 -07:00
workqueue.c
workqueue: Remove GPF argument from alloc_workqueue_attrs()
2019-06-27 14:12:19 -07:00
workqueue_internal.h
sched/core, workqueues: Distangle worker accounting from rq lock
2019-04-16 16:55:15 +02:00