Gregory Haskins
07903af152
sched: Fix race in cpupri introduced by cpumask_var changes
...
Background:
Several race conditions in the scheduler have cropped up
recently, which Steven and I have tracked down using ftrace.
The most recent one turns out to be a race in how the scheduler
determines a suitable migration target for RT tasks, introduced
recently with commit:
commit 68e74568fb
Date: Tue Nov 25 02:35:13 2008 +1030
sched: convert struct cpupri_vec cpumask_var_t.
The original design of cpupri allowed lockless readers to
quickly determine a best-estimate target. Races between the
pri_active bitmap and the vec->mask were handled in the
original code because we would detect and return "0" when this
occured. The design was predicated on the *effective*
atomicity (*) of caching the result of cpus_and() between the
cpus_allowed and the vec->mask.
Commit 68e74568 changed the behavior such that vec->mask is
accessed multiple times. This introduces a subtle race, the
result of which means we can have a result that returns "1",
but with an empty bitmap.
*) yes, we know cpus_and() is not a locked operator across the
entire composite array, but it is implicitly atomic on a
per-word basis which is all the design required to work.
Implementation:
Rather than forgoing the lockless design, or reverting to a
stack-based cpumask_t, we simply check for when the race has
been encountered and continue processing in the event that the
race is hit. This renders the removal race as if the priority
bit had been atomically cleared as well, and allows the
algorithm to execute correctly.
Signed-off-by: Gregory Haskins <ghaskins@novell.com>
CC: Rusty Russell <rusty@rustcorp.com.au>
CC: Steven Rostedt <srostedt@redhat.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <20090730145728.25226.92769.stgit@dev.haskins.net>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-08-02 14:23:29 +02:00
..
gcov
gcov: enable GCOV_PROFILE_ALL for x86_64
2009-06-18 13:03:58 -07:00
irq
genirq: Fix UP compile failure caused by irq_thread_check_affinity
2009-07-22 23:18:46 +02:00
power
headers: smp_lock.h redux
2009-07-12 12:22:34 -07:00
time
clocksource: Prevent NULL pointer dereference
2009-07-19 17:15:54 +02:00
trace
tracing/stat: Fix seqfile memory leak
2009-07-23 09:53:55 -04:00
.gitignore
acct.c
bsdacct: fix access to invalid filp in acct_on()
2009-06-30 18:56:00 -07:00
async.c
async: Fix lack of boot-time console due to insufficient synchronization
2009-06-08 12:31:53 -07:00
audit.c
Fix rule eviction order for AUDIT_DIR
2009-06-24 00:02:38 -04:00
audit.h
Fix rule eviction order for AUDIT_DIR
2009-06-24 00:02:38 -04:00
audit_tree.c
Fix rule eviction order for AUDIT_DIR
2009-06-24 00:02:38 -04:00
audit_watch.c
Audit: clean up all op= output to include string quoting
2009-06-24 00:00:52 -04:00
auditfilter.c
Audit: clean up all op= output to include string quoting
2009-06-24 00:00:52 -04:00
auditsc.c
Fix rule eviction order for AUDIT_DIR
2009-06-24 00:02:38 -04:00
backtracetest.c
bounds.c
capability.c
cgroup.c
cgroup avoid permanent sleep at rmdir
2009-07-29 19:10:35 -07:00
cgroup_debug.c
debug cgroup: remove unneeded cgroup_lock
2009-04-02 19:04:54 -07:00
cgroup_freezer.c
compat.c
signals: implement sys_rt_tgsigqueueinfo
2009-04-30 19:24:24 +02:00
configs.c
cpu.c
mm/init: cpu_hotplug_init() must be initialized before SLAB
2009-06-22 21:18:12 -07:00
cpuset.c
cpuset,mm: update tasks' mems_allowed in time
2009-06-16 19:47:31 -07:00
cred-internals.h
cred.c
CRED: Rename cred_exec_mutex to reflect that it's a guard against ptrace
2009-05-11 08:15:36 +10:00
delayacct.c
dma-coherent.c
dma.c
exec_domain.c
Get rid of indirect include of fs_struct.h
2009-03-31 23:00:27 -04:00
exit.c
headers: mnt_namespace.h redux
2009-07-08 09:31:56 -07:00
extable.c
Merge branch 'tracing-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-04-05 11:04:19 -07:00
fork.c
mm: copy over oom_adj value at fork time
2009-07-29 19:10:34 -07:00
freezer.c
sched: fix nr_uninterruptible accounting of frozen tasks really
2009-07-18 14:19:53 +02:00
futex.c
futexes: Fix infinite loop in get_futex_key() on huge page
2009-07-11 12:40:44 +02:00
futex_compat.c
groups.c
groups: move code to kernel/groups.c
2009-06-16 19:47:48 -07:00
hrtimer.c
hrtimer: Fix migration expiry check
2009-07-10 17:32:55 +02:00
hung_task.c
softlockup: ensure the task has been switched out once
2009-02-11 11:04:16 +01:00
itimer.c
timers: split process wide cpu clocks/timers
2009-02-05 13:04:33 +01:00
kallsyms.c
kernel/kallsyms.c: replace deprecated __initcall with device_initcall and fix whitespace
2009-06-09 22:37:52 +02:00
Kconfig.freezer
Kconfig.hz
Kconfig.preempt
kexec.c
kexec: fix omitting offset in extended crashkernel syntax
2009-07-29 19:10:34 -07:00
kfifo.c
kernel/kfifo.c: replace conditional test with is_power_of_2()
2009-06-16 19:47:47 -07:00
kgdb.c
sysrq, intel_fb: fix sysrq g collision
2009-05-15 07:56:24 -05:00
kmod.c
headers: mnt_namespace.h redux
2009-07-08 09:31:56 -07:00
kprobes.c
kprobes: Use kernel_text_address() for checking probe address
2009-07-30 16:44:06 -07:00
ksysfs.c
kthread.c
update the comment in kthread_stop()
2009-07-27 12:15:46 -07:00
latencytop.c
sched, latencytop: incorporate review feedback from Andrew Morton
2009-02-11 10:18:04 +01:00
lockdep.c
Merge branch 'linus' into tracing/core
2009-05-07 11:17:34 +02:00
lockdep_internals.h
lockdep: increase MAX_LOCKDEP_ENTRIES and MAX_LOCKDEP_CHAINS
2009-05-12 19:59:52 +02:00
lockdep_proc.c
lockstat: warn about disabled lock debugging
2009-02-14 23:28:28 +01:00
lockdep_states.h
lockdep: move state bit definitions around
2009-02-14 23:27:59 +01:00
Makefile
Merge branch 'tracing-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-06-28 11:05:04 -07:00
marker.c
module.c
module: use MODULE_SYMBOL_PREFIX with module_layout
2009-07-27 12:15:45 -07:00
mutex-debug.c
mutex-debug.h
mutex.c
Merge branch 'linus' into perfcounters/core
2009-06-11 17:55:42 +02:00
mutex.h
notifier.c
ns_cgroup.c
cgroups: relax ns_can_attach checks to allow attaching to grandchild cgroups
2009-04-02 19:04:53 -07:00
nsproxy.c
nsproxy: extract create_nsproxy()
2009-06-18 13:03:56 -07:00
panic.c
Fix caller information for warn_slowpath_null
2009-05-16 13:41:28 -07:00
params.c
module_param: allow 'bool' module_params to be bool, not just int.
2009-06-12 21:46:58 +09:30
perf_counter.c
Merge branch 'perf-counters-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/peterz/linux-2.6-perf
2009-07-22 11:41:56 -07:00
pid.c
kmemleak: Remove alloc_bootmem annotations introduced in the past
2009-07-09 17:07:02 +01:00
pid_namespace.c
pidns: rewrite copy_pid_ns()
2009-06-18 13:03:55 -07:00
pm_qos_params.c
posix-cpu-timers.c
kernel/posix-cpu-timers.c: fix sparse warning
2009-04-30 08:08:31 +02:00
posix-timers.c
printk.c
printk: Add KERN_DEFAULT printk log-level
2009-06-16 11:02:28 -07:00
profile.c
profile: suppress warning about large allocations when profile=1 is specified
2009-07-29 19:10:36 -07:00
ptrace.c
cred_guard_mutex: do not return -EINTR to user-space
2009-07-06 13:57:04 -07:00
rcuclassic.c
kmemtrace, rcu: fix linux/rcutree.h and linux/rcuclassic.h dependencies
2009-04-03 12:23:02 +02:00
rcupdate.c
RCU: Don't try and predeclare inline funcs as it upsets some versions of gcc
2009-04-15 13:55:14 -07:00
rcupreempt.c
rcu: rcu_sched_grace_period(): kill the bogus flush_signals()
2009-05-05 20:28:05 +02:00
rcupreempt_trace.c
rcutorture.c
cpumask: convert rcutorture.c
2009-03-30 22:05:16 +10:30
rcutree.c
rcu: Mark Hierarchical RCU no longer experimental
2009-06-24 15:02:48 +02:00
rcutree.h
kmemtrace, rcu: fix rcu_tree_trace.c data structure dependencies
2009-04-03 12:23:03 +02:00
rcutree_trace.c
rcu: Add __rcu_pending tracing to hierarchical RCU
2009-04-14 11:33:43 +02:00
relay.c
Merge branch 'tracing-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-04-05 11:04:19 -07:00
res_counter.c
memcg: add interface to reset limits
2009-06-18 13:03:48 -07:00
resource.c
kernel/resource.c: fix sign extension in reserve_setup()
2009-06-30 18:56:00 -07:00
rtmutex-debug.c
rtmutex-debug.h
rtmutex-tester.c
rtmutex.c
trivial: fix ETIMEOUT -> ETIMEDOUT typos
2009-06-12 18:01:50 +02:00
rtmutex.h
rtmutex_common.h
rt_mutex: add proxy lock routines
2009-04-06 11:14:02 +02:00
rwsem.c
sched.c
sched: fix load average accounting vs. cpu hotplug
2009-07-18 14:19:52 +02:00
sched_clock.c
sched: Fix fallback sched_clock()'s offset when using jiffies
2009-05-09 10:08:19 +02:00
sched_cpupri.c
sched: Fix race in cpupri introduced by cpumask_var changes
2009-08-02 14:23:29 +02:00
sched_cpupri.h
cpumask: remove cpumask_t from core
2009-03-30 22:05:17 +10:30
sched_debug.c
sched: Hide runqueues from direct refer at source code level
2009-06-17 18:29:42 +02:00
sched_fair.c
sched: Fix latencytop and sleep profiling vs group scheduling
2009-08-02 14:10:12 +02:00
sched_features.h
Merge branch 'locking-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-03-30 17:17:35 -07:00
sched_idletask.c
sched, timers: move calc_load() to scheduler
2009-05-15 15:32:45 +02:00
sched_rt.c
sched_rt: Fix overload bug on rt group scheduling
2009-07-10 10:43:29 +02:00
sched_stats.h
sched: remove unused fields from struct rq
2009-03-24 23:16:51 +01:00
seccomp.c
x86-64: seccomp: fix 32/64 syscall hole
2009-03-02 15:41:30 -08:00
semaphore.c
signal.c
ptrace: do_notify_parent_cldstop: fix the wrong ->nsproxy usage
2009-06-18 13:03:52 -07:00
slow-work.c
slow-work: use round_jiffies() for thread pool's cull and OOM timers
2009-06-16 19:47:49 -07:00
smp.c
cpumask: alloc zeroed cpumask for static cpumask_var_ts
2009-06-09 22:30:27 +09:30
softirq.c
softirq: introduce tasklet_hrtimer infrastructure
2009-07-22 17:01:17 +02:00
softlockup.c
spinlock.c
Allow rwlocks to re-enable interrupts
2009-04-02 19:05:11 -07:00
srcu.c
stacktrace.c
stop_machine.c
cpumask: remove cpumask_t from core
2009-03-30 22:05:17 +10:30
sys.c
groups: move code to kernel/groups.c
2009-06-16 19:47:48 -07:00
sys_ni.c
sysctl.c
Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-06-28 11:05:28 -07:00
sysctl_check.c
net: add ARP notify option for devices
2009-02-01 01:04:33 -08:00
taskstats.c
test_kprobes.c
time.c
timeconst.pl
timer.c
timer: Avoid reading uninitialized data
2009-07-18 23:11:43 +02:00
tracepoint.c
tracepoints: dont update zero-sized tracepoint sections
2009-03-18 19:55:00 +01:00
tsacct.c
Fix fixpoint divide exception in acct_update_integrals
2009-03-09 08:13:35 -07:00
uid16.c
up.c
user.c
sched: delayed cleanup of user_struct
2009-06-15 21:30:23 -07:00
user_namespace.c
Fix recursive lock in free_uid()/free_user_ns()
2009-02-27 16:26:21 -08:00
utsname.c
utsns: extract creeate_uts_ns()
2009-06-18 13:03:55 -07:00
utsname_sysctl.c
proc_sysctl: use CONFIG_PROC_SYSCTL around ipc and utsname proc_handlers
2009-04-02 19:05:01 -07:00
wait.c
wait: don't use __wake_up_common()
2009-04-14 17:17:16 +02:00
workqueue.c
ftrace, workqueuetrace: make workqueue tracepoints use TRACE_EVENT macro
2009-06-02 01:10:40 +02:00