Ben Greear
34376a50fb
Fix lockup related to stop_machine being stuck in __do_softirq.
...
The stop machine logic can lock up if all but one of the migration
threads make it through the disable-irq step and the one remaining
thread gets stuck in __do_softirq. The reason __do_softirq can hang is
that it has a bail-out based on jiffies timeout, but in the lockup case,
jiffies itself is not incremented.
To work around this, re-add the max_restart counter in __do_irq and stop
processing irqs after 10 restarts.
Thanks to Tejun Heo and Rusty Russell and others for helping me track
this down.
This was introduced in 3.9 by commit c10d73671a ("softirq: reduce
latencies").
It may be worth looking into ath9k to see if it has issues with its irq
handler at a later date.
The hang stack traces look something like this:
------------[ cut here ]------------
WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9c/0xa7()
Watchdog detected hard LOCKUP on cpu 2
Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
Pid: 23, comm: migration/2 Tainted: G C 3.9.4+ #11
Call Trace:
<NMI> warn_slowpath_common+0x85/0x9f
warn_slowpath_fmt+0x46/0x48
watchdog_overflow_callback+0x9c/0xa7
__perf_event_overflow+0x137/0x1cb
perf_event_overflow+0x14/0x16
intel_pmu_handle_irq+0x2dc/0x359
perf_event_nmi_handler+0x19/0x1b
nmi_handle+0x7f/0xc2
do_nmi+0xbc/0x304
end_repeat_nmi+0x1e/0x2e
<<EOE>>
cpu_stopper_thread+0xae/0x162
smpboot_thread_fn+0x258/0x260
kthread+0xc7/0xcf
ret_from_fork+0x7c/0xb0
---[ end trace 4947dfa9b0a4cec3 ]---
BUG: soft lockup - CPU#1 stuck for 22s! [migration/1:17]
Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
irq event stamp: 835637905
hardirqs last enabled at (835637904): __do_softirq+0x9f/0x257
hardirqs last disabled at (835637905): apic_timer_interrupt+0x6d/0x80
softirqs last enabled at (5654720): __do_softirq+0x1ff/0x257
softirqs last disabled at (5654725): irq_exit+0x5f/0xbb
CPU 1
Pid: 17, comm: migration/1 Tainted: G WC 3.9.4+ #11 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: tasklet_hi_action+0xf0/0xf0
Process migration/1
Call Trace:
<IRQ>
__do_softirq+0x117/0x257
irq_exit+0x5f/0xbb
smp_apic_timer_interrupt+0x8a/0x98
apic_timer_interrupt+0x72/0x80
<EOI>
printk+0x4d/0x4f
stop_machine_cpu_stop+0x22c/0x274
cpu_stopper_thread+0xae/0x162
smpboot_thread_fn+0x258/0x260
kthread+0xc7/0xcf
ret_from_fork+0x7c/0xb0
Signed-off-by: Ben Greear <greearb@candelatech.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Pekka Riikonen <priikone@iki.fi>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: stable@kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-06-10 17:46:57 -07:00
..
cpu
rcu/idle: Wrap cpu-idle poll mode within rcu_idle_enter/exit
2013-05-14 17:43:29 +02:00
debug
kgdb/sysrq: fix inconstistent help message of sysrq key
2013-04-30 17:04:10 -07:00
events
perf: Factor out auxiliary events notification
2013-05-07 13:17:29 +02:00
gcov
kernel/gcov: remove depends on CONFIG_EXPERIMENTAL
2013-01-11 11:39:33 -08:00
irq
irqdomain: document the simple domain first_irq
2013-06-08 21:15:09 +01:00
power
Merge branch 'drm-next' of git://people.freedesktop.org/~airlied/linux
2013-05-02 19:40:34 -07:00
sched
Merge branch 'timers-nohz-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-05-05 13:23:27 -07:00
time
Merge branch 'fortglx/3.10/time' of git://git.linaro.org/people/jstultz/linux into timers/urgent
2013-05-29 09:55:01 +02:00
trace
tracing: Use current_uid() for critical time tracing
2013-06-06 12:35:30 -04:00
.gitignore
kernel/hz.bc: ignore.
2013-04-22 07:09:06 -07:00
acct.c
fs: Fix hang with BSD accounting on frozen filesystem
2013-05-04 14:57:58 -04:00
async.c
async: rename and redefine async_func_ptr
2013-03-12 13:59:14 -07:00
audit.c
Merge git://git.infradead.org/users/eparis/audit
2013-05-11 14:29:11 -07:00
audit.h
Merge git://git.infradead.org/users/eparis/audit
2013-05-11 14:29:11 -07:00
audit_tree.c
kernel/audit_tree.c: tree will leak memory when failure occurs in audit_trim_trees()
2013-04-29 15:54:26 -07:00
audit_watch.c
audit: catch possible NULL audit buffers
2013-01-11 14:54:55 -08:00
auditfilter.c
auditfilter.c: fix kernel-doc warnings
2013-05-24 16:22:52 -07:00
auditsc.c
audit: Make testing for a valid loginuid explicit.
2013-05-07 22:27:15 -04:00
backtracetest.c
bounds.c
capability.c
Add file_ns_capable() helper function for open-time capability checking
2013-04-14 10:06:31 -07:00
cgroup.c
cgroup: warn about mismatching options of a new mount of an existing hierarchy
2013-05-29 07:59:39 +09:00
cgroup_freezer.c
cgroup: rename ->create/post_create/pre_destroy/destroy() to ->css_alloc/online/offline/free()
2012-11-19 08:13:38 -08:00
compat.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal
2013-05-01 07:21:43 -07:00
configs.c
proc: Supply PDE attribute setting accessor functions
2013-05-01 17:29:18 -04:00
context_tracking.c
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 18:19:48 -08:00
cpu.c
Merge branch 'smp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 19:04:55 -08:00
cpu_pm.c
cpuset.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-05-01 17:51:54 -07:00
crash_dump.c
cred.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
2012-12-18 10:55:28 -08:00
delayacct.c
cputime: Use accessors to read task cputime stats
2013-01-27 19:23:31 +01:00
dma.c
elfcore.c
exec_domain.c
exit.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-05-01 17:51:54 -07:00
extable.c
extable: Flip the sorting message
2013-04-15 13:25:16 +02:00
fork.c
Merge branch 'for-3.10/drivers' of git://git.kernel.dk/linux-block
2013-05-08 11:51:05 -07:00
freezer.c
freezer: change ptrace_stop/do_signal_stop to use freezable_schedule()
2012-10-26 14:27:49 -07:00
futex.c
futex: fix kernel-doc notation and spello
2013-03-12 20:42:10 -07:00
futex_compat.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal
2013-02-23 18:50:11 -08:00
groups.c
hrtimer.c
Merge branch 'timers-nohz-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-05-05 13:23:27 -07:00
hung_task.c
irq_work.c
Merge branch 'nohz/printk-v8' into irq/core
2013-02-05 00:48:46 +01:00
itimer.c
jump_label.c
jump_label: Export jump_label_rate_limit()
2012-08-06 19:00:35 +03:00
kallsyms.c
kernel: kallsyms: memory override issue, need check destination buffer length
2013-04-15 15:17:26 +09:30
kcmp.c
kcmp: include linux/ptrace.h
2012-12-20 17:40:19 -08:00
Kconfig.freezer
Kconfig.hz
Kconfig.locks
locking: Adjust spin lock inlining Kconfig options
2012-09-13 17:56:13 +02:00
Kconfig.preempt
kexec.c
kexec: Use min() and min_t() to simplify logic
2013-04-30 17:04:07 -07:00
kmod.c
usermodehelper: check subprocess_info->path != NULL
2013-05-16 12:01:11 -07:00
kprobes.c
kprobes: Fix a double lock bug of kprobe_mutex
2013-04-18 08:58:38 -07:00
ksysfs.c
Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2012-12-11 18:10:49 -08:00
kthread.c
kthread: implement probe_kthread_data()
2013-04-30 17:04:02 -07:00
latencytop.c
lglock.c
lockdep.c
Merge branch 'for-3.10/drivers' of git://git.kernel.dk/linux-block
2013-05-08 11:51:05 -07:00
lockdep_internals.h
lockdep_proc.c
lockdep: Use KSYM_NAME_LEN'ed buffer for __get_key_name()
2012-10-24 12:39:09 +02:00
lockdep_states.h
Makefile
We get rid of the general module prefix confusion with a binary config option,
2013-05-05 10:58:06 -07:00
modsign_certificate.S
CONFIG_SYMBOL_PREFIX: cleanup.
2013-03-15 15:09:43 +10:30
modsign_pubkey.c
keys: use keyring_alloc() to create module signing keyring
2012-12-20 17:40:21 -08:00
module-internal.h
MODSIGN: Move the magic string to the end of a module and eliminate the search
2012-10-19 17:30:40 -07:00
module.c
kmemleak: No need for scanning specific module sections
2013-05-17 09:53:36 +01:00
module_signing.c
MODSIGN: Don't use enum-type bitfields in module signature info block
2012-12-05 11:27:24 +10:30
mutex-debug.c
mutex-debug.h
mutex.c
mutex: Back out architecture specific check for negative mutex count
2013-04-19 09:33:36 +02:00
mutex.h
notifier.c
nsproxy.c
proc: Split the namespace stuff out into linux/proc_ns.h
2013-05-01 17:29:39 -04:00
padata.c
padata: use __this_cpu_read per-cpu helper
2012-12-06 17:16:23 +08:00
panic.c
dump_stack: implement arch-specific hardware description in task dumps
2013-04-30 17:04:02 -07:00
params.c
params: Fix potential memory leak in add_sysfs_param()
2013-03-18 11:40:21 +00:00
pid.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-05-01 17:51:54 -07:00
pid_namespace.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-05-01 17:51:54 -07:00
posix-cpu-timers.c
posix_timers: Fix pre-condition to stop the tick on full dynticks
2013-04-22 19:59:25 +02:00
posix-timers.c
posix-timers: Remove unused variable
2013-04-18 12:51:19 +02:00
printk.c
aio: don't include aio.h in sched.h
2013-05-07 20:16:25 -07:00
profile.c
proc: Supply PDE attribute setting accessor functions
2013-05-01 17:29:18 -04:00
ptrace.c
aio: don't include aio.h in sched.h
2013-05-07 20:16:25 -07:00
range.c
x86, range: fix missing merge during add range
2013-05-17 11:49:10 -07:00
rcu.h
rcu: Provide RCU CPU stall warnings for tiny RCU
2013-01-28 22:06:21 -08:00
rcupdate.c
Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD
2013-01-28 22:25:21 -08:00
rcutiny.c
Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD
2013-01-28 22:25:21 -08:00
rcutiny_plugin.h
rcu: Provide RCU CPU stall warnings for tiny RCU
2013-01-28 22:06:21 -08:00
rcutorture.c
rcu: Allow rcutorture to be built at low optimization levels
2013-02-04 12:18:20 -08:00
rcutree.c
Merge commit ' 8700c95adb' into timers/nohz
2013-05-02 17:54:19 +02:00
rcutree.h
Merge commit ' 8700c95adb' into timers/nohz
2013-05-02 17:54:19 +02:00
rcutree_plugin.h
rcu: Don't allocate bootmem from rcu_init()
2013-05-15 10:41:12 -07:00
rcutree_trace.c
rcutrace: single_open() leaks
2013-05-05 00:16:35 -04:00
relay.c
Merge branch 'for-3.10/core' of git://git.kernel.dk/linux-block
2013-05-08 10:13:35 -07:00
res_counter.c
res_counter: return amount of charges after res_counter_uncharge()
2012-12-18 15:02:12 -08:00
resource.c
mem hotunplug: fix kfree() of bootmem memory
2013-04-29 15:54:40 -07:00
rtmutex-debug.c
sched/rt: Move rt specific bits into new header file
2013-02-07 20:51:08 +01:00
rtmutex-debug.h
rtmutex-tester.c
locking/rtmutex/tester: Set correct permissions on sysfs files
2013-04-10 14:48:37 +02:00
rtmutex.c
sched/rt: Move rt specific bits into new header file
2013-02-07 20:51:08 +01:00
rtmutex.h
rtmutex_common.h
rwsem.c
Revert "rw_semaphore: remove up/down_read_non_owner"
2013-03-23 15:53:52 -07:00
seccomp.c
seccomp: allow BPF_XOR based ALU instructions.
2013-03-26 11:07:19 +11:00
semaphore.c
semaphore: use `bool' type for semaphore_waiter's up
2013-04-30 17:04:08 -07:00
signal.c
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
2013-05-01 14:08:52 -07:00
smp.c
kernel/smp.c: cleanups
2013-04-30 17:04:03 -07:00
smpboot.c
kthread: Prevent unpark race which puts threads on the wrong cpu
2013-04-12 14:18:43 +02:00
smpboot.h
smpboot: Provide infrastructure for percpu hotplug threads
2012-08-13 17:01:07 +02:00
softirq.c
Fix lockup related to stop_machine being stuck in __do_softirq.
2013-06-10 17:46:57 -07:00
spinlock.c
srcu.c
srcu: use ACCESS_ONCE() to access sp->completed in srcu_read_lock()
2013-02-07 15:19:36 -08:00
stacktrace.c
stop_machine.c
stop_machine: Mark per cpu stopper enabled early
2013-02-26 22:25:17 +01:00
sys.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal
2013-05-01 07:21:43 -07:00
sys_ni.c
unify compat fanotify_mark(2), switch to COMPAT_SYSCALL_DEFINE
2013-05-09 13:46:38 -04:00
sysctl.c
mm: replace hardcoded 3% with admin_reserve_pages knob
2013-04-29 15:54:36 -07:00
sysctl_binary.c
switch compat_sys_sysctl to COMPAT_SYSCALL_DEFINE
2013-05-09 14:53:20 -04:00
task_work.c
task_work: task_work_add() should not succeed after exit_task_work()
2012-09-13 16:47:34 +02:00
taskstats.c
taskstats: cgroupstats_user_cmd() may leak on error
2012-10-06 03:05:31 +09:00
test_kprobes.c
kernel/: rename random32() to prandom_u32()
2013-04-29 18:28:42 -07:00
time.c
timekeeping: Use inject_offset in warp_clock
2013-03-15 16:50:20 -07:00
timeconst.bc
kernel: Replace timeconst.pl with a bc script
2013-02-16 23:17:25 +01:00
timer.c
Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-05-15 14:05:17 -07:00
tracepoint.c
Tracing updates for Linux 3.10
2013-04-29 13:55:38 -07:00
tsacct.c
cputime: Use accessors to read task cputime stats
2013-01-27 19:23:31 +01:00
uid16.c
make SYSCALL_DEFINE<n>-generated wrappers do asmlinkage_protect
2013-03-03 22:58:33 -05:00
up.c
user-return-notifier.c
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
user.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-05-01 17:51:54 -07:00
user_namespace.c
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-05-01 17:51:54 -07:00
utsname.c
proc: Split the namespace stuff out into linux/proc_ns.h
2013-05-01 17:29:39 -04:00
utsname_sysctl.c
kernel/utsname_sysctl.c: put get/get_uts() into CONFIG_PROC_SYSCTL code block
2013-02-27 19:10:22 -08:00
wait.c
propagate name change to comments in kernel source
2012-12-06 10:39:54 +01:00
watchdog.c
watchdog: Add comments to explain the watchdog_disabled variable
2013-03-14 08:24:05 +01:00
workqueue.c
workqueue: don't perform NUMA-aware allocations on offline nodes in wq_numa_init()
2013-05-15 14:24:24 -07:00
workqueue_internal.h
workqueue: include workqueue info when printing debug dump of a worker task
2013-04-30 17:04:02 -07:00