Shaohua Li
09223371de
rcu: Use softirq to address performance regression ...
Commit a26ac2455ffcf3(rcu: move TREE_RCU from softirq to kthread)
introduced performance regression. In an AIM7 test, this commit degraded
performance by about 40%.
The commit runs rcu callbacks in a kthread instead of softirq. We observed
high rate of context switch which is caused by this. Out test system has
64 CPUs and HZ is 1000, so we saw more than 64k context switch per second
which is caused by RCU's per-CPU kthread. A trace showed that most of
the time the RCU per-CPU kthread doesn't actually handle any callbacks,
but instead just does a very small amount of work handling grace periods.
This means that RCU's per-CPU kthreads are making the scheduler do quite
a bit of work in order to allow a very small amount of RCU-related
processing to be done.
Alex Shi's analysis determined that this slowdown is due to lock
contention within the scheduler. Unfortunately, as Peter Zijlstra points
out, the scheduler's real-time semantics require global action, which
means that this contention is inherent in real-time scheduling. (Yes,
perhaps someone will come up with a workaround -- otherwise, -rt is not
going to do well on large SMP systems -- but this patch will work around
this issue in the meantime. And "the meantime" might well be forever.)
This patch therefore re-introduces softirq processing to RCU, but only
for core RCU work. RCU callbacks are still executed in kthread context,
so that only a small amount of RCU work runs in softirq context in the
common case. This should minimize ksoftirqd execution, allowing us to
skip boosting of ksoftirqd for CONFIG_RCU_BOOST=y kernels.
Signed-off-by: Shaohua Li <shaohua.li@intel.com>
Tested-by: "Alex,Shi" <alex.shi@intel.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
2011-06-14 15:25:39 -07:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
perf: Fix SIGIO handling
2011-05-28 17:04:59 +02:00
Merge branch 'trivial' of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild-2.6
2011-03-20 18:14:55 -07:00
irq: Remove smp_affinity_list when unregister irq proc
2011-05-26 13:15:28 +02:00
PM / Hibernate: Update kerneldoc comments in hibernate.c
2011-05-24 23:36:06 +02:00
hrtimers: Avoid touching inactive timer bases
2011-05-23 13:59:54 +02:00
ftrace: Add internal recursive checks
2011-05-25 22:13:49 -04:00
netlink: kill loginuid/sessionid/sid members from struct netlink_skb_parms
2011-03-03 10:55:40 -08:00
audit: make functions static
2010-10-30 01:42:19 -04:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
kill path_lookup()
2011-03-14 09:15:23 -04:00
netlink: kill loginuid/sessionid/sid members from struct netlink_skb_parms
2011-03-03 10:55:40 -08:00
audit: acquire creds selectively to reduce atomic op overhead
2011-04-27 15:11:03 +02:00
memcg: remove direct page_cgroup-to-page pointer
2011-03-23 19:46:28 -07:00
Merge branch 'master' into next
2011-05-19 18:51:57 +10:00
cgroup: remove the ns_cgroup
2011-05-26 17:12:34 -07:00
cgroups: add per-thread subsystem callbacks
2011-05-26 17:12:34 -07:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
2011-05-25 15:35:32 -07:00
llseek: automatically add .llseek fop
2010-10-15 15:53:27 +02:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
cpuset: Fix cpuset_cpus_allowed_fallback(), don't update tsk->rt.nr_cpus_allowed
2011-05-28 17:02:57 +02:00
crash_dump: export is_kdump_kernel to modules, consolidate elfcorehdr_addr, setup_elfcorehdr and saved_max_pfn
2011-03-23 19:47:19 -07:00
Merge branch 'docs-move' of git://git.kernel.org/pub/scm/linux/kernel/git/rdunlap/linux-docs
2011-05-27 10:25:02 -07:00
Merge branch 'ptrace' of git://git.kernel.org/pub/scm/linux/kernel/git/oleg/misc
2011-05-20 13:33:21 -07:00
extable, core_kernel_data(): Make sure all archs define _sdata
2011-05-20 08:56:56 +02:00
mm: Fix boot crash in mm_alloc()
2011-05-29 11:32:28 -07:00
Freezer: Use SMP barriers
2011-05-17 23:19:17 +02:00
futex: Set FLAGS_HAS_TIMEOUT during futex_wait restart setup
2011-04-15 16:34:32 +02:00
userns: user namespaces: convert several capable() calls
2011-03-23 19:47:08 -07:00
userns: user namespaces: convert several capable() calls
2011-03-23 19:47:08 -07:00
hrtimers: Fix typo causing erratic timers
2011-05-25 15:31:58 -07:00
watchdog, hung_task_timeout: Add Kconfig configurable default
2011-04-28 09:13:17 +02:00
irq_work: Use per cpu atomics instead of regular atomics
2010-12-18 15:54:48 +01:00
jump_label: Check entries limit in __jump_label_update
2011-05-25 19:56:36 -04:00
Merge branch 'core-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2011-03-25 17:52:22 -07:00
arch:Kconfig.locks Remove unused config option.
2011-04-10 17:01:05 +02:00
PM: Remove sysdev suspend, resume and shutdown operations
2011-05-11 21:37:15 +02:00
kfifo: fix scatterlist usage
2010-10-01 10:50:58 -07:00
Merge branch 'next' into for-linus
2011-05-24 22:55:24 +10:00
Merge branch 'for-2.6.38' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu
2011-01-07 17:02:58 -08:00
kernel/ksysfs.c: expose file_caps_enabled in sysfs
2011-04-19 16:45:51 -07:00
cpuset: Fix cpuset_cpus_allowed_fallback(), don't update tsk->rt.nr_cpus_allowed
2011-05-28 17:02:57 +02:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
lockdep: Remove cmpxchg to update nr_chain_hlocks
2011-04-22 11:06:59 +02:00
lockdep: Remove unused 'factor' variable from lockdep_stats_show()
2011-03-23 13:54:47 +01:00
cgroup: remove the ns_cgroup
2011-05-26 17:12:34 -07:00
Merge branch 'staging-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging-2.6
2011-05-23 12:49:28 -07:00
mutex: Use p->on_cpu for the adaptive spin
2011-04-14 08:52:33 +02:00
mutex: Use p->on_cpu for the adaptive spin
2011-04-14 08:52:33 +02:00
lockdep, mutex: provide mutex_lock_nest_lock
2011-05-25 08:39:17 -07:00
mutex: Use p->on_cpu for the adaptive spin
2011-04-14 08:52:33 +02:00
cgroup: remove the ns_cgroup
2011-05-26 17:12:34 -07:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
move x86 specific oops=panic to generic code
2011-03-22 17:44:11 -07:00
params.c: Use new strtobool function to process boolean inputs
2011-05-19 16:55:28 +09:30
next_pidmap: fix overflow condition
2011-04-18 10:35:30 -07:00
pidns: call pid_ns_prepare_proc() from create_pid_namespace()
2011-03-23 19:46:58 -07:00
Merge branch 'idle-release' of git://git.kernel.org/pub/scm/linux/kernel/git/lenb/linux-idle-2.6
2011-05-29 11:18:09 -07:00
hrtimers: Avoid touching inactive timer bases
2011-05-23 13:59:54 +02:00
posix-timers: RCU conversion
2011-05-24 12:10:51 +02:00
printk: allocate kernel log buffer earlier
2011-05-25 08:39:48 -07:00
kernel/profile.c: remove some duplicate code from profile_hits()
2011-05-26 17:12:37 -07:00
ptrace: ptrace_resume() shouldn't wake up !TASK_TRACED thread
2011-05-25 19:20:21 +02:00
kernel/range.c: fix clean_sort_range() for the case of full array
2010-11-12 07:55:31 -08:00
rcu: Use WARN_ON_ONCE for DEBUG_OBJECTS_RCU_HEAD warnings
2011-05-05 23:16:57 -07:00
sanitize <linux/prefetch.h> usage
2011-05-20 12:50:29 -07:00
rcu: Converge TINY_RCU expedited and normal boosting
2011-05-05 23:16:58 -07:00
rcu: mark rcutorture boosting callback as being on-stack
2011-05-05 23:16:57 -07:00
rcu: Use softirq to address performance regression
2011-06-14 15:25:39 -07:00
rcu: Use softirq to address performance regression
2011-06-14 15:25:39 -07:00
rcu: Use softirq to address performance regression
2011-06-14 15:25:39 -07:00
rcu: Decrease memory-barrier usage based on semi-formal proof
2011-05-26 09:42:23 -07:00
Clean up relay_alloc_page_array() slightly by using vzalloc rather than vmalloc and memset
2010-11-05 08:21:34 -07:00
memcg: res_counter_read_u64(): fix potential races on 32-bit machines
2011-03-23 19:46:22 -07:00
resources: add arch hook for preventing allocation in reserved areas
2010-12-17 10:01:09 -08:00
rtmutex: Simplify PI algorithm and make highest prio task get lock
2011-01-27 21:13:51 -05:00
rtmutex: tester: Remove the remaining BKL leftovers
2011-02-22 22:07:22 +01:00
rtmutex: Simplify PI algorithm and make highest prio task get lock
2011-01-27 21:13:51 -05:00
rtmutex: Simplify PI algorithm and make highest prio task get lock
2011-01-27 21:13:51 -05:00
cpuset: Fix cpuset_cpus_allowed_fallback(), don't update tsk->rt.nr_cpus_allowed
2011-05-28 17:02:57 +02:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
sched, autogroup: Stop going ahead if autogroup is disabled
2011-02-23 11:33:59 +01:00
sched: Add some clock info to sched_debug
2010-11-23 10:29:08 +01:00
sched: Get rid of lock_depth
2011-04-24 13:18:38 +02:00
sched: Fix ->min_vruntime calculation in dequeue_entity()
2011-05-28 17:02:56 +02:00
sched: Move the second half of ttwu() to the remote cpu
2011-04-14 08:52:41 +02:00
sched: Drop the rq argument to sched_class::select_task_rq()
2011-04-14 08:52:36 +02:00
sched: More sched_domain iterations fixes
2011-05-28 17:02:54 +02:00
sched: More sched_domain iterations fixes
2011-05-28 17:02:54 +02:00
sched: Drop the rq argument to sched_class::select_task_rq()
2011-04-14 08:52:36 +02:00
signal: sys_pause() should check signal_pending()
2011-05-25 19:22:27 +02:00
smp: move smp setup functions to kernel/smp.c
2011-03-22 17:44:11 -07:00
rcu: Use softirq to address performance regression
2011-06-14 15:25:39 -07:00
rcu: demote SRCU_SYNCHRONIZE_DELAY from kernel-parameter status
2011-01-14 04:56:49 -08:00
kthread: use kthread_create_on_node()
2011-03-22 17:44:01 -07:00
Merge branch 'driver-core-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core-2.6
2011-05-19 18:24:11 -07:00
ipc: Add missing sys_ni entries for ipc/compat.c functions
2011-05-20 13:53:02 -07:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
2011-05-25 15:35:32 -07:00
open-style analog of vfs_path_lookup()
2011-03-14 09:15:28 -04:00
sysctl_check: drop dead code
2011-03-23 19:46:51 -07:00
taskstats: use appropriate printk priority level
2011-03-23 19:47:14 -07:00
kprobes: Fix selftest to clear flags field for reusing probes
2010-10-14 08:55:27 +02:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2011-03-15 18:53:35 -07:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2011-03-15 18:53:35 -07:00
jump label: Introduce static_branch() interface
2011-04-04 12:48:08 -04:00
taskstats: use real microsecond granularity for CPU times
2010-10-27 18:03:17 -07:00
userns: user namespaces: convert several capable() calls
2011-03-23 19:47:08 -07:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
userns: add a user_namespace as creator/owner of uts_namespace
2011-03-23 19:46:59 -07:00
user_ns: improve the user_ns on-the-slab packaging
2011-01-13 08:03:18 -08:00
ns proc: Add support for the uts namespace
2011-05-10 14:35:35 -07:00
Fix common misspellings
2011-03-31 11:26:23 -03:00
kernel/watchdog.c: Use proper ANSI C prototypes
2011-05-23 21:07:40 -07:00
Merge branch 'for-2.6.40' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu
2011-05-24 11:53:42 -07:00
09223371de