Waiman Long
19c5d690e4
locking/rwsem: Add reader-owned state to the owner field ...
Currently, it is not possible to determine for sure if a reader
owns a rwsem by looking at the content of the rwsem data structure.
This patch adds a new state RWSEM_READER_OWNED to the owner field
to indicate that readers currently own the lock. This enables us to
address the following 2 issues in the rwsem optimistic spinning code:
1) rwsem_can_spin_on_owner() will disallow optimistic spinning if
the owner field is NULL which can mean either the readers own
the lock or the owning writer hasn't set the owner field yet.
In the latter case, we miss the chance to do optimistic spinning.
2) While a writer is waiting in the OSQ and a reader takes the lock,
the writer will continue to spin when out of the OSQ in the main
rwsem_optimistic_spin() loop as the owner field is NULL wasting
CPU cycles if some of readers are sleeping.
Adding the new state will allow optimistic spinning to go forward as
long as the owner field is not RWSEM_READER_OWNED and the owner is
running, if set, but stop immediately when that state has been reached.
On a 4-socket Haswell machine running on a 4.6-rc1 based kernel, the
fio test with multithreaded randrw and randwrite tests on the same
file on a XFS partition on top of a NVDIMM were run, the aggregated
bandwidths before and after the patch were as follows:
Test BW before patch BW after patch % change
---- --------------- -------------- --------
randrw 988 MB/s 1192 MB/s +21%
randwrite 1513 MB/s 1623 MB/s +7.3%
The perf profile of the rwsem_down_write_failed() function in randrw
before and after the patch were:
19.95% 5.88% fio [kernel.vmlinux] [k] rwsem_down_write_failed
14.20% 1.52% fio [kernel.vmlinux] [k] rwsem_down_write_failed
The actual CPU cycles spend in rwsem_down_write_failed() dropped from
5.88% to 1.52% after the patch.
The xfstests was also run and no regression was observed.
Signed-off-by: Waiman Long <Waiman.Long@hpe.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Jason Low <jason.low2@hp.com>
Acked-by: Davidlohr Bueso <dave@stgolabs.net>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Douglas Hatch <doug.hatch@hpe.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Hurley <peter@hurleysoftware.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Scott J Norton <scott.norton@hpe.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1463534783-38814-2-git-send-email-Waiman.Long@hpe.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-06-08 15:16:59 +02:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2016-05-31 22:28:28 -07:00
mm/init: Add 'rodata=off' boot cmdline parameter to disable read-only kernel mappings
2016-02-22 08:51:37 +01:00
Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2016-05-25 17:05:40 -07:00
gcov: disable for COMPILE_TEST
2016-05-10 17:12:49 +02:00
radix-tree: introduce radix_tree_empty
2016-05-20 17:58:30 -07:00
Merge branches 'for-4.7/core', 'for-4.7/livepatching-doc' and 'for-4.7/livepatching-ppc64' into for-linus
2016-05-17 12:06:35 +02:00
locking/rwsem: Add reader-owned state to the owner field
2016-06-08 15:16:59 +02:00
PM / Hibernate: Call flush_icache_range() on pages restored in-place
2016-04-28 13:35:48 +01:00
printk/nmi: flush NMI messages on the system panic
2016-05-20 17:58:30 -07:00
debugobjects: insulate non-fixup logic related to static obj from fixup callbacks
2016-05-19 19:12:14 -07:00
Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2016-05-25 17:11:43 -07:00
timer: Export destroy_hrtimer_on_stack()
2016-05-31 11:44:08 -07:00
Three more changes.
2016-05-22 19:40:39 -07:00
certs: add .gitignore to stop git nagging about x509_certificate_list
2015-10-21 15:18:35 +01:00
async: export current_is_async()
2015-11-19 17:51:48 +01:00
Merge branch 'stable-4.7' of git://git.infradead.org/users/pcmoore/audit
2016-05-18 18:46:55 -07:00
security: Make inode argument of inode_getsecid non-const
2015-12-24 11:09:39 -05:00
wrappers for ->i_mutex access
2016-01-22 18:04:28 -05:00
audit: cleanup prune_tree_thread
2016-04-04 09:46:47 -04:00
don't bother with ->d_inode->i_sb - it's always equal to ->d_sb
2016-04-10 17:11:51 -04:00
audit: Fix typo in comment
2016-02-08 11:25:39 -05:00
Merge branch 'stable-4.7' of git://git.infradead.org/users/pcmoore/audit
2016-05-18 18:46:55 -07:00
cgroup: fix compile warning
2016-05-12 11:05:27 -04:00
cgroup: kill cgrp_ss_priv[CGROUP_CANFORK_COUNT] and friends
2015-12-03 10:24:08 -05:00
cgroup_pids: fix a typo.
2015-12-14 14:54:37 -05:00
context_tracking: Switch to new static_branch API
2015-11-24 09:56:43 +01:00
sched/hotplug: Make activate() the last hotplug step
2016-05-06 14:58:25 +02:00
kernel/cpu_pm: fix cpu_cluster_pm_exit comment
2015-09-03 02:42:20 +02:00
cpuset: use static key better and convert to new API
2016-05-19 19:12:14 -07:00
kmemcg: account certain kmem allocations to memcg
2016-01-14 16:00:49 -08:00
kmemcg: account certain kmem allocations to memcg
2016-01-14 16:00:49 -08:00
wait: allow sys_waitid() to accept __WNOTHREAD/__WCLONE/__WALL
2016-05-23 17:04:14 -07:00
kernel/extable.c: remove duplicated include
2015-09-10 13:29:01 -07:00
mm: oom_reaper: remove some bloat
2016-05-26 15:35:44 -07:00
x86: remove more uaccess_32.h complexity
2016-05-22 17:21:27 -07:00
ptrace: use fsuid, fsgid, effective creds for fs access checks
2016-01-20 17:09:18 -08:00
kernel/hung_task.c: use timeout diff when timeout is updated
2016-03-22 15:36:02 -07:00
treewide: Remove old email address
2015-11-23 09:44:58 +01:00
treewide: Remove old email address
2015-11-23 09:44:58 +01:00
kallsyms: add support for relative offsets in kallsyms address table
2016-03-15 16:55:16 -07:00
ptrace: use fsuid, fsgid, effective creds for fs access checks
2016-01-20 17:09:18 -08:00
kcov: don't profile branches in kcov
2016-04-28 19:34:04 -07:00
s390/kexec: consolidate crash_map/unmap_reserved_pages() and arch_kexec_protect(unprotect)_crashkres()
2016-05-23 17:04:14 -07:00
s390/kexec: consolidate crash_map/unmap_reserved_pages() and arch_kexec_protect(unprotect)_crashkres()
2016-05-23 17:04:14 -07:00
kexec: introduce a protection mechanism for the crashkernel reserved memory
2016-05-23 17:04:14 -07:00
kexec: move some memembers and definitions within the scope of CONFIG_KEXEC_FILE
2016-01-20 17:09:18 -08:00
kmod: don't run async usermode helper as a child of kworker thread
2015-10-23 17:55:10 +09:00
rcu: Remove TINY_RCU bloat from pointless boot parameters
2015-12-07 16:59:37 -08:00
kernel/kthread.c:kthread_create_on_node(): clarify documentation
2015-09-04 16:54:41 -07:00
sched/debug: Make schedstats a runtime tunable that is disabled by default
2016-02-09 11:54:23 +01:00
ELF/MIPS build fix
2016-05-23 17:04:14 -07:00
sys_membarrier(): system-wide memory barrier (generic, x86)
2015-09-11 15:21:34 -07:00
memremap: add arch specific hook for MEMREMAP_WB mappings
2016-04-04 10:26:41 +02:00
module: preserve Elf information for livepatch modules
2016-04-01 15:00:10 +02:00
KEYS: Move the point of trust determination to __key_link()
2016-04-11 22:43:43 +01:00
cgroup: introduce cgroup namespaces
2016-02-16 13:04:58 -05:00
kernel/padata.c: hide unused functions
2016-05-19 19:12:14 -07:00
printk/nmi: flush NMI messages on the system panic
2016-05-20 17:58:30 -07:00
Nothing exciting, minor tweaks and cleanups.
2015-11-09 15:53:39 -08:00
remove lots of IS_ERR_VALUE abuses
2016-05-27 15:26:11 -07:00
profile: hide unused functions when !CONFIG_PROC_FS
2016-03-22 15:36:02 -07:00
ptrace: change __ptrace_unlink() to clear ->ptrace under ->siglock
2016-03-22 15:36:02 -07:00
kexec: split kexec_load syscall from kexec core code
2015-09-10 13:29:01 -07:00
wrappers for ->i_mutex access
2016-01-22 18:04:28 -05:00
/proc/iomem: only expose physical resource addresses to privileged users
2016-04-14 12:56:09 -07:00
Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus
2016-05-19 10:02:26 -07:00
kernel/signal.c: convert printk(KERN_<LEVEL> ...) to pr_<level>(...)
2016-05-23 17:04:14 -07:00
Merge branch 'smp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2016-03-15 13:50:29 -07:00
cpu/hotplug: Unpark smpboot threads from the state machine
2016-03-01 20:36:56 +01:00
cpu/hotplug: Create hotplug threads
2016-03-01 20:36:56 +01:00
arch, ftrace: for KASAN put hard/soft IRQ entries into separate sections
2016-03-25 16:37:42 -07:00
kernel/stop_machine.c: remove CONFIG_SMP dependencies
2016-01-16 11:17:24 -08:00
prctl: make PR_SET_THP_DISABLE wait for mmap_sem killable
2016-05-23 17:04:14 -07:00
vfs: add copy_file_range syscall and vfs helper
2015-12-01 14:00:53 -05:00
Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2016-05-25 17:05:40 -07:00
kernel/sysctl_binary.c: use generic UUID library
2016-05-20 17:58:30 -07:00
task_work: remove fifo ordering guarantee
2015-09-05 13:46:58 -07:00
taskstats: use the libnl API to align nlattr on 64-bit
2016-04-23 20:13:25 -04:00
rcutorture: Dump trace buffer upon shutdown
2016-04-21 13:47:04 -07:00
kernel/...: convert pr_warning to pr_warn
2016-03-22 15:36:02 -07:00
time, acct: Drop irq save & restore from __acct_update_integrals()
2016-02-29 09:53:09 +01:00
kernel/*: switch to memdup_user_nul()
2016-01-04 10:27:55 -05:00
watchdog: don't run proc_watchdog_update if new value is same as old
2016-03-17 15:09:34 -07:00
debugobjects: insulate non-fixup logic related to static obj from fixup callbacks
2016-05-19 19:12:14 -07:00
sched/core: Get rid of 'cpu' argument in wq_worker_sleeping()
2016-03-02 10:28:47 -05:00
19c5d690e4