Peter Zijlstra
aacedf26fb
sched/core: Optimize try_to_wake_up() for local wakeups ...
Jens reported that significant performance can be had on some block
workloads by special casing local wakeups. That is, wakeups on the
current task before it schedules out.
Given something like the normal wait pattern:
for (;;) {
set_current_state(TASK_UNINTERRUPTIBLE);
if (cond)
break;
schedule();
}
__set_current_state(TASK_RUNNING);
Any wakeup (on this CPU) after set_current_state() and before
schedule() would benefit from this.
Normal wakeups take p->pi_lock, which serializes wakeups to the same
task. By eliding that we gain concurrency on:
- ttwu_stat(); we already had concurrency on rq stats, this now also
brings it to task stats. -ENOCARE
- tracepoints; it is now possible to get multiple instances of
trace_sched_waking() (and possibly trace_sched_wakeup()) for the
same task. Tracers will have to learn to cope.
Furthermore, p->pi_lock is used by set_special_state(), to order
against TASK_RUNNING stores from other CPUs. But since this is
strictly CPU local, we don't need the lock, and set_special_state()'s
disabling of IRQs is sufficient.
After the normal wakeup takes p->pi_lock it issues
smp_mb__after_spinlock(), in order to ensure the woken task must
observe prior stores before we observe the p->state. If this is CPU
local, this will be satisfied with a compiler barrier, and we rely on
try_to_wake_up() being a funcation call, which implies such.
Since, when 'p == current', 'p->on_rq' must be true, the normal wakeup
would continue into the ttwu_remote() branch, which normally is
concerned with exactly this wakeup scenario, except from a remote CPU.
IOW we're waking a task that is still running. In this case, we can
trivially avoid taking rq->lock, all that's left from this is to set
p->state.
This then yields an extremely simple and fast path for 'p == current'.
Reported-by: Jens Axboe <axboe@kernel.dk>
Tested-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Qian Cai <cai@lca.pw>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: akpm@linux-foundation.org
Cc: gkohli@codeaurora.org
Cc: hch@lst.de
Cc: oleg@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-06-17 12:15:59 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 363
2019-06-05 17:37:09 +02:00
Linux 5.2-rc5
2019-06-17 12:12:27 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 333
2019-06-05 17:37:06 +02:00
perf/ring-buffer: Use regular variables for nesting
2019-05-24 09:00:11 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
This includes the following fixes:
2019-06-15 07:24:11 -10:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 436
2019-06-05 17:37:17 +02:00
SPDX update for 5.2-rc4
2019-06-08 12:52:42 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 13
2019-05-21 11:28:45 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
sched/core: Optimize try_to_wake_up() for local wakeups
2019-06-17 12:15:59 +02:00
Linux 5.2-rc5
2019-06-17 12:12:27 +02:00
Linux 5.2-rc5
2019-06-17 12:12:27 +02:00
Provide in-kernel headers to make extending kernel easier
2019-04-29 16:48:03 +02:00
acct_on(): don't mess with freeze protection
2019-04-04 21:04:13 -04:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
fsnotify: switch send_to_group() and ->handle_event to const struct qstr *
2019-04-26 13:51:03 -04:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
Merge branch 'work.dcache' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-05-07 20:03:32 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
kernel/compat.c: mark expected switch fall-throughs
2019-05-15 08:16:14 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
x86/power: Fix 'nosmt' vs hibernation triple fault during resume
2019-06-03 12:02:03 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 282
2019-06-05 17:36:37 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
2019-06-11 15:44:45 -10:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 25
2019-05-21 11:52:39 +02:00
cgroup: Call cgroup_release() before __exit_signal()
2019-05-31 10:38:57 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Switch printk users from %pf and %pF to %ps and %pS, respectively
2019-04-09 14:19:06 +02:00
sched/core: Provide a pointer to the valid CPU mask
2019-06-03 11:49:37 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
kheaders: Do not regenerate archive if config is not changed
2019-05-24 20:16:01 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
mm/resource: Use resource_overlaps() to simplify region_intersects()
2019-04-19 12:59:36 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
power/suspend: Add function to disable secondaries for suspend
2019-05-03 19:42:41 +02:00
mm: memblock: make keeping memblock memory opt-in rather than opt-out
2019-05-14 09:47:50 -07:00
kheaders: Move from proc to sysfs
2019-05-24 20:16:01 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 170
2019-05-30 11:26:39 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
kheaders: Move from proc to sysfs
2019-05-24 20:16:01 +02:00
mm/devm_memremap_pages: fix final page put race
2019-06-13 17:34:56 -10:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
padata: Replace padata_attr_type default_attrs field with groups
2019-04-25 22:06:11 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
2019-06-11 15:44:45 -10:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
rseq: Remove superfluous rseq_len from task_struct
2019-04-19 12:39:32 +02:00
audit/stable-5.2 PR 20190507
2019-05-07 19:06:04 -07:00
signal: improve comments
2019-06-05 15:06:07 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 430
2019-06-05 17:37:16 +02:00
Merge branch 'core-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-06-02 11:04:42 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 38
2019-05-24 17:27:11 +02:00
prctl_set_mm: downgrade mmap_sem to read lock
2019-06-01 15:51:31 -07:00
signal: support CLONE_PIDFD with pidfd_send_signal
2019-05-07 14:31:03 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 25
2019-05-21 11:52:39 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 156
2019-05-30 11:26:35 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
watchdog: Fix typo in comment
2019-04-18 14:05:51 +02:00
kernel/watchdog_hld.c: hard lockup message should end with a newline
2019-04-19 09:46:05 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
sched/core, workqueues: Distangle worker accounting from rq lock
2019-04-16 16:55:15 +02:00
aacedf26fb