Linus Torvalds
3510ca20ec
Minor page waitqueue cleanups ...
Tim Chen and Kan Liang have been battling a customer load that shows
extremely long page wakeup lists. The cause seems to be constant NUMA
migration of a hot page that is shared across a lot of threads, but the
actual root cause for the exact behavior has not been found.
Tim has a patch that batches the wait list traversal at wakeup time, so
that we at least don't get long uninterruptible cases where we traverse
and wake up thousands of processes and get nasty latency spikes. That
is likely 4.14 material, but we're still discussing the page waitqueue
specific parts of it.
In the meantime, I've tried to look at making the page wait queues less
expensive, and failing miserably. If you have thousands of threads
waiting for the same page, it will be painful. We'll need to try to
figure out the NUMA balancing issue some day, in addition to avoiding
the excessive spinlock hold times.
That said, having tried to rewrite the page wait queues, I can at least
fix up some of the braindamage in the current situation. In particular:
(a) we don't want to continue walking the page wait list if the bit
we're waiting for already got set again (which seems to be one of
the patterns of the bad load). That makes no progress and just
causes pointless cache pollution chasing the pointers.
(b) we don't want to put the non-locking waiters always on the front of
the queue, and the locking waiters always on the back. Not only is
that unfair, it means that we wake up thousands of reading threads
that will just end up being blocked by the writer later anyway.
Also add a comment about the layout of 'struct wait_page_key' - there is
an external user of it in the cachefiles code that means that it has to
match the layout of 'struct wait_bit_key' in the two first members. It
so happens to match, because 'struct page *' and 'unsigned long *' end
up having the same values simply because the page flags are the first
member in struct page.
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Kan Liang <kan.liang@intel.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Christopher Lameter <cl@linux.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-08-27 13:55:12 -07:00
bpf: fix bpf_prog_get_info_by_fd to dump correct xlated_prog_len
2017-07-29 23:29:41 -07:00
cpuset: fix a deadlock due to incomplete patching of cpusets_enabled()
2017-08-02 17:16:12 -07:00
config: android-base: disable CONFIG_NFSD and CONFIG_NFS_FS
2017-06-09 11:47:38 +02:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h>
2017-03-02 08:42:34 +01:00
perf/core: Fix group {cpu,task} validation
2017-08-25 11:00:34 +02:00
gcov: support GCC 7.1
2017-05-12 15:57:15 -07:00
genirq/ipi: Fixup checks against nr_cpu_ids
2017-08-20 10:49:05 +02:00
livepatch: Fix stacking of patches with respect to RCU
2017-06-20 10:42:19 +02:00
Merge branch 'locking-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2017-07-21 11:11:23 -07:00
mm: fix global NR_SLAB_.*CLAIMABLE counter reads
2017-08-10 15:54:06 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/pmladek/printk
2017-07-05 11:11:26 -07:00
rcu: Remove RCU CPU stall warnings from Tiny RCU
2017-06-08 18:52:45 -07:00
Minor page waitqueue cleanups
2017-08-27 13:55:12 -07:00
timers: Fix excessive granularity of new timers after a nohz idle
2017-08-24 11:40:18 +02:00
Various bug fixes:
2017-08-24 14:08:22 -07:00
sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h>
2017-03-02 08:42:39 +01:00
async: Adjust system_state checks
2017-05-23 10:01:37 +02:00
Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit
2017-07-20 10:22:26 -07:00
audit: style fix
2017-06-12 18:07:43 -04:00
Merge branch 'fsnotify' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs
2017-05-03 11:05:15 -07:00
Merge branch 'fsnotify' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs
2017-05-03 11:05:15 -07:00
audit/stable-4.13 PR 20170816
2017-08-16 16:48:34 -07:00
audit: kernel generated netlink traffic should have a portid of 0
2017-05-02 10:16:05 -04:00
Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit
2017-07-05 11:24:05 -07:00
Merge branch 'misc.compat' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2017-07-06 20:57:13 -07:00
smp/hotplug: Replace BUG_ON and react useful
2017-07-11 22:25:44 +02:00
kdump: protect vmcoreinfo data under the crash memory
2017-07-12 16:26:00 -07:00
doc: ReSTify credentials.txt
2017-05-18 10:30:19 -06:00
sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h>
2017-03-02 08:42:39 +01:00
kernel/exit.c: avoid undefined behaviour when calling wait4()
2017-07-10 16:32:36 -07:00
lib/extable.c: use bsearch() library function in search_extable()
2017-07-10 16:32:35 -07:00
fork: fix incorrect fput of ->exe_file causing use-after-free
2017-08-25 16:12:46 -07:00
futex: Remove unnecessary warning from get_futex_key
2017-08-09 14:00:54 -07:00
kernel/groups.c: use sort library function
2017-07-10 16:32:34 -07:00
kernel/hung_task.c: defer showing held locks
2017-05-08 17:15:10 -07:00
jump_label: Reorder hotplug lock and jump_label_lock
2017-05-26 10:10:45 +02:00
kernel/kallsyms.c: replace all_var with IS_ENABLED(CONFIG_KALLSYMS_ALL)
2017-07-10 16:32:34 -07:00
kcmp: add KCMP_EPOLL_TFD mode to compare epoll target files
2017-07-12 16:26:01 -07:00
kcov: simplify interrupt check
2017-05-08 17:15:12 -07:00
kdump: protect vmcoreinfo data under the crash memory
2017-07-12 16:26:00 -07:00
kdump: protect vmcoreinfo data under the crash memory
2017-07-12 16:26:00 -07:00
kexec_file: adjust declaration of kexec_purgatory
2017-07-12 16:26:02 -07:00
kexec_file: adjust declaration of kexec_purgatory
2017-07-12 16:26:02 -07:00
kmod: fix wait on recursive loop
2017-08-18 15:32:01 -07:00
kprobes: Ensure that jprobe probepoints are at function entry
2017-07-08 11:05:35 +02:00
kexec: move vmcoreinfo out of the kernel's .bss section
2017-07-12 16:25:59 -07:00
cgroup, kthread: close race window where new kthreads can be migrated to non-root cgroups
2017-03-17 10:18:47 -04:00
sched/headers: Prepare to move sched_info_on() and force_schedstat_enabled() from <linux/sched.h> to <linux/sched/stat.h>
2017-03-02 08:42:39 +01:00
kernel/watchdog: split up config options
2017-07-12 16:26:02 -07:00
mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
2017-07-06 16:24:32 -07:00
Modules updates for v4.13
2017-07-12 17:22:01 -07:00
perf: Add PERF_RECORD_NAMESPACES to include namespaces related info
2017-03-13 15:57:41 -03:00
padata: Avoid nested calls to cpus_read_lock() in pcrypt_init_padata()
2017-05-26 10:10:37 +02:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h>
2017-03-02 08:42:34 +01:00
boot/param: Move next_arg() function to lib/cmdline.c for later reuse
2017-04-18 10:37:13 +02:00
pids: make task_tgid_nr_ns() safe
2017-08-21 12:47:31 -07:00
pid_ns: Sleep in TASK_INTERRUPTIBLE in zap_pid_ns_processes
2017-05-13 17:26:01 -05:00
sched/headers: Prepare to move sched_info_on() and force_schedstat_enabled() from <linux/sched.h> to <linux/sched/stat.h>
2017-03-02 08:42:39 +01:00
ptrace: Properly initialize ptracer_cred on fork
2017-05-23 07:40:44 -05:00
Merge branch 'work.splice' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2017-05-02 11:38:06 -07:00
seccomp: Switch from atomic_t to recount_t
2017-06-26 09:24:00 -07:00
signal: don't remove SIGNAL_UNKILLABLE for traced tasks.
2017-08-18 15:32:02 -07:00
smp, cpumask: Use non-atomic cpumask_{set,clear}_cpu()
2017-05-23 10:01:32 +02:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/task.h>
2017-03-02 08:42:35 +01:00
sched/core: Remove 'task' parameter and rename tsk_restore_flags() to current_restore_flags()
2017-04-11 09:06:32 +02:00
stacktrace/x86: add function for detecting reliable stack traces
2017-03-08 09:18:02 +01:00
stop_machine: Provide stop_machine_cpuslocked()
2017-05-26 10:10:36 +02:00
fix a braino in compat_sys_getrlimit()
2017-07-12 09:15:00 -07:00
kernel/watchdog: split up config options
2017-07-12 16:26:02 -07:00
kernel/sysctl_binary.c: check name array length in deprecated_sysctl_warning()
2017-07-12 16:26:00 -07:00
taskstats: add e/u/stime for TGID command
2017-05-08 17:15:12 -07:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/clock.h>
2017-03-02 08:42:27 +01:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/task.h>
2017-03-02 08:42:35 +01:00
sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h>
2017-03-02 08:42:39 +01:00
ucount: Remove the atomicity from ucount->count
2017-03-06 15:26:37 -06:00
sched/headers: Prepare to remove <linux/cred.h> inclusion from <linux/sched.h>
2017-03-02 08:42:31 +01:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/user.h>
2017-03-02 08:42:29 +01:00
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/signal.h>
2017-03-02 08:42:29 +01:00
sched/headers: Prepare to move the task_lock()/unlock() APIs to <linux/sched/task.h>
2017-03-02 08:42:38 +01:00
sched/headers: Remove <linux/rwsem.h> from <linux/sched.h>
2017-03-03 01:45:36 +01:00
kernel/watchdog: Prevent false positives with turbo modes
2017-08-18 12:35:02 +02:00
kernel/watchdog: Prevent false positives with turbo modes
2017-08-18 12:35:02 +02:00
workqueue: Work around edge cases for calc of pool's cpumask
2017-07-28 11:05:52 -04:00
3510ca20ec