Linus Torvalds
c404e1e197
mm: allow a controlled amount of unfairness in the page lock ...
commit 5ef64cc8987a9211d3f3667331ba3411a94ddc79 upstream.
Commit 2a9127fcf229 ("mm: rewrite wait_on_page_bit_common() logic") made
the page locking entirely fair, in that if a waiter came in while the
lock was held, the lock would be transferred to the lockers strictly in
order.
That was intended to finally get rid of the long-reported watchdog
failures that involved the page lock under extreme load, where a process
could end up waiting essentially forever, as other page lockers stole
the lock from under it.
It also improved some benchmarks, but it ended up causing huge
performance regressions on others, simply because fair lock behavior
doesn't end up giving out the lock as aggressively, causing better
worst-case latency, but potentially much worse average latencies and
throughput.
Instead of reverting that change entirely, this introduces a controlled
amount of unfairness, with a sysctl knob to tune it if somebody needs
to. But the default value should hopefully be good for any normal load,
allowing a few rounds of lock stealing, but enforcing the strict
ordering before the lock has been stolen too many times.
There is also a hint from Matthieu Baerts that the fair page coloring
may end up exposing an ABBA deadlock that is hidden by the usual
optimistic lock stealing, and while the unfairness doesn't fix the
fundamental issue (and I'm still looking at that), it avoids it in
practice.
The amount of unfairness can be modified by writing a new value to the
'sysctl_page_lock_unfairness' variable (default value of 5, exposed
through /proc/sys/vm/page_lock_unfairness), but that is hopefully
something we'd use mainly for debugging rather than being necessary for
any deep system tuning.
This whole issue has exposed just how critical the page lock can be, and
how contended it gets under certain locks. And the main contention
doesn't really seem to be anything related to IO (which was the origin
of this lock), but for things like just verifying that the page file
mapping is stable while faulting in the page into a page table.
Link: https://lore.kernel.org/linux-fsdevel/ed8442fd-6f54-dd84-cd4a-941e8b7ee603@MichaelLarabel.com/
Link: https://www.phoronix.com/scan.php?page=article&item=linux-50-59&num=1
Link: https://lore.kernel.org/linux-fsdevel/c560a38d-8313-51fb-b1ec-e904bd8836bc@tessares.net/
Reported-and-tested-by: Michael Larabel <Michael@michaellarabel.com>
Tested-by: Matthieu Baerts <matthieu.baerts@tessares.net>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Chris Mason <clm@fb.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Saeed Mirzamohammadi <saeed.mirzamohammadi@oracle.com>
Tested-by: Maximilian Heyne <mheyne@amazon.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-08-30 16:27:26 +02:00
can: sja1000: fix size of OCR_MODE_MASK define
2022-12-19 12:24:16 +01:00
iio: st_sensors: Add a local lock for protecting odr
2022-06-14 18:11:59 +02:00
mailbox: zynqmp: Fix typo in IPI documentation
2023-05-17 11:35:55 +02:00
net: xgene: Move shared header file into include/linux
2023-02-06 07:52:47 +01:00
mfd: t7l66xb: Drop platform disable callback
2022-08-25 11:18:04 +02:00
RDMA/mlx4: Do not map the core_clock page to user space unless enabled
2021-06-16 11:59:44 +02:00
net/mlx5: Fix possible use-after-free in async command interface
2022-11-03 23:56:56 +09:00
mmc: core: Fix ambiguous TRIM and DISCARD arg
2022-12-08 11:23:04 +01:00
mtd: cfi_cmdset_0002: Use chip_ready() for write on S29GL064N
2022-06-14 18:12:03 +02:00
netfilter: add helper function to set up the nfnetlink header and use it
2023-07-27 08:37:27 +02:00
netfilter: arp_tables: add pre_exit hook for table unregister
2021-04-21 12:56:16 +02:00
netfilter: ebtables: reject blobs that don't provide all entry points
2022-09-05 10:27:41 +02:00
platform/x86: pmc_atom: Fix SLP_TYPx bitfield mask
2022-09-15 12:04:50 +02:00
power: supply: bq27xxx: Fix poll_interval handling and races on remove
2023-05-30 12:44:10 +01:00
mm: Move mm_cachep initialization to mm_init()
2023-08-08 19:56:36 +02:00
spi: Fix use-after-free with devm_spi_alloc_*
2021-05-14 09:44:20 +02:00
SUNRPC: remove the maximum number of retries in call_bind_status
2023-05-17 11:35:52 +02:00
usb: usbfs: Enforce page requirements for mmap
2023-06-14 10:59:58 +02:00
ACPI: tables: x86: Reserve memory occupied by ACPI tables
2021-05-07 10:51:36 +02:00
arm64: entry: Add vectors that have the bhb mitigation sequences
2022-03-19 13:40:15 +01:00
ata: fix ata_id_has_dipm()
2022-10-26 13:22:39 +02:00
block: return the correct bvec when checking for gaps
2021-07-14 16:53:48 +02:00
bitfield.h: Fix "type of reg too small for mask" test
2022-07-29 17:14:18 +02:00
ocfs2/dlm: move BITS_TO_BYTES() to bitops.h for wider use
2023-06-09 10:28:54 +02:00
block: don't merge across cgroup boundaries if blkcg is enabled
2022-04-15 14:18:01 +02:00
block: return ELEVATOR_DISCARD_MERGE if possible
2022-05-25 09:14:38 +02:00
x86/speculation: Add missing prototype for unpriv_ebpf_notify()
2022-06-14 18:11:34 +02:00
bpf: Fix pointer arithmetic mask tightening under state pruning
2021-09-15 09:47:39 +02:00
fs: only do a memory barrier for the first set_buffer_uptodate()
2022-09-15 12:04:49 +02:00
drivers: base: cacheinfo: Get rid of DEFINE_SMP_CALL_CACHE_FUNCTION()
2021-09-26 14:07:10 +02:00
cgroup: Use separate src/dst nodes when preloading css_sets for migration
2022-07-21 20:59:17 +02:00
memcg: fix possible use-after-free in memcg_write_event_control()
2022-12-14 11:30:43 +01:00
clk: Provide new devm_clk helpers for prepared and enabled clocks
2023-02-06 07:52:34 +01:00
compiler.h: Introduce absolute_pointer macro
2021-09-30 10:09:25 +02:00
serial: core: Fix initializing and restoring termios speed
2021-11-17 09:48:25 +01:00
vt: Fix character height handling with VT_RESIZEX
2021-05-26 12:05:20 +02:00
init: Provide arch_cpu_finalize_init()
2023-08-08 19:56:33 +02:00
firmware: arm_sdei: Fix sleep from invalid context BUG
2023-05-30 12:44:02 +01:00
debugfs: regset32: Add Runtime PM support
2023-05-17 11:35:32 +02:00
PM/devfreq: governor: Add a private governor_data for governor
2023-01-18 11:41:52 +01:00
dm: remove special-casing of bio-based immutable singleton target on NVMe
2022-06-25 12:44:35 +02:00
Revert "driver core: Annotate dev_err_probe() with __must_check"
2023-08-11 11:54:02 +02:00
linux/dim: Do nothing if no time delta between samples
2023-05-30 12:44:01 +01:00
dma-mapping: add generic helpers for mapping sgtable objects
2023-02-25 11:53:25 +01:00
dma-mapping: drop the dev argument to arch_sync_dma_for_*
2023-04-05 11:16:43 +02:00
dyndbg: fix module.dyndbg handling
2022-10-26 13:22:37 +02:00
efi: random: reduce seed size to 32 bytes
2022-11-10 17:57:55 +01:00
kyber: fix out of bounds access when preempted
2021-05-19 10:08:30 +02:00
elfcore: correct reference to CONFIG_UML
2021-10-27 09:54:27 +02:00
PM: EM: Increase energy calculation precision
2021-09-15 09:47:33 +02:00
net: create netdev->dev_addr assignment helpers
2023-07-27 08:37:08 +02:00
eventfd: change int to __u64 in eventfd_signal() ifndef CONFIG_EVENTFD
2023-01-18 11:40:58 +01:00
extcon: Add stubs for extcon_register_notifier_all() functions
2021-04-07 14:47:43 +02:00
fbcon: Prevent that screen size is smaller than font size
2022-07-12 16:30:47 +02:00
bpf: Prevent increasing bpf_jit_limit above max
2021-11-17 09:48:20 +01:00
fs: fix undefined behavior in bit shift for SB_NOUSER
2023-06-05 08:17:32 +02:00
devtmpfs regression fix: reconfigure on each mount
2022-01-20 09:19:17 +01:00
fscrypt: add fscrypt_symlink_getattr() for computing st_size
2021-09-12 08:56:38 +02:00
fsnotify: invalidate dcache before IN_DELETE event
2022-02-01 17:24:39 +01:00
ftrace: Mark get_lock_parent_ip() __always_inline
2023-04-20 12:07:34 +02:00
HID: core: Provide new max_buffer_size attribute to over-ride the default
2023-03-22 13:28:10 +01:00
mm/highmem: Lift memcpy_[to|from]_page to core
2023-01-18 11:41:55 +01:00
hrtimer: Ensure timerfd notification for HIGHRES=n
2021-09-15 09:47:26 +02:00
mm/thp: make is_huge_zero_pmd() safe and quicker
2021-06-30 08:47:52 -04:00
hugetlb: check for undefined shift on 32 bit architectures
2023-02-22 12:50:39 +01:00
random: pull add_hwgenerator_randomness() declaration into random.h
2022-06-22 14:11:12 +02:00
i2c: Add I2C_AQ_NO_REP_START adapter quirk
2021-05-19 10:08:22 +02:00
net: handle ARPHRD_PIMREG in dev_is_mac_header_xmit()
2022-03-23 09:12:07 +01:00
macvlan: macvlan_count_rx() needs to be aware of preemption
2021-03-30 14:35:25 +02:00
net: fix stack overflow when LRO is disabled for virtual interfaces
2023-05-30 12:44:09 +01:00
net: add vlan_get_protocol_and_depth() helper
2023-05-30 12:44:01 +01:00
ima: Align ima_file_mmap() parameters with mmap_file LSM hook
2023-03-11 16:44:02 +01:00
net: igmp: increase size of mr_ifc_count
2021-08-18 08:57:01 +02:00
interconnect: Add helpers for enabling/disabling a path
2023-08-30 16:27:13 +02:00
genirq: Add IRQF_NO_AUTOEN for request_irq/nmi()
2023-05-17 11:35:46 +02:00
iomap: iomap that extends beyond EOF should be marked dirty
2022-09-28 11:03:58 +02:00
iommu: Add gfp parameter to iommu_ops::map
2023-02-22 12:50:31 +01:00
iommu/iova: Fix module config properly
2022-10-26 13:22:46 +02:00
shm: extend forced shm destroy to support objects from several IPC nses
2021-12-01 09:23:35 +01:00
ipv6: make mc_forwarding atomic
2022-04-15 14:18:32 +02:00
genirq: Provide IRQCHIP_AFFINITY_PRE_STARTUP
2021-08-18 08:57:02 +02:00
irqdomain: Change the type of 'size' in __irq_domain_add() to be consistent
2023-03-17 08:32:49 +01:00
jbd2: fix kernel-doc markups
2023-08-11 11:53:43 +02:00
panic: Consolidate open-coded panic_on_warn checks
2023-02-06 07:52:50 +01:00
genirq: Fix the return type of kstat_cpu_irqs_sum()
2023-03-11 16:43:38 +01:00
kexec_file: drop weak attribute from arch_kexec_apply_relocations[_add]
2022-07-02 16:28:50 +02:00
kfifo: fix kfifo_to_user() return type
2022-08-25 11:18:04 +02:00
x86/kprobes: Fix arch_check_optimized_kprobe check within optimized_kprobe range
2023-03-11 16:44:02 +01:00
x86/kvm/vmx: Make noinstr clean
2022-10-07 09:16:54 +02:00
libata: fix read log timeout value
2021-11-17 09:48:17 +01:00
list: add "list_del_init_careful()" to go with "list_empty_careful()"
2023-06-28 10:18:35 +02:00
binder: use cred instead of task for selinux checks
2021-11-17 09:48:16 +01:00
mailbox: forward the hrtimer if not queued and under a lock
2022-06-14 18:11:42 +02:00
net: phy: marvell: fix detection of PHY on Topaz switches
2021-04-21 12:56:18 +02:00
ext4: fix deadlock due to mbcache entry corruption
2023-01-18 11:41:56 +01:00
rtc: mc146818-lib: Fix the AltCentury for AMD platforms
2022-05-25 09:14:32 +02:00
net: mdio: introduce a shutdown method to mdio device drivers
2021-10-09 14:39:49 +02:00
memblock: fix memblock_phys_alloc() section mismatch error
2022-01-05 12:37:44 +01:00
mm: memcontrol: fix occasional OOMs due to proportional memory.low reclaim
2021-08-26 08:36:22 -04:00
mm/memory_hotplug: use "unsigned long" for PFN in zone_for_pfn_range()
2021-09-22 12:26:43 +02:00
mm: allow a controlled amount of unfairness in the page lock
2023-08-30 16:27:26 +02:00
mm: fix struct page layout on 32-bit systems
2021-05-19 10:08:31 +02:00
mmap locking API: initial implementation as rwsem wrappers
2022-07-29 17:14:18 +02:00
mm: Fix '.data.once' orphan section warning
2022-12-08 11:23:04 +01:00
arm: remove CONFIG_ARCH_HAS_HOLES_MEMORYMODEL
2022-05-15 19:54:46 +02:00
x86/cpu: Add a steppings field to struct x86_cpu_id
2022-10-07 09:16:54 +02:00
modules: inherit TAINT_PROPRIETARY_MODULE
2021-05-11 14:04:04 +02:00
PCI/MSI: Protect msi_desc::masked for multi-MSI
2021-08-18 08:57:03 +02:00
locking/mutex: Fix non debug version of mutex_lock_io_nested()
2021-03-30 14:35:29 +02:00
net: Fix features skip in for_each_netdev_feature()
2022-05-18 09:47:24 +02:00
net: create netdev->dev_addr assignment helpers
2023-07-27 08:37:08 +02:00
Remove DECnet support from kernel
2023-06-21 15:44:10 +02:00
Remove DECnet support from kernel
2023-06-21 15:44:10 +02:00
NFS: swap IO handling is slightly different for O_DIRECT IO
2022-04-15 14:18:35 +02:00
watchdog/perf: define dummy watchdog_update_hrtimer_threshold() on correct config
2023-07-27 08:37:10 +02:00
nodemask: Fix return values to be unsigned
2022-06-14 18:12:02 +02:00
uaccess: Add speculation barrier to copy_from_user()
2023-02-25 11:53:26 +01:00
nvme-tcp: fix nvme_tcp_term_pdu to match spec
2023-04-05 11:16:38 +02:00
nvme: handle the persistent internal error AER
2023-05-17 11:35:43 +02:00
of: clk: Make <linux/of_clk.h> self-contained
2021-12-08 09:01:08 +01:00
once: fix section mismatch on clang builds
2022-11-01 19:06:42 +01:00
mm, oom: make the calculation of oom badness more accurate
2021-09-03 10:08:12 +02:00
mm, futex: fix shared futex pgoff on shmem huge page
2021-06-30 08:47:55 -04:00
PCI/ACPI: Add Ampere Altra SOC MCFG quirk
2021-09-26 14:07:05 +02:00
PCI: Add pci_clear_master() stub for non-CONFIG_PCI
2023-07-27 08:37:16 +02:00
PCI: Add SolidRun vendor ID
2023-03-17 08:32:52 +01:00
perf: Protect perf_guest_cbs with RCU
2022-01-20 09:19:18 +01:00
PM: runtime: Fix unpaired parent child_count for force_resume
2021-05-19 10:08:20 +02:00
PM: domains: Restore comment indentation for generic_pm_domain.child_links
2023-05-17 11:36:04 +02:00
PM: runtime: Add pm_runtime_get_if_active()
2023-08-30 16:27:12 +02:00
PM / wakeirq: support enabling wake-up irq after runtime_suspend called
2023-08-11 11:54:00 +02:00
power: supply: core: Refactor power_supply_set_input_current_limit_from_supplier()
2023-06-05 08:17:31 +02:00
siphash: use one source of truth for siphash permutations
2022-06-22 14:11:16 +02:00
printk: declare printk_deferred_{enter,safe}() in include/linux/printk.h
2023-05-17 11:36:04 +02:00
net, proc: Provide PROC_FS=n fallback for proc_create_net_single_write()
2023-01-18 11:41:02 +01:00
psi: Fix uaf issue when psi trigger is destroyed while being polled
2022-02-05 12:35:36 +01:00
psi: Fix uaf issue when psi trigger is destroyed while being polled
2022-02-05 12:35:36 +01:00
ptp: improve max_adj check against unreasonable values
2021-06-23 14:41:26 +02:00
ptrace/xtensa: Replace PT_SINGLESTEP with TIF_SINGLESTEP
2022-06-14 18:11:24 +02:00
quota: Factor out setup of quota inode
2023-01-18 11:42:02 +01:00
random: always mix cycle counter in add_latent_entropy()
2023-02-25 11:53:25 +01:00
random: quiet urandom warning ratelimit suppression message
2022-06-29 08:58:45 +02:00
rcu: Upgrade rcu_swap_protected() to rcu_replace_pointer()
2023-06-28 10:18:38 +02:00
locking/refcount: Consolidate implementations of refcount_t
2022-07-29 17:14:17 +02:00
tracing/ring-buffer: Have polling block on watermark
2022-12-08 11:23:05 +01:00
mm/rmap: Fix anon_vma->degree ambiguity leading to double-reuse
2022-09-05 10:27:46 +02:00
rpmsg: Fix rpmsg_create_ept return when RPMSG config is not defined
2021-11-17 09:48:44 +01:00
misc: rtsx_usb: use separate command and response buffers
2022-07-12 16:30:49 +02:00
scatterlist: add generic wrappers for iterating over sgtable objects
2023-02-25 11:53:25 +01:00
kernel/sched: Remove dl_boosted flag comment
2022-09-05 10:27:39 +02:00
lockdown: also lock down previous kgdb use
2022-06-06 08:33:48 +02:00
Revert "8250: add support for ASIX devices with a FIFO bug"
2023-07-27 08:37:38 +02:00
serial: Create uart_xmit_advance()
2022-09-28 11:04:08 +02:00
sh: intc: Avoid spurious sizeof-pointer-div warning
2023-03-22 13:28:08 +01:00
siphash: use one source of truth for siphash permutations
2022-06-22 14:11:16 +02:00
netfilter: nf_tables: fix ifdef to also consider nf_tables=m
2023-04-26 11:24:02 +02:00
bpf, sockmap: Fix sk->prot unhash op reset
2021-04-14 08:24:12 +02:00
smp: Fix smp_call_function_single_async prototype
2021-05-14 09:44:33 +02:00
net: make get_net_ns return error if NET_NS is disabled
2021-06-23 14:41:25 +02:00
net: stmmac: do not stop RX_CLK in Rx LPI state for qcs404 SoC
2023-02-22 12:50:34 +01:00
lib/string_helpers: Introduce string_upper() and string_lower() helpers
2023-05-30 12:44:08 +01:00
PM: s2idle: ACPI: Fix wakeup interrupts handling
2022-02-16 12:52:50 +01:00
sysctl: add a new register_sysctl_init() interface
2023-02-06 07:52:48 +01:00
tcp: annotate data-races around fastopenq.max_qlen
2023-07-27 08:37:45 +02:00
tee: export teedev_open() and teedev_close_context()
2022-03-02 11:41:04 +01:00
thermal/core: Fix thermal_cooling_device_register() prototype
2021-09-26 14:07:10 +02:00
kernel, fs: Introduce and use set_restart_fn() and arch_set_restart_data()
2021-03-24 11:26:44 +01:00
tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem
2023-05-17 11:35:57 +02:00
Revert "time: Handle negative seconds correctly in timespec64_to_ns()"
2021-09-16 12:56:13 +02:00
timerqueue: Use rb_entry_safe() in timerqueue_getnext()
2023-01-18 11:40:53 +01:00
timekeeping: Add raw clock fallback for random_get_entropy()
2022-06-22 14:11:15 +02:00
sched/topology: Make sched_init_numa() use a set for the deduplicating sort
2022-03-08 19:07:48 +01:00
efi: tpm: Avoid READ_ONCE() for accessing the event log
2023-01-18 11:42:00 +01:00
tracing: Make tracepoint lockdep check actually test something
2023-03-22 13:28:08 +01:00
tty: Prevent writing chars during tcsetattr TCSADRAIN/FLUSH
2023-05-17 11:36:04 +02:00
tty: fix return value for unsupported ioctls
2021-05-14 09:44:22 +02:00
tty: use new tty_insert_flip_string_and_push_buffer() in pty_write()
2022-07-29 17:14:20 +02:00
u64_stats,lockdep: Fix u64_stats_init() vs lockdep
2021-03-30 14:35:22 +02:00
uaccess: Add minimum bounds check on kernel buffer size
2023-03-11 16:43:56 +01:00
USB: core: Add routines for endpoint checks in old drivers
2023-05-30 12:44:10 +01:00
usb-storage: Add quirk to defeat Kindle's automatic unload
2021-03-24 11:26:40 +01:00
virtio: unexport virtio_finalize_features
2022-03-16 13:21:48 +01:00
virtio: acknowledge all features before access
2022-03-16 13:21:48 +01:00
net: do not allow gso_size to be set to GSO_BY_FRAGS
2023-08-30 16:27:17 +02:00
linux/vt_buffer.h: allow either builtin or modular for macros
2023-05-17 11:35:46 +02:00
mm: allow a controlled amount of unfairness in the page lock
2023-08-30 16:27:26 +02:00
workqueue: clean up WORK_* constant types, clarify masking
2023-07-27 08:37:29 +02:00
XArray: add xas_split
2021-06-10 13:37:14 +02:00
