mirror of
https://github.com/BobTheBlinker/android_kernel_motorola_sm6375.git
synced 2026-10-16 09:43:28 -04:00
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Blagovest Kolenichev
|
7546983cd7 |
Merge android11-5.4.58+ (33cc3e7) into msm-5.4
* refs/heads/tmp-33cc3e7:
ANDROID: db845c: set BUILD_INITRAMFS=1
ANDROID: GKI: Build CEC_{CORE,NOTIFIER,PIN} in
ANDROID: GKI: Enable CEC support
ANDROID: ABI: Update ABI snapshot after additions to struct io_pgtable_cfg
ANDROID: iommu/io-pgtable: Allow IOMMU drivers to allocate/free page-table memory
ANDROID: update ABI
UPSTREAM: update to latest version of scripts/checkpatch.pl
FROMLIST: ufs: change the way to complete fDeviceInit
ANDROID: GKI: add built-in BCM Bluetooth driver
Revert "Revert "genetlink: remove genl_bind""
ANDROID: GKI: enable CONFIG_LEDS_TRIGGER_TIMER
ANDROID: GKI: add abi_gki_aarch64_oplus
ANDROID: GKI: fix up abi .xml file due to previous changes.
ANDROID: GKI: add some padding to some driver core structures
ANDROID: vendor_hooks: add waiting information for blocked tasks
ANDROID: GKI: Update abi_gki_aarch64_exynos
ANDROID: include/linux: add low power mode and fps to notifier
BACKPORT: scsi: ufs: Fix and simplify setup_xfer_req variant operation
ANDROID: GKI: enable USB UVC drivers
ANDROID: GKI: enabled CONFIG_USB_CONFIGFS_NCM=y
ANDROID: GKI: enable LEDS_CLASS_FLASH
Conflicts:
drivers/iommu/io-pgtable-arm.c
drivers/iommu/io-pgtable.c
include/linux/io-pgtable.h
scripts/checkpatch.pl
Change-Id: I6cbac7d913653388280a4c58f4f98e6f8b6b29f7
Signed-off-by: Ivaylo Georgiev <irgeorgiev@codeaurora.org>
Signed-off-by: Blagovest Kolenichev <bkolenichev@codeaurora.org>
|
||
Branden Bonaby
|
065a08cb14 |
soc: qcom: Create framework for qcom watchdogs
Separate core watchdog functionality from the qcom watchdog driver to create a framework for other qcom watchdogs. This will reduce the duplication of code for future watchdog drivers and allow them to take advantage of this framework. Change-Id: I1234af9d92de6841bd09157042a4fbba1745f11b Signed-off-by: Branden Bonaby <bbonaby@codeaurora.org> |
||
Sangmoon Kim
|
b519ac4237 |
ANDROID: vendor_hooks: add waiting information for blocked tasks
- Add the hook to get mutex/rwsem information that the tasks are waiting for. - Add the hook to print messages for sched_show_task. - ANDROID_VENDOR_DATA_ARRAY added to task_struct Bug: 162776704 Signed-off-by: Sangmoon Kim <sangmoon.kim@samsung.com> Change-Id: Ib436fbd8d0ad509c3b5a73ea8f5170e0761a13fd |
||
|
Blagovest Kolenichev
|
441301dd42 |
Merge android-5.4-stable.45 (a9a13ee) into msm-5.4
* refs/heads/tmp-a9a13ee: Linux 5.4.45 net: smsc911x: Fix runtime PM imbalance on error selftests: mlxsw: qos_mc_aware: Specify arping timeout as an integer net: ethernet: stmmac: Enable interface clocks on probe for IPQ806x net/ethernet/freescale: rework quiesce/activate for ucc_geth null_blk: return error for invalid zone size s390/mm: fix set_huge_pte_at() for empty ptes drm/edid: Add Oculus Rift S to non-desktop list net: bmac: Fix read of MAC address from ROM x86/mmiotrace: Use cpumask_available() for cpumask_var_t variables io_uring: initialize ctx->sqo_wait earlier i2c: altera: Fix race between xfer_msg and isr thread scsi: pm: Balance pm_only counter of request queue during system resume evm: Fix RCU list related warnings ARC: [plat-eznps]: Restrict to CONFIG_ISA_ARCOMPACT ARC: Fix ICCM & DCCM runtime size checks RDMA/qedr: Fix synchronization methods and memory leaks in qedr RDMA/qedr: Fix qpids xarray api used s390/ftrace: save traced function caller ASoC: intel - fix the card names spi: dw: use "smp_mb()" to avoid sending spi data error powerpc/xmon: Restrict when kernel is locked down powerpc/powernv: Avoid re-registration of imc debugfs directory scsi: hisi_sas: Check sas_port before using it drm/i915: fix port checks for MST support on gen >= 11 airo: Fix read overflows sending packets net: dsa: mt7530: set CPU port to fallback mode scsi: ufs: Release clock if DMA map fails media: staging: ipu3-imgu: Move alignment attribute to field media: Revert "staging: imgu: Address a compiler warning on alignment" mmc: fix compilation of user API kernel/relay.c: handle alloc_percpu returning NULL in relay_open mt76: mt76x02u: Add support for newer versions of the XBox One wifi adapter p54usb: add AirVasT USB stick device-id HID: i2c-hid: add Schneider SCL142ALM to descriptor override HID: multitouch: enable multi-input as a quirk for some devices HID: sony: Fix for broken buttons on DS3 USB dongles mm: Fix mremap not considering huge pmd devmap Revert "cgroup: Add memory barriers to plug cgroup_rstat_updated() race window" ANDROID: GKI: Enable SYS_HYPERVISOR ANDROID: Update ABI representation ANDROID: GKI: disable CONFIG_I2C_CHARDEV ANDROID: GKI: remove netfilter apis from supported symbols Linux 5.4.44 perf: Make perf able to build with latest libbfd netfilter: nf_conntrack_pptp: fix compilation warning with W=1 build netfilter: conntrack: Pass value of ctinfo to __nf_conntrack_update netfilter: conntrack: comparison of unsigned in cthelper confirmation bonding: Fix reference count leak in bond_sysfs_slave_add. net: dsa: declare lockless TX feature for slave ports ipv4: nexthop version of fib_info_nh_uses_dev nexthop: Expand nexthop_is_multipath in a few places nexthops: don't modify published nexthop groups nexthops: Move code from remove_nexthop_from_groups to remove_nh_grp_entry crypto: chelsio/chtls: properly set tp->lsndtime qlcnic: fix missing release in qlcnic_83xx_interrupt_test. xsk: Add overflow check for u64 division, stored into u32 ieee80211: Fix incorrect mask for default PE duration bnxt_en: Fix accumulation of bp->net_stats_prev. esp6: get the right proto for transport mode in esp6_gso_encap netfilter: nf_conntrack_pptp: prevent buffer overflows in debug code netfilter: nfnetlink_cthelper: unbreak userspace helper support netfilter: conntrack: make conntrack userspace helpers work again netfilter: ipset: Fix subcounter update skip netfilter: nft_reject_bridge: enable reject with bridge vlan ip_vti: receive ipip packet by calling ip_tunnel_rcv xfrm: fix error in comment xfrm: fix a NULL-ptr deref in xfrm_local_error xfrm: fix a warning in xfrm_policy_insert_list xfrm interface: fix oops when deleting a x-netns interface xfrm: call xfrm_output_gso when inner_protocol is set in xfrm_output xfrm: remove the xfrm_state_put call becofe going to out_reset xfrm: do pskb_pull properly in __xfrm_transport_prep xfrm: allow to accept packets with ipv6 NEXTHDR_HOP in xfrm_input copy_xstate_to_kernel(): don't leave parts of destination uninitialized x86/dma: Fix max PFN arithmetic overflow on 32 bit systems mac80211: mesh: fix discovery timer re-arming issue / crash x86/syscalls: Revert "x86/syscalls: Make __X32_SYSCALL_BIT be unsigned long" cfg80211: fix debugfs rename crash parisc: Fix kernel panic in mem_init() iommu: Fix reference count leak in iommu_group_alloc. gpio: fix locking open drain IRQ lines Revert "block: end bio with BLK_STS_AGAIN in case of non-mq devs and REQ_NOWAIT" include/asm-generic/topology.h: guard cpumask_of_node() macro argument fs/binfmt_elf.c: allocate initialized memory in fill_thread_core_info() mm: remove VM_BUG_ON(PageSlab()) from page_mapcount() mm,thp: stop leaking unreleased file pages IB/ipoib: Fix double free of skb in case of multicast traffic in CM mode drm/amd/display: drop cursor position check in atomic test RDMA/core: Fix double destruction of uobject ceph: flush release queue when handling caps for unknown inode libceph: ignore pool overlay and cache logic on redirects ALSA: hda/realtek - Add new codec supported for ALC287 ALSA: usb-audio: Quirks for Gigabyte TRX40 Aorus Master onboard audio clk: qcom: gcc: Fix parent for gpll0_out_even exec: Always set cap_ambient in cap_bprm_set_creds ALSA: usb-audio: mixer: volume quirk for ESS Technology Asus USB DAC ALSA: hda/realtek - Add a model for Thinkpad T570 without DAC workaround ALSA: hwdep: fix a left shifting 1 by 31 UB bug RDMA/pvrdma: Fix missing pci disable in pvrdma_pci_probe() gpio: bcm-kona: Fix return value of bcm_kona_gpio_probe() gpio: pxa: Fix return value of pxa_gpio_probe() mmc: block: Fix use-after-free issue for rpmb ARM: dts: bcm: HR2: Fix PPI interrupt types ARM: dts: bcm2835-rpi-zero-w: Fix led polarity ARM: dts/imx6q-bx50v3: Set display interface clock parents IB/qib: Call kobject_put() when kobject_init_and_add() fails gpu/drm: Ingenic: Fix opaque pointer casted to wrong type soc: mediatek: cmdq: return send msg error code arm64: dts: mt8173: fix vcodec-enc clock gpio: exar: Fix bad handling for ida_simple_get error path ARM: uaccess: fix DACR mismatch with nested exceptions ARM: uaccess: integrate uaccess_save and uaccess_restore ARM: uaccess: consolidate uaccess asm to asm/uaccess-asm.h ARM: 8970/1: decompressor: increase tag size Input: synaptics-rmi4 - fix error return code in rmi_driver_probe() Input: synaptics-rmi4 - really fix attn_data use-after-free Input: i8042 - add ThinkPad S230u to i8042 reset list Input: dlink-dir685-touchkeys - fix a typo in driver name Input: xpad - add custom init packet for Xbox One S controllers Input: evdev - call input_flush_device() on release(), not flush() Input: usbtouchscreen - add support for BonXeon TP drivers: net: hamradio: Fix suspicious RCU usage warning in bpqether.c samples: bpf: Fix build error csky: Fixup raw_copy_from_user() cifs: Fix null pointer check in cifs_read hwmon: (nct7904) Fix incorrect range of temperature limit registers csky: Fixup remove duplicate irq_disable csky: Fixup perf callchain unwind csky: Fixup msa highest 3 bits mask clk: ti: am33xx: fix RTC clock parent riscv: stacktrace: Fix undefined reference to `walk_stackframe' IB/i40iw: Remove bogus call to netdev_master_upper_dev_get() net: freescale: select CONFIG_FIXED_PHY where needed usb: gadget: legacy: fix redundant initialization warnings usb: phy: twl6030-usb: Fix a resource leak in an error handling path in 'twl6030_usb_probe()' usb: dwc3: pci: Enable extcon driver for Intel Merrifield cachefiles: Fix race between read_waiter and read_copier involving op->to_do drm/amdgpu: Use GEM obj reference for KFD BOs drm/amd/powerplay: perform PG ungate prior to CG ungate drm/amdgpu: drop unnecessary cancel_delayed_work_sync on PG ungate gfs2: Grab glock reference sooner in gfs2_add_revoke gfs2: move privileged user check to gfs2_quota_lock_check net: microchip: encx24j600: add missed kthread_stop ALSA: usb-audio: add mapping for ASRock TRX40 Creator gpio: tegra: mask GPIO IRQs during IRQ shutdown ARM: dts: rockchip: fix pinctrl sub nodename for spi in rk322x.dtsi ARM: dts: rockchip: swap clock-names of gpu nodes arm64: dts: rockchip: swap interrupts interrupt-names rk3399 gpu node arm64: dts: rockchip: fix status for &gmac2phy in rk3328-evb.dts ARM: dts: rockchip: fix phy nodename for rk3229-xms6 ARM: dts: rockchip: fix phy nodename for rk3228-evb mlxsw: spectrum: Fix use-after-free of split/unsplit/type_set in case reload fails net/mlx4_core: fix a memory leak bug. net: sun: fix missing release regions in cas_init_one(). net/tls: free record only on encryption error net/tls: fix encryption error checking net/mlx5: Annotate mutex destroy for root ns net/mlx5: Fix error flow in case of function_setup failure net/mlx5e: Update netdev txq on completions during closure net/mlx5: Fix memory leak in mlx5_events_init net/mlx5e: Fix inner tirs handling net/mlx5e: kTLS, Destroy key object after destroying the TIS tipc: block BH before using dst_cache sctp: Start shutdown on association restart if in SHUTDOWN-SENT state and socket is closed sctp: Don't add the shutdown timer if its already been added r8152: support additional Microsoft Surface Ethernet Adapter variant nexthop: Fix attribute checking for groups net/tls: fix race condition causing kernel panic net sched: fix reporting the first-time use timestamp net: revert "net: get rid of an signed integer overflow in ip_idents_reserve()" net: qrtr: Fix passing invalid reference to qrtr_local_enqueue() net: nlmsg_cancel() if put fails for nhmsg net: mvpp2: fix RX hashing for non-10G ports net/mlx5: Add command entry handling completion net: ipip: fix wrong address family in init error path net: inet_csk: Fix so_reuseport bind-address cache in tb->fast* __netif_receive_skb_core: pass skb by reference net: ethernet: ti: cpsw: fix ASSERT_RTNL() warning during suspend net: dsa: mt7530: fix roaming from DSA user ports net: don't return invalid table id error when we fall back to PF_UNSPEC dpaa_eth: fix usage as DSA master, try 3 ax25: fix setsockopt(SO_BINDTODEVICE) ANDROID: GKI: enable CONFIG_USB_DUMMY_HCD FROMGIT: USB: dummy-hcd: use configurable endpoint naming scheme ANDROID: GKI: enable a bunch of HID config options =y UPSTREAM: usb: raw-gadget: fix null-ptr-deref when reenabling endpoints UPSTREAM: usb: raw-gadget: documentation updates UPSTREAM: usb: raw-gadget: support stalling/halting/wedging endpoints UPSTREAM: usb: raw-gadget: fix gadget endpoint selection UPSTREAM: usb: raw-gadget: improve uapi headers comments UPSTREAM: usb: raw-gadget: fix return value of ep read ioctls UPSTREAM: usb: raw-gadget: fix raw_event_queue_fetch locking UPSTREAM: usb: raw-gadget: Fix copy_to/from_user() checks ANDROID: GKI: update abi definitions after enabling PCIEPORTBUS Revert "ANDROID: gki_defconfig: Drop CONFIG_PCIEPORTBUS" ANDROID: net: bpf: permit redirect from ingress L3 to egress L2 devices at near max mtu ANDROID: GKI: update abi definitions now that sdcardfs is gone Revert "ANDROID: sdcardfs: Enable modular sdcardfs" ANDROID: remove sdcardfs-needed exports from whitelist Revert "ANDROID: vfs: Add setattr2 for filesystems with per mount permissions" Revert "ANDROID: vfs: Add permission2 for filesystems with per mount permissions" Revert "ANDROID: vfs: add d_canonical_path for stacked filesystem support" Revert "ANDROID: fs: Restore vfs_path_lookup() export" ANDROID: sdcardfs: remove sdcardfs from system FROMGIT: driver core: Update device link status correctly for SYNC_STATE_ONLY links UPSTREAM: driver core: Fix handling of SYNC_STATE_ONLY + STATELESS device links UPSTREAM: driver core: Fix SYNC_STATE_ONLY device link implementation Revert "ANDROID: Incremental fs: Avoid continually recalculating hashes" ANDROID: remove unused variable Linux 5.4.43 sched/fair: Fix enqueue_task_fair() warning some more sched/fair: Fix reordering of enqueue/dequeue_task_fair() sched/fair: Reorder enqueue/dequeue_task_fair path tpm: check event log version before reading final events rxrpc: Fix ack discard rxrpc: Trace discarded ACKs iio: adc: stm32-dfsdm: fix device used to request dma iio: adc: stm32-dfsdm: Use dma_request_chan() instead dma_request_slave_channel() iio: adc: stm32-adc: fix device used to request dma iio: adc: stm32-adc: Use dma_request_chan() instead dma_request_slave_channel() x86/unwind/orc: Fix unwind_get_return_address_ptr() for inactive tasks flow_dissector: Drop BPF flow dissector prog ref on netns cleanup s390/kexec_file: fix initrd location for kdump kernel rxrpc: Fix a memory leak in rxkad_verify_response() rxrpc: Fix the excessive initial retransmission timeout kasan: disable branch tracing for core runtime rapidio: fix an error in get_user_pages_fast() error handling device-dax: don't leak kernel memory to user space after unloading kmem s390/kaslr: add support for R_390_JMP_SLOT relocation type s390/pci: Fix s390_mmio_read/write with MIO ipack: tpci200: fix error return code in tpci200_register() mei: release me_cl object reference tty: serial: add missing spin_lock_init for SiFive serial console misc: rtsx: Add short delay after exit from ASPM iio: adc: ti-ads8344: Fix channel selection iio: dac: vf610: Fix an error handling path in 'vf610_dac_probe()' iio: sca3000: Remove an erroneous 'get_device()' staging: greybus: Fix uninitialized scalar variable staging: kpc2000: fix error return code in kp2000_pcie_probe() staging: iio: ad2s1210: Fix SPI reading media: fdp1: Fix R-Car M3-N naming in debug message Revert "gfs2: Don't demote a glock until its revokes are written" kbuild: Remove debug info from kallsyms linking bpf: Avoid setting bpf insns pages read-only when prog is jited powerpc/64s: Disable STRICT_KERNEL_RWX powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE drm/i915: Propagate error from completed fences drm/i915/gvt: Init DPLL/DDI vreg for virtual display instead of inheritance. vsprintf: don't obfuscate NULL and error pointers dmaengine: owl: Use correct lock in owl_dma_get_pchan() dmaengine: dmatest: Restore default for channel drm/etnaviv: Fix a leak in submit_pin_objects() dmaengine: tegra210-adma: Fix an error handling path in 'tegra_adma_probe()' apparmor: Fix aa_label refcnt leak in policy_update apparmor: fix potential label refcnt leak in aa_change_profile apparmor: Fix use-after-free in aa_audit_rule_init drm/etnaviv: fix perfmon domain interation arm64: Fix PTRACE_SYSEMU semantics scsi: target: Put lun_ref at end of tmr processing scsi: qla2xxx: Do not log message when reading port speed via sysfs ALSA: hda/realtek - Add more fixup entries for Clevo machines ALSA: hda/realtek - Fix silent output on Gigabyte X570 Aorus Xtreme ALSA: pcm: fix incorrect hw_base increase ALSA: iec1712: Initialize STDSP24 properly when using the model=staudio option KVM: x86: Fix pkru save/restore when guest CR4.PKE=0, move it to x86.c ALSA: hda/realtek: Enable headset mic of ASUS UX581LV with ALC295 ALSA: hda/realtek - Enable headset mic of ASUS UX550GE with ALC295 ALSA: hda/realtek - Enable headset mic of ASUS GL503VM with ALC295 ALSA: hda/realtek: Add quirk for Samsung Notebook ALSA: hda/realtek - Add HP new mute led supported for ALC236 ALSA: hda/realtek - Add supported new mute Led for HP ALSA: hda: Manage concurrent reg access more properly ALSA: hda: patch_realtek: fix empty macro usage in if block ALSA: hda - constify and cleanup static NodeID tables scripts/gdb: repair rb_first() and rb_last() ARM: futex: Address build warning KVM: selftests: Fix build for evmcs.h drm/amd/display: Prevent dpcd reads with passive dongles iommu/amd: Call domain_flush_complete() in update_domain() platform/x86: asus-nb-wmi: Do not load on Asus T100TA and T200TA USB: core: Fix misleading driver bug report stmmac: fix pointer check after utilization in stmmac_interrupt ceph: fix double unlock in handle_cap_export() HID: quirks: Add HID_QUIRK_NO_INIT_REPORTS quirk for Dell K12A keyboard-dock gtp: set NLM_F_MULTI flag in gtp_genl_dump_pdp() x86/apic: Move TSC deadline timer debug printk ftrace/selftest: make unresolved cases cause failure if --fail-unresolved set ibmvnic: Skip fatal error reset after passive init x86/mm/cpa: Flush direct map alias during cpa HID: i2c-hid: reset Synaptics SYNA2393 on resume scsi: ibmvscsi: Fix WARN_ON during event pool release net/ena: Fix build warning in ena_xdp_set() component: Silence bind error on -EPROBE_DEFER aquantia: Fix the media type of AQC100 ethernet controller in the driver vhost/vsock: fix packet delivery order to monitoring devices configfs: fix config_item refcnt leak in configfs_rmdir() scsi: qla2xxx: Delete all sessions before unregister local nvme port scsi: qla2xxx: Fix hang when issuing nvme disconnect-all in NPIV HID: alps: ALPS_1657 is too specific; use U1_UNICORN_LEGACY instead HID: alps: Add AUI1657 device ID HID: multitouch: add eGalaxTouch P80H84 support gcc-common.h: Update for GCC 10 net: drop_monitor: use IS_REACHABLE() to guard net_dm_hw_report() kbuild: avoid concurrency issue in parallel building dtbs and dtbs_check mtd: Fix mtd not registered due to nvmem name collision afs: Don't unlock fetched data pages until the op completes successfully ubi: Fix seq_file usage in detailed_erase_block_info debugfs file i2c: mux: demux-pinctrl: Fix an error handling path in 'i2c_demux_pinctrl_probe()' evm: Fix a small race in init_desc() iommu/amd: Fix over-read of ACPI UID from IVRS table i2c: fix missing pm_runtime_put_sync in i2c_device_probe ubifs: remove broken lazytime support fix multiplication overflow in copy_fdtable() mtd: spinand: Propagate ECC information to the MTD structure ACPI: EC: PM: Avoid flushing EC work when EC GPE is inactive ubifs: fix wrong use of crypto_shash_descsize() ima: Fix return value of ima_write_policy() evm: Check also if *tfm is an error pointer in init_desc() ima: Set file->f_mode instead of file->f_flags in ima_calc_file_hash() KVM: SVM: Fix potential memory leak in svm_cpu_init() i2c: dev: Fix the race between the release of i2c_dev and cdev ANDROID: scsi: ufs: Handle clocks when lrbp fails ANDROID: ABI/Whitelist: add erofs whitelist to unisoc whitelist ANDROID: GKI: Update abi for 5.4.42 update UPSTREAM: serial: samsung: 32-bit access for TX/RX hold registers BACKPORT: serial: samsung: dt-bindings: Add reg-io-width compatible UPSTREAM: serial: samsung: Replace rd_regb/wr_regb with rd_reg/wr_reg ANDROID: fscrypt: handle direct I/O with IV_INO_LBLK_32 BACKPORT: FROMLIST: fscrypt: add support for IV_INO_LBLK_32 policies Linux 5.4.42 bpf: Test_progs, fix test_get_stack_rawtp_err.c build selftest/bpf: fix backported test_select_reuseport selftest changes libbpf: Extract and generalize CPU mask parsing logic Makefile: disallow data races on gcc-10 as well KVM: x86: Fix off-by-one error in kvm_vcpu_ioctl_x86_setup_mce bpf: Fix sk_psock refcnt leak when receiving message SUNRPC: Revert |
||
Rohit Vaswani
|
bf98d686be |
lib: spinlock: Cause a watchdog bite on spin_dump
Currently we cause a BUG_ON once a spin_bug is detected, but that causes a whole lot of processing and the other CPUs would have proceeded to perform other actions and the state of the system is moved by the time we can analyze it. Provide an option to trigger a watchdog bite instead so that we can get the traces as close to the issue as possible. Change-Id:Ic8d692ebd02c6940a3b4e5798463744db20b0026 Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org> Signed-off-by: Rishabh Bhatnagar <rishabhb@codeaurora.org> [rananta@codeaurora.org: Resolved trivial merge conflicts] Signed-off-by: Raghavendra Rao Ananta <rananta@codeaurora.org> |
||
Paul E. McKenney
|
0c72ec11d8 |
locktorture: Print ratio of acquisitions, not failures
commit 80c503e0e68fbe271680ab48f0fe29bc034b01b7 upstream.
The __torture_print_stats() function in locktorture.c carefully
initializes local variable "min" to statp[0].n_lock_acquired, but
then compares it to statp[i].n_lock_fail. Given that the .n_lock_fail
field should normally be zero, and given the initialization, it seems
reasonable to display the maximum and minimum number acquisitions
instead of miscomputing the maximum and minimum number of failures.
This commit therefore switches from failures to acquisitions.
And this turns out to be not only a day-zero bug, but entirely my
own fault. I hate it when that happens!
Fixes:
|
||
Boqun Feng
|
bd9afea9bd |
locking/lockdep: Avoid recursion in lockdep_count_{for,back}ward_deps()
[ Upstream commit 25016bd7f4caf5fc983bbab7403d08e64cba3004 ] Qian Cai reported a bug when PROVE_RCU_LIST=y, and read on /proc/lockdep triggered a warning: [ ] DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled) ... [ ] Call Trace: [ ] lock_is_held_type+0x5d/0x150 [ ] ? rcu_lockdep_current_cpu_online+0x64/0x80 [ ] rcu_read_lock_any_held+0xac/0x100 [ ] ? rcu_read_lock_held+0xc0/0xc0 [ ] ? __slab_free+0x421/0x540 [ ] ? kasan_kmalloc+0x9/0x10 [ ] ? __kmalloc_node+0x1d7/0x320 [ ] ? kvmalloc_node+0x6f/0x80 [ ] __bfs+0x28a/0x3c0 [ ] ? class_equal+0x30/0x30 [ ] lockdep_count_forward_deps+0x11a/0x1a0 The warning got triggered because lockdep_count_forward_deps() call __bfs() without current->lockdep_recursion being set, as a result a lockdep internal function (__bfs()) is checked by lockdep, which is unexpected, and the inconsistency between the irq-off state and the state traced by lockdep caused the warning. Apart from this warning, lockdep internal functions like __bfs() should always be protected by current->lockdep_recursion to avoid potential deadlocks and data inconsistency, therefore add the current->lockdep_recursion on-and-off section to protect __bfs() in both lockdep_count_forward_deps() and lockdep_count_backward_deps() Reported-by: Qian Cai <cai@lca.pw> Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lkml.kernel.org/r/20200312151258.128036-1-boqun.feng@gmail.com Signed-off-by: Sasha Levin <sashal@kernel.org> |
||
Waiman Long
|
2482da809f |
locking/lockdep: Fix lockdep_stats indentation problem
commit a030f9767da1a6bbcec840fc54770eb11c2414b6 upstream.
It was found that two lines in the output of /proc/lockdep_stats have
indentation problem:
# cat /proc/lockdep_stats
:
in-process chains: 25057
stack-trace entries: 137827 [max: 524288]
number of stack traces: 7973
number of stack hash chains: 6355
combined max dependencies: 1356414598
hardirq-safe locks: 57
hardirq-unsafe locks: 1286
:
All the numbers displayed in /proc/lockdep_stats except the two stack
trace numbers are formatted with a field with of 11. To properly align
all the numbers, a field width of 11 is now added to the two stack
trace numbers.
Fixes:
|
||
|
Waiman Long
|
b90976941e |
locking/lockdep: Fix buffer overrun problem in stack_trace[]
commit d91f3057263ceb691ef527e71b41a56b17f6c869 upstream.
If the lockdep code is really running out of the stack_trace entries,
it is likely that buffer overrun can happen and the data immediately
after stack_trace[] will be corrupted.
If there is less than LOCK_TRACE_SIZE_IN_LONGS entries left before
the call to save_trace(), the max_entries computation will leave it
with a very large positive number because of its unsigned nature. The
subsequent call to stack_trace_save() will then corrupt the data after
stack_trace[]. Fix that by changing max_entries to a signed integer
and check for negative value before calling stack_trace_save().
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Fixes:
|
||
|
Waiman Long
|
710d9fd2f4 |
locking/rwsem: Fix kernel crash when spinning on RWSEM_OWNER_UNKNOWN
commit 39e7234f00bc93613c086ae42d852d5f4147120a upstream. The commit |
||
Marco Elver
|
c120c3dbeb |
locking/spinlock/debug: Fix various data races
[ Upstream commit 1a365e822372ba24c9da0822bc583894f6f3d821 ] This fixes various data races in spinlock_debug. By testing with KCSAN, it is observable that the console gets spammed with data races reports, suggesting these are extremely frequent. Example data race report: read to 0xffff8ab24f403c48 of 4 bytes by task 221 on cpu 2: debug_spin_lock_before kernel/locking/spinlock_debug.c:85 [inline] do_raw_spin_lock+0x9b/0x210 kernel/locking/spinlock_debug.c:112 __raw_spin_lock include/linux/spinlock_api_smp.h:143 [inline] _raw_spin_lock+0x39/0x40 kernel/locking/spinlock.c:151 spin_lock include/linux/spinlock.h:338 [inline] get_partial_node.isra.0.part.0+0x32/0x2f0 mm/slub.c:1873 get_partial_node mm/slub.c:1870 [inline] <snip> write to 0xffff8ab24f403c48 of 4 bytes by task 167 on cpu 3: debug_spin_unlock kernel/locking/spinlock_debug.c:103 [inline] do_raw_spin_unlock+0xc9/0x1a0 kernel/locking/spinlock_debug.c:138 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:159 [inline] _raw_spin_unlock_irqrestore+0x2d/0x50 kernel/locking/spinlock.c:191 spin_unlock_irqrestore include/linux/spinlock.h:393 [inline] free_debug_processing+0x1b3/0x210 mm/slub.c:1214 __slab_free+0x292/0x400 mm/slub.c:2864 <snip> As a side-effect, with KCSAN, this eventually locks up the console, most likely due to deadlock, e.g. .. -> printk lock -> spinlock_debug -> KCSAN detects data race -> kcsan_print_report() -> printk lock -> deadlock. This fix will 1) avoid the data races, and 2) allow using lock debugging together with KCSAN. Reported-by: Qian Cai <cai@lca.pw> Signed-off-by: Marco Elver <elver@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Link: https://lkml.kernel.org/r/20191120155715.28089-1-elver@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org> |
||
Wanpeng Li
|
89340d0935 |
Revert "locking/pvqspinlock: Don't wait if vCPU is preempted"
This patch reverts commit |
||
Linus Torvalds
|
7e67a85999 |
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
- MAINTAINERS: Add Mark Rutland as perf submaintainer, Juri Lelli and
Vincent Guittot as scheduler submaintainers. Add Dietmar Eggemann,
Steven Rostedt, Ben Segall and Mel Gorman as scheduler reviewers.
As perf and the scheduler is getting bigger and more complex,
document the status quo of current responsibilities and interests,
and spread the review pain^H^H^H^H fun via an increase in the Cc:
linecount generated by scripts/get_maintainer.pl. :-)
- Add another series of patches that brings the -rt (PREEMPT_RT) tree
closer to mainline: split the monolithic CONFIG_PREEMPT dependencies
into a new CONFIG_PREEMPTION category that will allow the eventual
introduction of CONFIG_PREEMPT_RT. Still a few more hundred patches
to go though.
- Extend the CPU cgroup controller with uclamp.min and uclamp.max to
allow the finer shaping of CPU bandwidth usage.
- Micro-optimize energy-aware wake-ups from O(CPUS^2) to O(CPUS).
- Improve the behavior of high CPU count, high thread count
applications running under cpu.cfs_quota_us constraints.
- Improve balancing with SCHED_IDLE (SCHED_BATCH) tasks present.
- Improve CPU isolation housekeeping CPU allocation NUMA locality.
- Fix deadline scheduler bandwidth calculations and logic when cpusets
rebuilds the topology, or when it gets deadline-throttled while it's
being offlined.
- Convert the cpuset_mutex to percpu_rwsem, to allow it to be used from
setscheduler() system calls without creating global serialization.
Add new synchronization between cpuset topology-changing events and
the deadline acceptance tests in setscheduler(), which were broken
before.
- Rework the active_mm state machine to be less confusing and more
optimal.
- Rework (simplify) the pick_next_task() slowpath.
- Improve load-balancing on AMD EPYC systems.
- ... and misc cleanups, smaller fixes and improvements - please see
the Git log for more details.
* 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (53 commits)
sched/psi: Correct overly pessimistic size calculation
sched/fair: Speed-up energy-aware wake-ups
sched/uclamp: Always use 'enum uclamp_id' for clamp_id values
sched/uclamp: Update CPU's refcount on TG's clamp changes
sched/uclamp: Use TG's clamps to restrict TASK's clamps
sched/uclamp: Propagate system defaults to the root group
sched/uclamp: Propagate parent clamps
sched/uclamp: Extend CPU's cgroup controller
sched/topology: Improve load balancing on AMD EPYC systems
arch, ia64: Make NUMA select SMP
sched, perf: MAINTAINERS update, add submaintainers and reviewers
sched/fair: Use rq_lock/unlock in online_fair_sched_group
cpufreq: schedutil: fix equation in comment
sched: Rework pick_next_task() slow-path
sched: Allow put_prev_task() to drop rq->lock
sched/fair: Expose newidle_balance()
sched: Add task_struct pointer to sched_class::set_curr_task
sched: Rework CPU hotplug task selection
sched/{rt,deadline}: Fix set_next_task vs pick_next_task
sched: Fix kerneldoc comment for ia64_set_curr_task
...
|
||
|
Linus Torvalds
|
c7eba51cfd |
Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar: - improve rwsem scalability - add uninitialized rwsem debugging check - reduce lockdep's stacktrace memory usage and add diagnostics - misc cleanups, code consolidation and constification * 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: mutex: Fix up mutex_waiter usage locking/mutex: Use mutex flags macro instead of hard code locking/mutex: Make __mutex_owner static to mutex.c locking/qspinlock,x86: Clarify virt_spin_lock_key locking/rwsem: Check for operations on an uninitialized rwsem locking/rwsem: Make handoff writer optimistically spin on owner locking/lockdep: Report more stack trace statistics locking/lockdep: Reduce space occupied by stack traces stacktrace: Constify 'entries' arguments locking/lockdep: Make it clear that what lock_class::key points at is not modified |
||
Ingo Molnar
|
563c4f85f9 |
Merge branch 'sched/rt' into sched/core, to pick up -rt changes
Pick up the first couple of patches working towards PREEMPT_RT. Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
Peter Zijlstra
|
e57d143091 |
mutex: Fix up mutex_waiter usage
The patch moving bits into mutex.c was a little too much; by also
moving struct mutex_waiter a few less common CONFIGs would no longer
build.
Fixes:
|
||
Mukesh Ojha
|
a037d26922 |
locking/mutex: Use mutex flags macro instead of hard code
Use the mutex flag macro instead of hard code value inside __mutex_owner(). Signed-off-by: Mukesh Ojha <mojha@codeaurora.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: mingo@redhat.com Cc: will@kernel.org Link: https://lkml.kernel.org/r/1564585504-3543-2-git-send-email-mojha@codeaurora.org |
||
|
Mukesh Ojha
|
5f35d5a66b |
locking/mutex: Make __mutex_owner static to mutex.c
__mutex_owner() should only be used by the mutex api's. So, to put this restiction let's move the __mutex_owner() function definition from linux/mutex.h to mutex.c file. There exist functions that uses __mutex_owner() like mutex_is_locked() and mutex_trylock_recursive(), So to keep legacy thing intact move them as well and export them. Move mutex_waiter structure also to keep it private to the file. Signed-off-by: Mukesh Ojha <mojha@codeaurora.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: mingo@redhat.com Cc: will@kernel.org Link: https://lkml.kernel.org/r/1564585504-3543-1-git-send-email-mojha@codeaurora.org |
||
Davidlohr Bueso
|
fce45cd411 |
locking/rwsem: Check for operations on an uninitialized rwsem
Currently rwsems is the only locking primitive that lacks this debug feature. Add it under CONFIG_DEBUG_RWSEMS and do the magic checking in the locking fastpath (trylock) operation such that we cover all cases. The unlocking part is pretty straightforward. Signed-off-by: Davidlohr Bueso <dbueso@suse.de> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Waiman Long <longman@redhat.com> Cc: mingo@kernel.org Cc: Davidlohr Bueso <dave@stgolabs.net> Link: https://lkml.kernel.org/r/20190729044735.9632-1-dave@stgolabs.net |
||
|
Waiman Long
|
91d2a812df |
locking/rwsem: Make handoff writer optimistically spin on owner
When the handoff bit is set by a writer, no other tasks other than
the setting writer itself is allowed to acquire the lock. If the
to-be-handoff'ed writer goes to sleep, there will be a wakeup latency
period where the lock is free, but no one can acquire it. That is less
than ideal.
To reduce that latency, the handoff writer will now optimistically spin
on the owner if it happens to be a on-cpu writer. It will spin until
it releases the lock and the to-be-handoff'ed writer can then acquire
the lock immediately without any delay. Of course, if the owner is not
a on-cpu writer, the to-be-handoff'ed writer will have to sleep anyway.
The optimistic spinning code is also modified to not stop spinning
when the handoff bit is set. This will prevent an occasional setting of
handoff bit from causing a bunch of optimistic spinners from entering
into the wait queue causing significant reduction in throughput.
On a 1-socket 22-core 44-thread Skylake system, the AIM7 shared_memory
workload was run with 7000 users. The throughput (jobs/min) of the
following kernels were as follows:
1) 5.2-rc6
- 8,092,486
2) 5.2-rc6 + tip's rwsem patches
- 7,567,568
3) 5.2-rc6 + tip's rwsem patches + this patch
- 7,954,545
Using perf-record(1), the %cpu time used by rwsem_down_write_slowpath(),
rwsem_down_write_failed() and their callees for the 3 kernels were 1.70%,
5.46% and 2.08% respectively.
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: x86@kernel.org
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: huang ying <huang.ying.caritas@gmail.com>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Link: https://lkml.kernel.org/r/20190625143913.24154-1-longman@redhat.com
|
||
Paul E. McKenney
|
519248f36d |
lockdep: Make print_lock() address visible
Security is a wonderful thing, but so is the ability to debug based on lockdep warnings. This commit therefore makes lockdep lock addresses visible in the clear. Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com> |
||
Matthew Wilcox (Oracle)
|
7b3c92b85a |
sched/core: Convert get_task_struct() to return the task
Returning the pointer that was passed in allows us to write slightly more idiomatic code. Convert a few users. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lkml.kernel.org/r/20190704221323.24290-1-willy@infradead.org Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
Bart Van Assche
|
8c779229d0 |
locking/lockdep: Report more stack trace statistics
Report the number of stack traces and the number of stack trace hash chains. These two numbers are useful because these allow to estimate the number of stack trace hash collisions. Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Waiman Long <longman@redhat.com> Cc: Will Deacon <will.deacon@arm.com> Link: https://lkml.kernel.org/r/20190722182443.216015-5-bvanassche@acm.org Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Bart Van Assche
|
12593b7467 |
locking/lockdep: Reduce space occupied by stack traces
Although commit
|
||
|
Bart Van Assche
|
364f6afc4f |
locking/lockdep: Make it clear that what lock_class::key points at is not modified
This patch does not change the behavior of the lockdep code. Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Waiman Long <longman@redhat.com> Cc: Will Deacon <will.deacon@arm.com> Link: https://lkml.kernel.org/r/20190722182443.216015-2-bvanassche@acm.org Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
Sebastian Andrzej Siewior
|
6c11c6e3d5 |
locking/mutex: Test for initialized mutex
An uninitialized/ zeroed mutex will go unnoticed because there is no check for it. There is a magic check in the unlock's slowpath path which might go unnoticed if the unlock happens in the fastpath. Add a ->magic check early in the mutex_lock() and mutex_trylock() path. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lkml.kernel.org/r/20190703092125.lsdf4gpsh2plhavb@linutronix.de Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
Arnd Bergmann
|
30a35f79fa |
locking/lockdep: Clean up #ifdef checks
As Will Deacon points out, CONFIG_PROVE_LOCKING implies TRACE_IRQFLAGS,
so the conditions I added in the previous patch, and some others in the
same file can be simplified by only checking for the former.
No functional change.
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Will Deacon <will@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Waiman Long <longman@redhat.com>
Cc: Yuyang Du <duyuyang@gmail.com>
Fixes:
|
||
|
Arnd Bergmann
|
68037aa782 |
locking/lockdep: Hide unused 'class' variable
The usage is now hidden in an #ifdef, so we need to move
the variable itself in there as well to avoid this warning:
kernel/locking/lockdep_proc.c:203:21: error: unused variable 'class' [-Werror,-Wunused-variable]
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Qian Cai <cai@lca.pw>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yuyang Du <duyuyang@gmail.com>
Cc: frederic@kernel.org
Fixes:
|
||
|
Peter Zijlstra
|
6ffddfb9e1 |
locking/rwsem: Add ACQUIRE comments
Since we just reviewed read_slowpath for ACQUIRE correctness, add a few coments to retain our findings. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Peter Zijlstra
|
99143f82a2 |
lcoking/rwsem: Add missing ACQUIRE to read_slowpath sleep loop
While reviewing another read_slowpath patch, both Will and I noticed
another missing ACQUIRE, namely:
X = 0;
CPU0 CPU1
rwsem_down_read()
for (;;) {
set_current_state(TASK_UNINTERRUPTIBLE);
X = 1;
rwsem_up_write();
rwsem_mark_wake()
atomic_long_add(adjustment, &sem->count);
smp_store_release(&waiter->task, NULL);
if (!waiter.task)
break;
...
}
r = X;
Allows 'r == 0'.
Reported-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reported-by: Will Deacon <will@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Will Deacon <will@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Fixes:
|
||
Jan Stancek
|
e1b98fa316 |
locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty
LTP mtest06 has been observed to occasionally hit "still mapped when
deleted" and following BUG_ON on arm64.
The extra mapcount originated from pagefault handler, which handled
pagefault for vma that has already been detached. vma is detached
under mmap_sem write lock by detach_vmas_to_be_unmapped(), which
also invalidates vmacache.
When the pagefault handler (under mmap_sem read lock) calls
find_vma(), vmacache_valid() wrongly reports vmacache as valid.
After rwsem down_read() returns via 'queue empty' path (as of v5.2),
it does so without an ACQUIRE on sem->count:
down_read()
__down_read()
rwsem_down_read_failed()
__rwsem_down_read_failed_common()
raw_spin_lock_irq(&sem->wait_lock);
if (list_empty(&sem->wait_list)) {
if (atomic_long_read(&sem->count) >= 0) {
raw_spin_unlock_irq(&sem->wait_lock);
return sem;
The problem can be reproduced by running LTP mtest06 in a loop and
building the kernel (-j $NCPUS) in parallel. It does reproduces since
v4.20 on arm64 HPE Apollo 70 (224 CPUs, 256GB RAM, 2 nodes). It
triggers reliably in about an hour.
The patched kernel ran fine for 10+ hours.
Signed-off-by: Jan Stancek <jstancek@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Will Deacon <will@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dbueso@suse.de
Fixes:
|
||
|
Waiman Long
|
7813430057 |
locking/rwsem: Don't call owner_on_cpu() on read-owner
For writer, the owner value is cleared on unlock. For reader, it is
left intact on unlock for providing better debugging aid on crash dump
and the unlock of one reader may not mean the lock is free.
As a result, the owner_on_cpu() shouldn't be used on read-owner
as the task pointer value may not be valid and it might have
been freed. That is the case in rwsem_spin_on_owner(), but not in
rwsem_can_spin_on_owner(). This can lead to use-after-free error from
KASAN. For example,
BUG: KASAN: use-after-free in rwsem_down_write_slowpath
(/home/miguel/kernel/linux/kernel/locking/rwsem.c:669
/home/miguel/kernel/linux/kernel/locking/rwsem.c:1125)
Fix this by checking for RWSEM_READER_OWNED flag before calling
owner_on_cpu().
Reported-by: Luis Henriques <lhenriques@suse.com>
Tested-by: Luis Henriques <lhenriques@suse.com>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: huang ying <huang.ying.caritas@gmail.com>
Fixes:
|
||
Mauro Carvalho Chehab
|
387b14684f |
docs: locking: convert docs to ReST and rename to *.rst
Convert the locking documents to ReST and add them to the kernel development book where it belongs. Most of the stuff here is just to make Sphinx to properly parse the text file, as they're already in good shape, not requiring massive changes in order to be parsed. The conversion is actually: - add blank lines and identation in order to identify paragraphs; - fix tables markups; - add some lists markups; - mark literal blocks; - adjust title markups. At its new index.rst, let's add a :orphan: while this is not linked to the main index.rst file, in order to avoid build warnings. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> Acked-by: Federico Vaga <federico.vaga@vaga.pv.it> |
||
Yuyang Du
|
68d41d8c94 |
locking/lockdep: Fix lock used or unused stats error
The stats variable nr_unused_locks is incremented every time a new lock class is register and decremented when the lock is first used in __lock_acquire(). And after all, it is shown and checked in lockdep_stats. However, under configurations that either CONFIG_TRACE_IRQFLAGS or CONFIG_PROVE_LOCKING is not defined: The commit: |
||
|
Linus Torvalds
|
e192832869 |
Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar: "The main changes in this cycle are: - rwsem scalability improvements, phase #2, by Waiman Long, which are rather impressive: "On a 2-socket 40-core 80-thread Skylake system with 40 reader and writer locking threads, the min/mean/max locking operations done in a 5-second testing window before the patchset were: 40 readers, Iterations Min/Mean/Max = 1,807/1,808/1,810 40 writers, Iterations Min/Mean/Max = 1,807/50,344/151,255 After the patchset, they became: 40 readers, Iterations Min/Mean/Max = 30,057/31,359/32,741 40 writers, Iterations Min/Mean/Max = 94,466/95,845/97,098" There's a lot of changes to the locking implementation that makes it similar to qrwlock, including owner handoff for more fair locking. Another microbenchmark shows how across the spectrum the improvements are: "With a locking microbenchmark running on 5.1 based kernel, the total locking rates (in kops/s) on a 2-socket Skylake system with equal numbers of readers and writers (mixed) before and after this patchset were: # of Threads Before Patch After Patch ------------ ------------ ----------- 2 2,618 4,193 4 1,202 3,726 8 802 3,622 16 729 3,359 32 319 2,826 64 102 2,744" The changes are extensive and the patch-set has been through several iterations addressing various locking workloads. There might be more regressions, but unless they are pathological I believe we want to use this new implementation as the baseline going forward. - jump-label optimizations by Daniel Bristot de Oliveira: the primary motivation was to remove IPI disturbance of isolated RT-workload CPUs, which resulted in the implementation of batched jump-label updates. Beyond the improvement of the real-time characteristics kernel, in one test this patchset improved static key update overhead from 57 msecs to just 1.4 msecs - which is a nice speedup as well. - atomic64_t cross-arch type cleanups by Mark Rutland: over the last ~10 years of atomic64_t existence the various types used by the APIs only had to be self-consistent within each architecture - which means they became wildly inconsistent across architectures. Mark puts and end to this by reworking all the atomic64 implementations to use 's64' as the base type for atomic64_t, and to ensure that this type is consistently used for parameters and return values in the API, avoiding further problems in this area. - A large set of small improvements to lockdep by Yuyang Du: type cleanups, output cleanups, function return type and othr cleanups all around the place. - A set of percpu ops cleanups and fixes by Peter Zijlstra. - Misc other changes - please see the Git log for more details" * 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (82 commits) locking/lockdep: increase size of counters for lockdep statistics locking/atomics: Use sed(1) instead of non-standard head(1) option locking/lockdep: Move mark_lock() inside CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING x86/jump_label: Make tp_vec_nr static x86/percpu: Optimize raw_cpu_xchg() x86/percpu, sched/fair: Avoid local_clock() x86/percpu, x86/irq: Relax {set,get}_irq_regs() x86/percpu: Relax smp_processor_id() x86/percpu: Differentiate this_cpu_{}() and __this_cpu_{}() locking/rwsem: Guard against making count negative locking/rwsem: Adaptive disabling of reader optimistic spinning locking/rwsem: Enable time-based spinning on reader-owned rwsem locking/rwsem: Make rwsem->owner an atomic_long_t locking/rwsem: Enable readers spinning on writer locking/rwsem: Clarify usage of owner's nonspinaable bit locking/rwsem: Wake up almost all readers in wait queue locking/rwsem: More optimal RT task handling of null owner locking/rwsem: Always release wait_lock before waking up tasks locking/rwsem: Implement lock handoff to prevent lock starvation locking/rwsem: Make rwsem_spin_on_owner() return owner state ... |
||
|
Ingo Molnar
|
83086d654d |
Merge branch 'for-mingo' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
Pull rcu/next + tools/memory-model changes from Paul E. McKenney: - RCU flavor consolidation cleanups and optmizations - Documentation updates - Miscellaneous fixes - SRCU updates - RCU-sync flavor consolidation - Torture-test updates - Linux-kernel memory-consistency-model updates, most notably the addition of plain C-language accesses Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
Kobe Wu
|
9156e54576 |
locking/lockdep: increase size of counters for lockdep statistics
When system has been running for a long time, signed integer counters are not enough for some lockdep statistics. Using unsigned long counters can satisfy the requirement. Besides, most of lockdep statistics are unsigned. It is better to use unsigned int instead of int. Remove unused variables. - max_recursion_depth - nr_cyclic_check_recursions - nr_find_usage_forwards_recursions - nr_find_usage_backwards_recursions Signed-off-by: Kobe Wu <kobe-cp.wu@mediatek.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: <linux-mediatek@lists.infradead.org> Cc: <wsd_upstream@mediatek.com> Cc: Eason Lin <eason-yh.lin@mediatek.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Link: https://lkml.kernel.org/r/1561365348-16050-1-git-send-email-kobe-cp.wu@mediatek.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Arnd Bergmann
|
886532aee3 |
locking/lockdep: Move mark_lock() inside CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING
The last cleanup patch triggered another issue, as now another function
should be moved into the same section:
kernel/locking/lockdep.c:3580:12: error: 'mark_lock' defined but not used [-Werror=unused-function]
static int mark_lock(struct task_struct *curr, struct held_lock *this,
Move mark_lock() into the same #ifdef section as its only caller, and
remove the now-unused mark_lock_irq() stub helper.
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Yuyang Du <duyuyang@gmail.com>
Fixes:
|
||
|
Paul E. McKenney
|
11ca7a9d54 |
Merge branches 'consolidate.2019.05.28a', 'doc.2019.05.28a', 'fixes.2019.06.13a', 'srcu.2019.05.28a', 'sync.2019.05.28a' and 'torture.2019.05.28a' into HEAD
consolidate.2019.05.28a: RCU flavor consolidation cleanups and optmizations. doc.2019.05.28a: Documentation updates. fixes.2019.06.13a: Miscellaneous fixes. srcu.2019.05.28a: SRCU updates. sync.2019.05.28a: RCU-sync flavor consolidation. torture.2019.05.28a: Torture-test updates. |
||
|
Waiman Long
|
a15ea1a35f |
locking/rwsem: Guard against making count negative
The upper bits of the count field is used as reader count. When sufficient number of active readers are present, the most significant bit will be set and the count becomes negative. If the number of active readers keep on piling up, we may eventually overflow the reader counts. This is not likely to happen unless the number of bits reserved for reader count is reduced because those bits are need for other purpose. To prevent this count overflow from happening, the most significant bit is now treated as a guard bit (RWSEM_FLAG_READFAIL). Read-lock attempts will now fail for both the fast and slow paths whenever this bit is set. So all those extra readers will be put to sleep in the wait list. Wakeup will not happen until the reader count reaches 0. Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: huang ying <huang.ying.caritas@gmail.com> Link: https://lkml.kernel.org/r/20190520205918.22251-17-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Waiman Long
|
5cfd92e12e |
locking/rwsem: Adaptive disabling of reader optimistic spinning
Reader optimistic spinning is helpful when the reader critical section
is short and there aren't that many readers around. It makes readers
relatively more preferred than writers. When a writer times out spinning
on a reader-owned lock and set the nospinnable bits, there are two main
reasons for that.
1) The reader critical section is long, perhaps the task sleeps after
acquiring the read lock.
2) There are just too many readers contending the lock causing it to
take a while to service all of them.
In the former case, long reader critical section will impede the progress
of writers which is usually more important for system performance.
In the later case, reader optimistic spinning tends to make the reader
groups that contain readers that acquire the lock together smaller
leading to more of them. That may hurt performance in some cases. In
other words, the setting of nonspinnable bits indicates that reader
optimistic spinning may not be helpful for those workloads that cause it.
Therefore, any writers that have observed the setting of the writer
nonspinnable bit for a given rwsem after they fail to acquire the lock
via optimistic spinning will set the reader nonspinnable bit once they
acquire the write lock. Similarly, readers that observe the setting
of reader nonspinnable bit at slowpath entry will also set the reader
nonspinnable bit when they acquire the read lock via the wakeup path.
Once the reader nonspinnable bit is on, it will only be reset when
a writer is able to acquire the rwsem in the fast path or somehow a
reader or writer in the slowpath doesn't observe the nonspinable bit.
This is to discourage reader optmistic spinning on that particular
rwsem and make writers more preferred. This adaptive disabling of reader
optimistic spinning will alleviate some of the negative side effect of
this feature.
In addition, this patch tries to make readers in the spinning queue
follow the phase-fair principle after quitting optimistic spinning
by checking if another reader has somehow acquired a read lock after
this reader enters the optimistic spinning queue. If so and the rwsem
is still reader-owned, this reader is in the right read-phase and can
attempt to acquire the lock.
On a 2-socket 40-core 80-thread Skylake system, the page_fault1 test of
the will-it-scale benchmark was run with various number of threads. The
number of operations done before reader optimistic spinning patches,
this patch and after this patch were:
Threads Before rspin Before patch After patch %change
------- ------------ ------------ ----------- -------
20 5541068
|
||
|
Waiman Long
|
7d43f1ce9d |
locking/rwsem: Enable time-based spinning on reader-owned rwsem
When the rwsem is owned by reader, writers stop optimistic spinning
simply because there is no easy way to figure out if all the readers
are actively running or not. However, there are scenarios where
the readers are unlikely to sleep and optimistic spinning can help
performance.
This patch provides a simple mechanism for spinning on a reader-owned
rwsem by a writer. It is a time threshold based spinning where the
allowable spinning time can vary from 10us to 25us depending on the
condition of the rwsem.
When the time threshold is exceeded, the nonspinnable bits will be set
in the owner field to indicate that no more optimistic spinning will
be allowed on this rwsem until it becomes writer owned again. Not even
readers is allowed to acquire the reader-locked rwsem by optimistic
spinning for fairness.
We also want a writer to acquire the lock after the readers hold the
lock for a relatively long time. In order to give preference to writers
under such a circumstance, the single RWSEM_NONSPINNABLE bit is now split
into two - one for reader and one for writer. When optimistic spinning
is disabled, both bits will be set. When the reader count drop down
to 0, the writer nonspinnable bit will be cleared to allow writers to
spin on the lock, but not the readers. When a writer acquires the lock,
it will write its own task structure pointer into sem->owner and clear
the reader nonspinnable bit in the process.
The time taken for each iteration of the reader-owned rwsem spinning
loop varies. Below are sample minimum elapsed times for 16 iterations
of the loop.
System Time for 16 Iterations
------ ----------------------
1-socket Skylake ~800ns
4-socket Broadwell ~300ns
2-socket ThunderX2 (arm64) ~250ns
When the lock cacheline is contended, we can see up to almost 10X
increase in elapsed time. So 25us will be at most 500, 1300 and 1600
iterations for each of the above systems.
With a locking microbenchmark running on 5.1 based kernel, the total
locking rates (in kops/s) on a 8-socket IvyBridge-EX system with
equal numbers of readers and writers before and after this patch were
as follows:
# of Threads Pre-patch Post-patch
------------ --------- ----------
2 1,759 6,684
4 1,684 6,738
8 1,074 7,222
16 900 7,163
32 458 7,316
64 208 520
128 168 425
240 143 474
This patch gives a big boost in performance for mixed reader/writer
workloads.
With 32 locking threads, the rwsem lock event data were:
rwsem_opt_fail=79850
rwsem_opt_nospin=5069
rwsem_opt_rlock=597484
rwsem_opt_wlock=957339
rwsem_sleep_reader=57782
rwsem_sleep_writer=55663
With 64 locking threads, the data looked like:
rwsem_opt_fail=346723
rwsem_opt_nospin=6293
rwsem_opt_rlock=1127119
rwsem_opt_wlock=1400628
rwsem_sleep_reader=308201
rwsem_sleep_writer=72281
So a lot more threads acquired the lock in the slowpath and more threads
went to sleep.
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: huang ying <huang.ying.caritas@gmail.com>
Link: https://lkml.kernel.org/r/20190520205918.22251-15-longman@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
||
|
Waiman Long
|
94a9717b3c |
locking/rwsem: Make rwsem->owner an atomic_long_t
The rwsem->owner contains not just the task structure pointer, it also holds some flags for storing the current state of the rwsem. Some of the flags may have to be atomically updated. To reflect the new reality, the owner is now changed to an atomic_long_t type. New helper functions are added to properly separate out the task structure pointer and the embedded flags. Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: huang ying <huang.ying.caritas@gmail.com> Link: https://lkml.kernel.org/r/20190520205918.22251-14-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Waiman Long
|
cf69482d62 |
locking/rwsem: Enable readers spinning on writer
This patch enables readers to optimistically spin on a
rwsem when it is owned by a writer instead of going to sleep
directly. The rwsem_can_spin_on_owner() function is extracted
out of rwsem_optimistic_spin() and is called directly by
rwsem_down_read_slowpath() and rwsem_down_write_slowpath().
With a locking microbenchmark running on 5.1 based kernel, the total
locking rates (in kops/s) on a 8-socket IvyBrige-EX system with equal
numbers of readers and writers before and after the patch were as
follows:
# of Threads Pre-patch Post-patch
------------ --------- ----------
4 1,674 1,684
8 1,062 1,074
16 924 900
32 300 458
64 195 208
128 164 168
240 149 143
The performance change wasn't significant in this case, but this change
is required by a follow-on patch.
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: huang ying <huang.ying.caritas@gmail.com>
Link: https://lkml.kernel.org/r/20190520205918.22251-13-longman@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
||
|
Waiman Long
|
02f1082b00 |
locking/rwsem: Clarify usage of owner's nonspinaable bit
Bit 1 of sem->owner (RWSEM_ANONYMOUSLY_OWNED) is used to designate an anonymous owner - readers or an anonymous writer. The setting of this anonymous bit is used as an indicator that optimistic spinning cannot be done on this rwsem. With the upcoming reader optimistic spinning patches, a reader-owned rwsem can be spinned on for a limit period of time. We still need this bit to indicate a rwsem is nonspinnable, but not setting this bit loses its meaning that the owner is known. So rename the bit to RWSEM_NONSPINNABLE to clarify its meaning. This patch also fixes a DEBUG_RWSEMS_WARN_ON() bug in __up_write(). Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: huang ying <huang.ying.caritas@gmail.com> Link: https://lkml.kernel.org/r/20190520205918.22251-12-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Waiman Long
|
d3681e269f |
locking/rwsem: Wake up almost all readers in wait queue
When the front of the wait queue is a reader, other readers
immediately following the first reader will also be woken up at the
same time. However, if there is a writer in between. Those readers
behind the writer will not be woken up.
Because of optimistic spinning, the lock acquisition order is not FIFO
anyway. The lock handoff mechanism will ensure that lock starvation
will not happen.
Assuming that the lock hold times of the other readers still in the
queue will be about the same as the readers that are being woken up,
there is really not much additional cost other than the additional
latency due to the wakeup of additional tasks by the waker. Therefore
all the readers up to a maximum of 256 in the queue are woken up when
the first waiter is a reader to improve reader throughput. This is
somewhat similar in concept to a phase-fair R/W lock.
With a locking microbenchmark running on 5.1 based kernel, the total
locking rates (in kops/s) on a 8-socket IvyBridge-EX system with
equal numbers of readers and writers before and after this patch were
as follows:
# of Threads Pre-Patch Post-patch
------------ --------- ----------
4 1,641 1,674
8 731 1,062
16 564 924
32 78 300
64 38 195
240 50 149
There is no performance gain at low contention level. At high contention
level, however, this patch gives a pretty decent performance boost.
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: huang ying <huang.ying.caritas@gmail.com>
Link: https://lkml.kernel.org/r/20190520205918.22251-11-longman@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
||
|
Waiman Long
|
990fa7384a |
locking/rwsem: More optimal RT task handling of null owner
An RT task can do optimistic spinning only if the lock holder is
actually running. If the state of the lock holder isn't known, there
is a possibility that high priority of the RT task may block forward
progress of the lock holder if it happens to reside on the same CPU.
This will lead to deadlock. So we have to make sure that an RT task
will not spin on a reader-owned rwsem.
When the owner is temporarily set to NULL, there are two cases
where we may want to continue spinning:
1) The lock owner is in the process of releasing the lock, sem->owner
is cleared but the lock has not been released yet.
2) The lock was free and owner cleared, but another task just comes
in and acquire the lock before we try to get it. The new owner may
be a spinnable writer.
So an RT task is now made to retry one more time to see if it can
acquire the lock or continue spinning on the new owning writer.
When testing on a 8-socket IvyBridge-EX system, the one additional retry
seems to improve locking performance of RT write locking threads under
heavy contentions. The table below shows the locking rates (in kops/s)
with various write locking threads before and after the patch.
Locking threads Pre-patch Post-patch
--------------- --------- -----------
4 2,753 2,608
8 2,529 2,520
16 1,727 1,918
32 1,263 1,956
64 889 1,343
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: huang ying <huang.ying.caritas@gmail.com>
Link: https://lkml.kernel.org/r/20190520205918.22251-10-longman@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
||
|
Waiman Long
|
00f3c5a3df |
locking/rwsem: Always release wait_lock before waking up tasks
With the use of wake_q, we can do task wakeups without holding the wait_lock. There is one exception in the rwsem code, though. It is when the writer in the slowpath detects that there are waiters ahead but the rwsem is not held by a writer. This can lead to a long wait_lock hold time especially when a large number of readers are to be woken up. Remediate this situation by releasing the wait_lock before waking up tasks and re-acquiring it afterward. The rwsem_try_write_lock() function is also modified to read the rwsem count directly to avoid stale count value. Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: huang ying <huang.ying.caritas@gmail.com> Link: https://lkml.kernel.org/r/20190520205918.22251-9-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Waiman Long
|
4f23dbc1e6 |
locking/rwsem: Implement lock handoff to prevent lock starvation
Because of writer lock stealing, it is possible that a constant stream of incoming writers will cause a waiting writer or reader to wait indefinitely leading to lock starvation. This patch implements a lock handoff mechanism to disable lock stealing and force lock handoff to the first waiter or waiters (for readers) in the queue after at least a 4ms waiting period unless it is a RT writer task which doesn't need to wait. The waiting period is used to avoid discouraging lock stealing too much to affect performance. The setting and clearing of the handoff bit is serialized by the wait_lock. So racing is not possible. A rwsem microbenchmark was run for 5 seconds on a 2-socket 40-core 80-thread Skylake system with a v5.1 based kernel and 240 write_lock threads with 5us sleep critical section. Before the patch, the min/mean/max numbers of locking operations for the locking threads were 1/7,792/173,696. After the patch, the figures became 5,842/6,542/7,458. It can be seen that the rwsem became much more fair, though there was a drop of about 16% in the mean locking operations done which was a tradeoff of having better fairness. Making the waiter set the handoff bit right after the first wakeup can impact performance especially with a mixed reader/writer workload. With the same microbenchmark with short critical section and equal number of reader and writer threads (40/40), the reader/writer locking operation counts with the current patch were: 40 readers, Iterations Min/Mean/Max = 1,793/1,794/1,796 40 writers, Iterations Min/Mean/Max = 1,793/34,956/86,081 By making waiter set handoff bit immediately after wakeup: 40 readers, Iterations Min/Mean/Max = 43/44/46 40 writers, Iterations Min/Mean/Max = 43/1,263/3,191 Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: huang ying <huang.ying.caritas@gmail.com> Link: https://lkml.kernel.org/r/20190520205918.22251-8-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |
||
|
Waiman Long
|
3f6d517a3e |
locking/rwsem: Make rwsem_spin_on_owner() return owner state
This patch modifies rwsem_spin_on_owner() to return four possible values to better reflect the state of lock holder which enables us to make a better decision of what to do next. Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: huang ying <huang.ying.caritas@gmail.com> Link: https://lkml.kernel.org/r/20190520205918.22251-7-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> |