Linus Torvalds
d1984c92f0
signal: avoid double atomic counter increments for user accounting ...
[ Upstream commit fda31c50292a5062332fa0343c084bd9f46604d9 ]
When queueing a signal, we increment both the users count of pending
signals (for RLIMIT_SIGPENDING tracking) and we increment the refcount
of the user struct itself (because we keep a reference to the user in
the signal structure in order to correctly account for it when freeing).
That turns out to be fairly expensive, because both of them are atomic
updates, and particularly under extreme signal handling pressure on big
machines, you can get a lot of cache contention on the user struct.
That can then cause horrid cacheline ping-pong when you do these
multiple accesses.
So change the reference counting to only pin the user for the _first_
pending signal, and to unpin it when the last pending signal is
dequeued. That means that when a user sees a lot of concurrent signal
queuing - which is the only situation when this matters - the only
atomic access needed is generally the 'sigpending' count update.
This was noticed because of a particularly odd timing artifact on a
dual-socket 96C/192T Cascade Lake platform: when you get into bad
contention, on that machine for some reason seems to be much worse when
the contention happens in the upper 32-byte half of the cacheline.
As a result, the kernel test robot will-it-scale 'signal1' benchmark had
an odd performance regression simply due to random alignment of the
'struct user_struct' (and pointed to a completely unrelated and
apparently nonsensical commit for the regression).
Avoiding the double increments (and decrements on the dequeueing side,
of course) makes for much less contention and hugely improved
performance on that will-it-scale microbenchmark.
Quoting Feng Tang:
"It makes a big difference, that the performance score is tripled! bump
from original 17000 to 54000. Also the gap between 5.0-rc6 and
5.0-rc6+Jiri's patch is reduced to around 2%"
[ The "2% gap" is the odd cacheline placement difference on that
platform: under the extreme contention case, the effect of which half
of the cacheline was hot was 5%, so with the reduced contention the
odd timing artifact is reduced too ]
It does help in the non-contended case too, but is not nearly as
noticeable.
Reported-and-tested-by: Feng Tang <feng.tang@intel.com>
Cc: Eric W. Biederman <ebiederm@xmission.com>
Cc: Huang, Ying <ying.huang@intel.com>
Cc: Philip Li <philip.li@intel.com>
Cc: Andi Kleen <andi.kleen@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2020-03-21 08:11:53 +01:00
bpf, offload: Replace bitwise AND by logical AND in bpf_prog_offload_info_fill
2020-02-28 17:22:27 +01:00
cgroup: Iterate tasks that did not finish do_exit()
2020-03-18 07:17:48 +01:00
kgdb: don't use a notifier to enter kgdb at panic; call directly
2019-09-25 17:51:40 -07:00
dma-direct: don't check swiotlb=force in dma_direct_map_resource
2020-01-26 10:01:08 +01:00
perf/core: Fix mlock accounting in perf_mmap()
2020-02-11 04:35:54 -08:00
Revert "um: Enable CONFIG_CONSTRUCTORS"
2020-02-01 09:34:53 +00:00
genirq/proc: Reject invalid affinity masks (again)
2020-02-28 17:22:26 +01:00
livepatch: Nullify obj->mod in klp_module_coming()'s error path
2019-08-19 13:03:37 +02:00
locking/lockdep: Fix lockdep_stats indentation problem
2020-03-05 16:43:51 +01:00
ACPI: PM: s2idle: Avoid possible race related to the EC GPE
2020-02-19 19:52:56 +01:00
printk: fix exclusive_console replaying
2020-02-24 08:36:24 +01:00
rcu: Allow only one expedited GP to run concurrently with wakeups
2020-03-05 16:43:50 +01:00
sched/fair: Optimize select_idle_cpu
2020-03-05 16:43:48 +01:00
lib/vdso: Update coarse timekeeper unconditionally
2020-03-05 16:43:49 +01:00
tracing: Fix number printing bug in print_synth_event()
2020-03-21 08:11:49 +01:00
Provide in-kernel headers to make extending kernel easier
2019-04-29 16:48:03 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
audit: always check the netlink payload length in audit_receive_msg()
2020-03-05 16:43:42 +01:00
audit/stable-5.3 PR 20190702
2019-07-08 18:55:42 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
fsnotify: switch send_to_group() and ->handle_event to const struct qstr *
2019-04-26 13:51:03 -04:00
audit_get_nd(): don't unlock parent too early
2019-11-10 11:56:55 -05:00
audit: fix error handling in audit_data_to_entry()
2020-03-05 16:43:42 +01:00
audit: enforce op for string fields
2019-05-28 17:46:43 -04:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500
2019-06-19 17:09:55 +02:00
kernel/configs: Replace GPL boilerplate code with SPDX identifier
2019-07-30 18:34:15 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
cpu/hotplug, stop_machine: Fix stop_machine vs hotplug order
2020-02-24 08:36:23 +01:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 282
2019-06-05 17:36:37 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 230
2019-06-19 17:09:06 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
keys: Fix request_key() cache
2020-01-17 19:48:42 +01:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 25
2019-05-21 11:52:39 +02:00
kernel/elfcore.c: include proper prototypes
2019-09-25 17:51:39 -07:00
exit: panic before exit_mm() on global init exit
2020-01-09 10:20:01 +01:00
extable: Add function to search only kernel exception table
2019-08-21 22:23:48 +10:00
fail_function: no need to check return value of debugfs_create functions
2019-06-03 15:49:06 +02:00
clone3: ensure copy_thread_tls is implemented
2020-01-14 20:08:35 +01:00
Revert "libata, freezer: avoid block device removal while system is frozen"
2019-10-06 09:11:37 -06:00
futex: Prevent exit livelock
2019-11-29 10:10:14 +01:00
kheaders: substituting --sort in archive creation
2019-10-17 09:08:19 +09:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
mm/nvdimm: add is_ioremap_addr and use that to check ioremap address
2019-07-12 11:05:40 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
jump_label: Don't warn on __exit jump entries
2019-08-29 15:10:10 +01:00
kallsyms: Don't let kallsyms_lookup_size_offset() fail on retrieving the first symbol
2019-08-27 16:19:56 +01:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
treewide: Add SPDX license identifier - Makefile/Kconfig
2019-05-21 10:50:46 +02:00
sched/rt, Kconfig: Unbreak def/oldconfig with CONFIG_PREEMPT=y
2019-07-22 18:05:11 +02:00
kexec_load: Disable at runtime if the kernel is locked down
2019-08-19 21:54:15 -07:00
kexec: bail out upon SIGKILL when allocating memory.
2019-09-25 17:51:40 -07:00
kexec_elf: support 32 bit ELF files
2019-09-06 23:58:44 +02:00
Merge branch 'next-lockdown' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security
2019-09-28 08:14:15 -07:00
kheaders: Move from proc to sysfs
2019-05-24 20:16:01 +02:00
kprobes: Fix optimize_kprobe()/unoptimize_kprobe() cancellation logic
2020-03-12 13:00:09 +01:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 170
2019-05-30 11:26:39 -07:00
kthread: make __kthread_queue_delayed_work static
2019-10-16 09:20:58 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
Merge branch 'next-integrity' of git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity
2019-09-27 19:37:27 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 36
2019-05-24 17:27:11 +02:00
module: avoid setting info->name early in case we can fall back to info->mod->name
2020-02-24 08:36:54 +01:00
MODSIGN: Export module signature definitions
2019-08-05 18:39:56 -04:00
MODSIGN: Export module signature definitions
2019-08-05 18:39:56 -04:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
padata: validate cpumask without removed CPU during offline
2020-02-24 08:36:34 +01:00
panic: ensure preemption is disabled during panic()
2019-10-07 15:47:19 -07:00
lockdown: Lock down module params that specify hardware parameters (eg. ioport)
2019-08-19 21:54:16 -07:00
kernel/pid.c: convert struct pid count to refcount_t
2019-07-16 19:23:24 -07:00
proc/sysctl: add shared variables for range check
2019-07-18 17:08:07 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
ptrace: reintroduce usage of subjective credentials in ptrace_has_cap()
2020-01-23 08:22:36 +01:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
mm/memory_hotplug.c: use PFN_UP / PFN_DOWN in walk_system_ram_range()
2019-09-24 15:54:09 -07:00
signal: Remove task parameter from force_sig
2019-05-27 09:36:28 -05:00
seccomp: Check that seccomp_notif is zeroed out by the user
2020-01-09 10:19:57 +01:00
signal: avoid double atomic counter increments for user accounting
2020-03-21 08:11:53 +01:00
smp: Warn on function calls from softirq context
2019-07-20 11:27:16 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
Merge branch 'irq-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-07-08 11:01:13 -07:00
stacktrace: Don't skip first entry on noncurrent tasks
2019-11-04 21:19:25 +01:00
stop_machine: Avoid potential race behaviour
2019-10-17 12:47:12 +02:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-09-17 12:35:15 -07:00
arch: handle arches who do not yet define clone3
2019-06-21 01:54:53 +02:00
kernel: sysctl: make drop_caches write-only
2020-01-04 19:18:32 +01:00
taskstats: fix data-race
2020-01-09 10:19:54 +01:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 25
2019-05-21 11:52:39 +02:00
torture: Remove exporting of internal functions
2019-08-01 14:30:22 -07:00
The main changes in this release include:
2019-07-18 11:51:00 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
2019-05-30 11:26:37 -07:00
proc/sysctl: add shared variables for range check
2019-07-18 17:08:07 -07:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
smp: Remove smp_call_function() and on_each_cpu() return values
2019-06-23 14:26:26 +02:00
treewide: Add SPDX license identifier for missed files
2019-05-21 10:50:45 +02:00
Keyrings namespacing
2019-07-08 19:36:47 -07:00
Keyrings namespacing
2019-07-08 19:36:47 -07:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441
2019-06-05 17:37:17 +02:00
watchdog/softlockup: Enforce that timestamp is valid on boot
2020-02-24 08:36:52 +01:00
kernel/watchdog_hld.c: hard lockup message should end with a newline
2019-04-19 09:46:05 -07:00
workqueue: don't use wq_select_unbound_cpu() for bound works
2020-03-18 07:17:50 +01:00
d1984c92f0