Jason A. Donenfeld
aec35934d0
UPSTREAM: wireguard: queueing: get rid of per-peer ring buffers ...
Having two ring buffers per-peer means that every peer results in two
massive ring allocations. On an 8-core x86_64 machine, this commit
reduces the per-peer allocation from 18,688 bytes to 1,856 bytes, which
is an 90% reduction. Ninety percent! With some single-machine
deployments approaching 500,000 peers, we're talking about a reduction
from 7 gigs of memory down to 700 megs of memory.
In order to get rid of these per-peer allocations, this commit switches
to using a list-based queueing approach. Currently GSO fragments are
chained together using the skb->next pointer (the skb_list_* singly
linked list approach), so we form the per-peer queue around the unused
skb->prev pointer (which sort of makes sense because the links are
pointing backwards). Use of skb_queue_* is not possible here, because
that is based on doubly linked lists and spinlocks. Multiple cores can
write into the queue at any given time, because its writes occur in the
start_xmit path or in the udp_recv path. But reads happen in a single
workqueue item per-peer, amounting to a multi-producer, single-consumer
paradigm.
The MPSC queue is implemented locklessly and never blocks. However, it
is not linearizable (though it is serializable), with a very tight and
unlikely race on writes, which, when hit (some tiny fraction of the
0.15% of partial adds on a fully loaded 16-core x86_64 system), causes
the queue reader to terminate early. However, because every packet sent
queues up the same workqueue item after it is fully added, the worker
resumes again, and stopping early isn't actually a problem, since at
that point the packet wouldn't have yet been added to the encryption
queue. These properties allow us to avoid disabling interrupts or
spinning. The design is based on Dmitry Vyukov's algorithm [1].
Performance-wise, ordinarily list-based queues aren't preferable to
ringbuffers, because of cache misses when following pointers around.
However, we *already* have to follow the adjacent pointers when working
through fragments, so there shouldn't actually be any change there. A
potential downside is that dequeueing is a bit more complicated, but the
ptr_ring structure used prior had a spinlock when dequeueing, so all and
all the difference appears to be a wash.
Actually, from profiling, the biggest performance hit, by far, of this
commit winds up being atomic_add_unless(count, 1, max) and atomic_
dec(count), which account for the majority of CPU time, according to
perf. In that sense, the previous ring buffer was superior in that it
could check if it was full by head==tail, which the list-based approach
cannot do.
But all and all, this enables us to get massive memory savings, allowing
WireGuard to scale for real world deployments, without taking much of a
performance hit.
[1] http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
Fixes: e7096c131e51 ("net: WireGuard secure network tunnel")
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 8b5553ace83cced775eefd0f3f18b5c6214ccf7a)
Bug: 187129171
Signed-off-by: Connor O'Brien <connoro@google.com>
Change-Id: I399dd0f0f76b9eadf5c06b7d99bfcb8fc3b7f13e
2025-09-24 12:16:38 +02:00
This is the 5.4.296 stable release
2025-07-18 10:57:54 +00:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
ata: pata_cs5536: fix build on 32-bit UML
2025-07-17 18:24:57 +02:00
atm: idt77252: Add missing dma_map_error()
2025-07-17 18:25:05 +02:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-06-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-06-30 10:49:17 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-06-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-06-30 10:49:17 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
fpga: altera-cvp: Increase credit timeout
2025-06-04 14:32:32 +02:00
gpio: zynq: Fix wakeup source leaks on device unbind
2025-05-02 07:39:17 +02:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
HSI: ssi_protocol: Fix use after free vulnerability in ssi_protocol Driver Due to Race Condition
2025-05-02 07:39:18 +02:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-05-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-05-18 08:00:20 +00:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-06-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-06-30 10:49:17 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
This is the 5.4.296 stable release
2025-07-18 10:57:54 +00:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-05-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-05-18 08:00:20 +00:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
mcb: fix a double free bug in chameleon_parse_gdd()
2025-05-02 07:39:26 +02:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
UPSTREAM: wireguard: queueing: get rid of per-peer ring buffers
2025-09-24 12:16:38 +02:00
ntb: reduce stack usage in idt_scan_mws
2025-05-02 07:39:28 +02:00
libnvdimm/labels: Fix divide error in nd_label_data_init()
2025-06-04 14:32:30 +02:00
nvmet-tcp: don't restore null sk_state_change
2025-06-04 14:32:35 +02:00
Merge tag 'ASB-2025-05-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-05-18 08:00:20 +00:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-06-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-06-30 10:49:17 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
drivers/rapidio/rio_cm.c: prevent possible heap overwrite
2025-06-27 11:02:56 +01:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
scsi: s390: zfcp: Ensure synchronous unit_add
2025-06-27 11:02:56 +01:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-05-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-05-18 08:00:20 +00:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
This is the 5.4.294 stable release
2025-06-05 07:11:21 +00:00
tee: Prevent size calculation wraparound on 32-bit kernels
2025-06-27 11:02:55 +01:00
Merge tag 'ASB-2025-06-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-06-30 10:49:17 +03:00
thunderbolt: Do not double dequeue a configuration request
2025-06-27 11:02:44 +01:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
Merge tag 'ASB-2025-09-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-12 18:48:05 +03:00
Merge tag 'ASB-2025-08-05_11-5.4' of https://android.googlesource.com/kernel/common into android13-5.4-lahaina
2025-09-01 13:17:46 +03:00
watchdog: da9052_wdt: respect TWDMIN
2025-06-27 11:02:55 +01:00
xen/swiotlb: relax alignment requirements
2025-06-04 14:32:37 +02:00
aec35934d0