Tom Herbert
0a9627f264
rps: Receive Packet Steering ...
This patch implements software receive side packet steering (RPS). RPS
distributes the load of received packet processing across multiple CPUs.
Problem statement: Protocol processing done in the NAPI context for received
packets is serialized per device queue and becomes a bottleneck under high
packet load. This substantially limits pps that can be achieved on a single
queue NIC and provides no scaling with multiple cores.
This solution queues packets early on in the receive path on the backlog queues
of other CPUs. This allows protocol processing (e.g. IP and TCP) to be
performed on packets in parallel. For each device (or each receive queue in
a multi-queue device) a mask of CPUs is set to indicate the CPUs that can
process packets. A CPU is selected on a per packet basis by hashing contents
of the packet header (e.g. the TCP or UDP 4-tuple) and using the result to index
into the CPU mask. The IPI mechanism is used to raise networking receive
softirqs between CPUs. This effectively emulates in software what a multi-queue
NIC can provide, but is generic requiring no device support.
Many devices now provide a hash over the 4-tuple on a per packet basis
(e.g. the Toeplitz hash). This patch allow drivers to set the HW reported hash
in an skb field, and that value in turn is used to index into the RPS maps.
Using the HW generated hash can avoid cache misses on the packet when
steering it to a remote CPU.
The CPU mask is set on a per device and per queue basis in the sysfs variable
/sys/class/net/<device>/queues/rx-<n>/rps_cpus. This is a set of canonical
bit maps for receive queues in the device (numbered by <n>). If a device
does not support multi-queue, a single variable is used for the device (rx-0).
Generally, we have found this technique increases pps capabilities of a single
queue device with good CPU utilization. Optimal settings for the CPU mask
seem to depend on architectures and cache hierarcy. Below are some results
running 500 instances of netperf TCP_RR test with 1 byte req. and resp.
Results show cumulative transaction rate and system CPU utilization.
e1000e on 8 core Intel
Without RPS: 108K tps at 33% CPU
With RPS: 311K tps at 64% CPU
forcedeth on 16 core AMD
Without RPS: 156K tps at 15% CPU
With RPS: 404K tps at 49% CPU
bnx2x on 16 core AMD
Without RPS 567K tps at 61% CPU (4 HW RX queues)
Without RPS 738K tps at 96% CPU (8 HW RX queues)
With RPS: 854K tps at 76% CPU (4 HW RX queues)
Caveats:
- The benefits of this patch are dependent on architecture and cache hierarchy.
Tuning the masks to get best performance is probably necessary.
- This patch adds overhead in the path for processing a single packet. In
a lightly loaded server this overhead may eliminate the advantages of
increased parallelism, and possibly cause some relative performance degradation.
We have found that masks that are cache aware (share same caches with
the interrupting CPU) mitigate much of this.
- The RPS masks can be changed dynamically, however whenever the mask is changed
this introduces the possibility of generating out of order packets. It's
probably best not change the masks too frequently.
Signed-off-by: Tom Herbert <therbert@google.com>
include/linux/netdevice.h | 32 ++++-
include/linux/skbuff.h | 3 +
net/core/dev.c | 335 +++++++++++++++++++++++++++++++++++++--------
net/core/net-sysfs.c | 225 ++++++++++++++++++++++++++++++-
net/core/skbuff.c | 2 +
5 files changed, 538 insertions(+), 59 deletions(-)
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-03-16 21:23:18 -07:00
ARM: 5951/1: ARM: fix documentation of the PrimeCell bus
2010-02-20 14:10:48 +00:00
can: ti hecc module : add platform specific initialization callback.
2010-02-26 05:22:03 -08:00
decompress: fix new decompressor for PIC
2010-03-12 15:52:44 -08:00
mfd: Introduce remove_script function for twl4030
2010-03-07 22:17:32 +01:00
Input: sh_keysc - update the driver with mode 6
2010-02-10 23:04:18 -08:00
CAPI: Rework locking of controller data structures
2010-02-16 16:01:22 -08:00
mfd: Fix ucb1x00 build failure for collie_defconfig
2010-03-07 22:17:39 +01:00
sdio: put active devices into 1-bit mode during suspend
2010-03-06 11:26:37 -08:00
mtd: sh_flctl SHBUSSEL and SEL_16BIT support
2010-02-02 13:02:16 +09:00
netfilter: xtables: replace XT_MATCH_ITERATE macro
2010-02-24 18:34:48 +01:00
netfilter: xtables: replace XT_ENTRY_ITERATE macro
2010-02-24 18:32:59 +01:00
netfilter: xtables: replace XT_MATCH_ITERATE macro
2010-02-24 18:34:48 +01:00
netfilter: xtables: replace XT_MATCH_ITERATE macro
2010-02-24 18:34:48 +01:00
regulator: enable max8649 regulator driver
2010-03-03 14:49:24 +00:00
Input: ads7846 - add wakeup support
2010-03-11 00:02:43 -08:00
nfs41 do not allocate unused back channel pages
2010-02-10 08:31:02 -05:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound-2.6
2010-03-08 07:34:26 -08:00
sysctl extern cleanup: acct
2010-03-12 15:53:10 -08:00
ata: Detect Delkin Devices compact flash
2010-03-01 15:11:32 -05:00
Lose the first argument of audit_inode_child()
2010-02-08 14:38:36 -05:00
coredump: pass mm->flags as a coredump parameter for consistency
2010-03-06 11:26:46 -08:00
bitops: rename for_each_bit() to for_each_set_bit()
2010-03-06 11:26:23 -08:00
block: Consolidate phys_segment and hw_segment limits
2010-02-26 13:58:08 +01:00
percpu: add __percpu sparse annotations to core kernel subsystems
2010-02-17 11:17:38 +09:00
x86: Make 64 bit use early_res instead of bootmem before slab
2010-02-12 09:41:59 -08:00
tg3: Push phylib definitions to phylib
2010-02-17 17:27:40 -08:00
cciss: Consolidate duplicate bits in cciss_cmd.h & cciss_ioctl.h
2010-02-22 13:44:45 +01:00
cciss: Consolidate duplicate bits in cciss_cmd.h & cciss_ioctl.h
2010-02-22 13:44:45 +01:00
cgroups: remove events before destroying subsystem state objects
2010-03-12 15:52:37 -08:00
clocksource: add suspend callback
2010-02-05 14:54:10 +01:00
Add generic sys_old_select()
2010-03-12 15:52:32 -08:00
percpu: add __percpu for sparse
2010-02-05 07:35:05 -08:00
connector: Delete buggy notification code.
2010-02-02 15:58:48 -08:00
coredump: plug a memory leak situation on dump_seek()
2010-03-12 15:52:32 -08:00
cpumask: let num_*_cpus() function always return unsigned values
2010-03-06 11:26:29 -08:00
rcu: Use wrapper function instead of exporting tasklist_lock
2010-03-04 11:46:14 +01:00
TI Davinci EMAC : Abstract Buffer address translation logic.
2010-02-04 13:29:53 -08:00
dm table: remove unused dm_get_device range parameters
2010-03-06 02:32:27 +00:00
Driver core: create lock/unlock functions for struct device
2010-03-07 17:04:52 -08:00
dm: eliminate some holes data structures
2010-03-06 02:32:33 +00:00
dm ioctl: introduce flag indicating uevent was generated
2010-03-06 02:32:31 +00:00
Fix spelling of 'platform' in comments and doc
2010-02-05 12:22:34 +01:00
dma-mapping: dma-mapping.h: add dma_set_coherent_mask
2010-03-12 15:52:42 -08:00
Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/djbw/async_tx
2010-03-04 08:20:14 -08:00
early_res: Add free_early_partial()
2010-02-26 08:25:35 +01:00
elf coredump: add extended numbering support
2010-03-06 11:26:46 -08:00
elf coredump: add extended numbering support
2010-03-06 11:26:46 -08:00
ethtool: Add direct access to ops->get_sset_count
2010-03-05 14:00:17 -08:00
commit_metadata export operation replacing nfsd_sync_dir
2010-02-20 13:13:44 -08:00
Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs-2.6
2010-03-05 13:20:53 -08:00
ext3: Use bitops to read/modify EXT3_I(inode)->i_state
2010-03-05 00:20:20 +01:00
failslab: add ability to filter slab caches
2010-02-26 19:19:39 +02:00
vga_switcheroo: initial implementation (v15)
2010-03-01 16:20:37 +10:00
vfs: Abstract rcu_dereference_check for files-fdtable use
2010-02-25 10:34:49 +01:00
firewire: cdev: increment ABI version number
2010-02-24 20:36:55 +01:00
memory-hotplug: create /sys/firmware/memmap entry for new memory
2010-03-06 11:26:25 -08:00
include/linux/fs.h: convert FMODE_* constants to hex
2010-03-06 11:26:25 -08:00
Lose the first argument of audit_inode_child()
2010-02-08 14:38:36 -05:00
Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-02-28 10:20:25 -08:00
Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-02-28 10:20:25 -08:00
percpu: add __percpu sparse annotations to core kernel subsystems
2010-02-17 11:17:38 +09:00
mm: add comment about deprecation of __GFP_NOFAIL
2010-03-06 11:26:27 -08:00
GFS2: Remove old, unused linked list code from quota
2010-03-01 14:08:10 +00:00
Input: gpio-keys - add support for disabling gpios through sysfs
2010-02-04 00:50:44 -08:00
Merge branches 'upstream', 'raw_report_modifications' and 'apple_magic_mouse' into for-linus
2010-02-25 17:39:16 +01:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/jejb/xfs-vipt
2010-02-26 17:05:10 -08:00
tree-wide: Assorted spelling fixes
2010-02-09 11:13:56 +01:00
mfd: Add HTCPLD driver
2010-03-07 22:17:09 +01:00
percpu: Add __percpu sparse annotations to hw_breakpoint
2010-02-27 16:23:39 +01:00
i2c-algo-bit: Add pre- and post-xfer hooks
2010-03-13 20:56:56 +01:00
ARM: PNX4008: move i2c_adapter structure inside the drivers private data
2010-02-12 17:32:41 +00:00
i2c: Add SMBus alert support
2010-03-02 12:23:42 +01:00
Add the platform data include for the Xilinx XPS IIC Bus Interface
2010-03-14 11:14:58 -07:00
i2c: Document the message size limit
2010-03-02 12:23:49 +01:00
block: Consolidate phys_segment and hw_segment limits
2010-02-26 13:58:08 +01:00
ipv6: drop unused "dev" arg of icmpv6_send()
2010-02-18 14:30:17 -08:00
if_link: Add SR-IOV configuration methods
2010-02-12 16:56:07 -08:00
net/macvtap: add vhost support
2010-02-18 14:08:38 -08:00
packet: Add GSO/csum offload support.
2010-02-04 20:24:10 -08:00
tun: socket filter support
2010-02-17 16:35:17 -08:00
igmp: fix ip_mc_sf_allow race [v5]
2010-02-02 07:32:29 -08:00
ima: rename ima_path_check to ima_file_check
2010-02-07 03:06:22 -05:00
net ipv4: Decouple ipv4 interface parameters from binary sysctl numbers
2010-02-16 15:55:17 -08:00
nsproxy: remove INIT_NSPROXY()
2010-03-12 15:52:40 -08:00
Input: scancode in get/set_keycodes should be unsigned
2010-03-08 23:19:15 -08:00
cgroups: blkio subsystem as module
2010-03-12 15:52:36 -08:00
resource: add window support
2010-03-14 20:08:36 -04:00
nsproxy: remove INIT_NSPROXY()
2010-03-12 15:52:40 -08:00
ipmi: remove ipmi_smi.h self-include
2010-03-12 15:52:40 -08:00
Merge branch 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-03-03 08:15:37 -08:00
jbd[2]: remove references to BUFFER_DEBUG
2010-03-05 00:20:20 +01:00
Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs-2.6
2010-03-05 13:20:53 -08:00
Input: Mac button emulation - implement as an input filter
2010-01-30 01:47:49 -08:00
Merge branch 'for-linus' of git://gitorious.org/linux-omap-dss2/linux
2010-03-02 10:41:26 -08:00
Merge branch 'x86-bootmem-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-03-03 08:15:05 -08:00
CAPI: Rework controller state notifier
2010-02-16 16:01:21 -08:00
percpu: add __percpu sparse annotations to core kernel subsystems
2010-02-17 11:17:38 +09:00
kfifo: Don't use integer as NULL pointer
2010-02-16 15:11:08 -08:00
sysctl extern cleanup: module
2010-03-12 15:53:10 -08:00
Driver core: Constify struct sysfs_ops in struct kobj_type
2010-03-07 17:04:49 -08:00
kprobes: Jump optimization sysctl interface
2010-02-25 17:49:25 +01:00
KVM: x86: Add KVM_CAP_X86_ROBUST_SINGLESTEP
2010-03-01 12:36:14 -03:00
KVM: Convert kvm->requests_lock to raw_spinlock_t
2010-03-01 12:36:13 -03:00
libata: Pass host flags into the pci helper
2010-03-01 14:58:46 -05:00
lib: fix first line of kernel-doc for a few functions
2010-03-06 11:26:35 -08:00
lmb: Add lmb_free()
2010-02-03 17:39:50 +11:00
sysctl extern cleanup: lockdep
2010-03-12 15:53:10 -08:00
tree-wide: Assorted spelling fixes
2010-02-09 11:13:56 +01:00
switch inotify_user to anon_inode
2010-02-19 03:35:12 -05:00
memcg: fix oom kill behavior
2010-03-12 15:52:38 -08:00
mm: introduce dump_page() and print symbolic flag names
2010-03-12 15:52:28 -08:00
nommu: fix build breakage
2010-03-12 15:52:28 -08:00
Merge branch 'for-next' into for-linus
2010-03-08 16:55:37 +01:00
take check for new events in namespace (guts of mounts_poll()) to namespace.c
2010-03-03 14:07:59 -05:00
sysctl extern cleanup: module
2010-03-12 15:53:10 -08:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs-2.6
2010-03-04 08:15:33 -08:00
fat: Fix stat->f_namelen
2010-02-10 23:49:08 +09:00
net: CONFIG_COMPAT redux
2010-02-03 20:32:28 -08:00
rps: Receive Packet Steering
2010-03-16 21:23:18 -07:00
netfilter: restore POST_ROUTING hook in NF_HOOK_COND
2010-02-19 08:03:28 +01:00
NFS: Remove requirement for inode->i_mutex from nfs_invalidate_mapping
2010-03-05 15:44:56 -05:00
Merge branch 'writeback-for-2.6.34' into nfs-for-2.6.34
2010-03-05 15:46:18 -05:00
nilfs2: issue discard request after cleaning segments
2010-02-13 12:26:02 +09:00
nl80211: add power save commands
2010-02-19 15:52:40 -05:00
nodemask: fix the declaration of NODEMASK_ALLOC()
2010-03-12 15:52:38 -08:00
of/sparc: Remove sparc-local declaration of allnodes and devtree_lock
2010-02-14 13:02:44 -07:00
of/flattree: Don't assume HAVE_LMB
2010-02-14 07:13:47 -07:00
OMAP: DSS2: OMAPFB: implement OMAPFB_GET_DISPLAY_INFO
2010-02-15 15:14:34 +02:00
memcg: move charges of anonymous swap
2010-03-12 15:52:36 -08:00
PCI / ACPI / PM: Platform support for PCI PME wake-up
2010-02-22 16:21:02 -08:00
dma-mapping.h: add the dma_unmap state API
2010-03-12 15:52:42 -08:00
dma-mapping: pci: move pci_set_dma_mask and pci_set_consistent_dma_mask to pci-dma-compat.h
2010-03-12 15:52:42 -08:00
PCI: Make current and maximum bus speeds part of the PCI core
2010-02-22 16:15:17 -08:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty-2.6
2010-03-03 08:47:22 -08:00
x86/PCI: Moorestown PCI support
2010-02-23 23:14:47 -08:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu
2010-03-03 07:34:18 -08:00
perf: Provide generic perf_sample_data initialization
2010-03-10 13:22:23 +01:00
libphy: add phy_find_first function
2010-02-04 10:23:02 -08:00
pktcdvd: use BIO list management functions
2010-02-24 08:30:08 +01:00
Driver core: make struct platform_driver.id_table const
2010-03-07 17:04:49 -08:00
PM: Provide generic subsystem-level callbacks
2010-03-06 21:28:37 +01:00
PM: Provide generic subsystem-level callbacks
2010-03-06 21:28:37 +01:00
sysctl extern cleanup: poll
2010-03-12 15:53:11 -08:00
ptrace: move user_enable_single_step & co prototypes to linux/ptrace.h
2010-03-12 15:52:38 -08:00
quota: stop using QUOTA_OK / NO_QUOTA
2010-03-05 00:20:31 +01:00
dquot: cleanup dquot initialize routine
2010-03-05 00:20:30 +01:00
[SCSI] mpt2sas: Added raid transport support
2010-02-08 18:19:41 -06:00
x86/pci: Add cap_resource()
2010-02-10 17:47:17 -08:00
doc: fix typo in comment explaining rb_tree usage
2010-02-25 11:45:20 +01:00
Merge branch 'master' of /home/davem/src/GIT/linux-2.6/
2010-02-28 19:23:06 -08:00
rcu: Disable lockdep checking in RCU list-traversal primitives
2010-02-25 09:41:02 +01:00
Merge branch 'core-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-03-13 14:43:01 -08:00
rcu: Make rcu_read_lock_sched_held() take boot time into account
2010-02-26 08:20:46 +01:00
rcu: Make rcu_read_lock_sched_held() take boot time into account
2010-02-26 08:20:46 +01:00
sysctl extern cleanup: C_A_D
2010-03-12 15:52:44 -08:00
pass writeback_control to ->write_inode
2010-03-05 13:25:52 -05:00
PM: Asynchronous suspend and resume of devices
2010-02-26 20:39:09 +01:00
rfkill: Add support for KEY_RFKILL
2010-03-02 14:28:49 -05:00
vmscan: detect mapped file pages used only once
2010-03-06 11:26:27 -08:00
rtc/hctosys: only claim the RTC provided the system time if it did
2010-03-12 15:52:28 -08:00
sysctl extern cleanup: rtmutex
2010-03-12 15:53:10 -08:00
Merge branch 'master' of /home/davem/src/GIT/linux-2.6/
2010-02-28 19:23:06 -08:00
locking: Make sparse work with inline spinlocks and rwlocks
2010-03-13 01:21:21 +01:00
Merge branch 'core-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-03-13 14:43:01 -08:00
Security: add static to security_ops and default_security_ops variable
2010-02-24 08:11:02 +11:00
seq_file: add RCU versions of new hlist/list iterators (v3)
2010-02-22 15:45:54 -08:00
pps: serial clients support
2010-03-12 15:52:43 -08:00
serial: sh-sci: Add DMA support.
2010-03-02 11:39:15 +09:00
sh: extend INTC with force_disable
2010-02-16 13:38:56 +09:00
sysctl extern cleanup: signal
2010-03-12 15:52:44 -08:00
rps: Receive Packet Steering
2010-03-16 21:23:18 -07:00
failslab: add ability to filter slab caches
2010-02-26 19:19:39 +02:00
smp: fix documentation in include/linux/smp.h
2010-03-06 11:26:32 -08:00
tcp: Add SNMP counters for backlog and min_ttl drops
2010-03-08 10:45:27 -08:00
locking: Make sparse work with inline spinlocks and rwlocks
2010-03-13 01:21:21 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu
2010-03-03 07:34:18 -08:00
memcg: move charges of anonymous swap
2010-03-12 15:52:36 -08:00
Add generic sys_olduname()
2010-03-12 15:52:32 -08:00
net neigh: Decouple per interface neighbour table controls from binary sysctls
2010-02-16 15:55:18 -08:00
sysdev: Add sysdev_create/remove_files
2010-03-07 17:04:47 -08:00
sysfs: fix for thinko with sysfs_bin_attr_init()
2010-03-07 17:04:52 -08:00
syslog: use defined constants instead of raw numbers
2010-02-04 14:20:41 +11:00
copy_signal() cleanup: kill taskstats_tgid_init() and acct_init_pacct()
2010-03-12 15:52:39 -08:00
net: TCP thin dupack
2010-02-18 15:43:09 -08:00
ntp: Make time_esterror and time_maxerror static
2010-01-29 10:15:19 +01:00
ldisc n_tty: add new method n_tty_inherit_ops()
2010-03-12 15:52:43 -08:00
USB: tty: Add a function to insert a string of characters with the same flag
2010-03-02 14:55:11 -08:00
ldisc: new dcd_change() method for line disciplines
2010-03-12 15:52:43 -08:00
Driver core: create lock/unlock functions for struct device
2010-03-07 17:04:52 -08:00
vga_switcheroo: fix build on platforms with no ACPI
2010-03-01 22:21:58 +11:00
V4L/DVB (14003): gspca_cpai1: New gspca subdriver for CPIA CPiA version 1 cams
2010-02-26 15:10:40 -03:00
net/9p: Add sysfs mount_tag file for virtio 9P device
2010-03-13 08:57:29 -06:00
net/9p: Use the tag name in the config space for identifying mount point
2010-03-13 08:57:28 -06:00
virtio: Add memory statistics reporting to the balloon driver (V4)
2010-02-24 14:22:08 +10:30
virtio_blk: add block topology support
2010-02-24 14:22:26 +10:30
virtio: console: Add ability to hot-unplug ports
2010-02-24 14:23:00 +10:30
tty: Add a new VT mode which is like VT_PROCESS but doesn't require a VT_RELDISP ioctl call
2010-03-02 14:43:23 -08:00
xfrm: introduce basic mark infrastructure
2010-02-22 16:19:45 -08:00
0a9627f264