Nick Piggin
6416ccb789
fs: scale files_lock ...
fs: scale files_lock
Improve scalability of files_lock by adding per-cpu, per-sb files lists,
protected with an lglock. The lglock provides fast access to the per-cpu lists
to add and remove files. It also provides a snapshot of all the per-cpu lists
(although this is very slow).
One difficulty with this approach is that a file can be removed from the list
by another CPU. We must track which per-cpu list the file is on with a new
variale in the file struct (packed into a hole on 64-bit archs). Scalability
could suffer if files are frequently removed from different cpu's list.
However loads with frequent removal of files imply short interval between
adding and removing the files, and the scheduler attempts to avoid moving
processes too far away. Also, even in the case of cross-CPU removal, the
hardware has much more opportunity to parallelise cacheline transfers with N
cachelines than with 1.
A worst-case test of 1 CPU allocating files subsequently being freed by N CPUs
degenerates to contending on a single lock, which is no worse than before. When
more than one CPU are allocating files, even if they are always freed by
different CPUs, there will be more parallelism than the single-lock case.
Testing results:
On a 2 socket, 8 core opteron, I measure the number of times the lock is taken
to remove the file, the number of times it is removed by the same CPU that
added it, and the number of times it is removed by the same node that added it.
Booting: locks= 25049 cpu-hits= 23174 (92.5%) node-hits= 23945 (95.6%)
kbuild -j16 locks=2281913 cpu-hits=2208126 (96.8%) node-hits=2252674 (98.7%)
dbench 64 locks=4306582 cpu-hits=4287247 (99.6%) node-hits=4299527 (99.8%)
So a file is removed from the same CPU it was added by over 90% of the time.
It remains within the same node 95% of the time.
Tim Chen ran some numbers for a 64 thread Nehalem system performing a compile.
throughput
2.6.34-rc2 24.5
+patch 24.9
us sys idle IO wait (in %)
2.6.34-rc2 51.25 28.25 17.25 3.25
+patch 53.75 18.5 19 8.75
So significantly less CPU time spent in kernel code, higher idle time and
slightly higher throughput.
Single threaded performance difference was within the noise of microbenchmarks.
That is not to say penalty does not exist, the code is larger and more memory
accesses required so it will be slightly slower.
Cc: linux-kernel@vger.kernel.org
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2010-08-18 08:35:48 -04:00
Merge branch 'misc' into devel
2010-07-31 14:20:02 +01:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
CAN: Add Flexcan CAN controller driver
2010-07-22 18:06:25 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
hwmon: (ltc4245) Expose all GPIO pins as analog voltages
2010-08-14 21:08:49 +02:00
Input: cy8ctmg110 - capacitive touchscreen support
2010-07-31 02:37:17 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/lrg/voltage-2.6
2010-08-12 10:01:30 -07:00
mmc: add erase, secure erase, trim and secure trim operations
2010-08-12 08:43:30 -07:00
mtd: Remove obsolete <mtd/compatmac.h> include
2010-08-08 21:19:42 +01:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
POWER: Add JZ4740 battery driver.
2010-08-05 13:26:20 +01:00
Make lib/raid6/test build correctly.
2010-08-12 06:38:24 +10:00
ab8500-mfd: add regulator support to ab8500 mfd device
2010-08-11 11:38:01 +01:00
Merge branch 'next-spi' of git://git.secretlab.ca/git/linux-2.6
2010-08-14 11:54:09 -07:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
USB delay init quirk for logitech Harmony 700-series devices
2010-08-10 14:35:41 -07:00
ACPI / Sleep: Allow the NVS saving to be skipped during suspend to RAM
2010-07-24 23:26:09 -04:00
ahci_platform: Provide for vendor specific init
2010-08-01 19:36:03 -04:00
atm: propagate signal changes via notifier
2010-07-09 00:09:20 -07:00
gcc-4.6: fs: fix unused but set warnings
2010-08-07 18:23:12 +02:00
autofs/autofs4: Move compat_ioctl handling into fs
2010-08-09 00:13:34 +02:00
mm: fix writeback_in_progress()
2010-08-12 08:43:30 -07:00
bio, fs: separate out bio_types.h and define READ/WRITE constants in terms of BIO_RW_* flags
2010-08-07 18:53:10 +02:00
block: add secure discard
2010-08-12 08:43:30 -07:00
block: add secure discard
2010-08-12 08:43:30 -07:00
block: fix for block tracing build error
2010-08-07 18:26:29 +02:00
remove SWRITE* I/O types
2010-08-18 01:09:01 -04:00
security: move LSM xattrnames to xattr.h
2010-08-02 15:34:57 +10:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next-2.6
2010-08-04 11:47:58 -07:00
clocksource: Add __clocksource_updatefreq_hz/khz methods
2010-07-27 12:40:55 +02:00
coda: fixup clash with block layer REQ_* defines
2010-08-07 18:53:13 +02:00
Mark arguments to certain syscalls as being const
2010-08-13 16:53:13 -07:00
include/linux/compiler-gcc.h: use __same_type() in __must_be_array()
2010-08-09 20:45:04 -07:00
compiler-gcc.h: gcc-4.5 needs noclone and noinline on __naked functions
2010-06-29 15:29:31 -07:00
console: Fix compilation regression
2010-08-06 15:36:47 -05:00
tty: Move the vt_tty field from the vc_data into the standard tty_port
2010-08-10 13:47:42 -07:00
ACPI, APEI, Rename CPER and GHES severity constants
2010-08-08 14:55:26 -04:00
[CPUFREQ] unexport (un)lock_policy_rwsem* functions
2010-08-03 13:47:04 -04:00
cpuidle: extend cpuidle and menu governor to handle dynamic states
2010-08-09 20:45:04 -07:00
CRED: Fix __task_cred()'s lockdep check and banner comment
2010-07-29 15:16:18 -07:00
vfs: show unreachable paths in getcwd and proc
2010-08-11 00:29:47 -04:00
timer: Added usleep_range timer
2010-08-04 11:00:45 +02:00
dm: factor out max_io_len_target_boundary
2010-08-12 04:14:10 +01:00
driver core: device_rename's new_name can be const
2010-08-05 13:53:35 -07:00
dm: allow autoloading of dm mod
2010-08-12 04:14:05 +01:00
dma-mapping: fix build errors on !HAS_DMA architectures
2010-08-13 12:03:24 -07:00
DMAENGINE: generic slave channel control v3
2010-08-04 14:13:02 -07:00
PCI: export SMBIOS provided firmware instance and label to sysfs
2010-07-30 09:36:01 -07:00
dnotify: move dir_notify_enable declaration
2010-07-28 09:59:01 -04:00
DNS: Separate out CIFS DNS Resolver code
2010-08-05 17:17:51 +00:00
quota: Clean up the namespace in dqblk_xfs.h
2010-07-21 16:01:46 +02:00
drbd: revert "delay probes", feature is being re-implemented differently
2010-08-07 18:53:57 +02:00
drbd: revert "delay probes", feature is being re-implemented differently
2010-08-07 18:53:57 +02:00
module: initialize module dynamic debug later
2010-07-04 20:17:22 -07:00
eeprom_93cx6: Add support for 93c86 EEPROMs.
2010-06-30 15:00:50 -04:00
etherdevice.h: fix kernel-doc typo
2010-08-10 00:09:20 -07:00
Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
2010-07-07 15:59:38 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs-2.6
2010-08-10 11:26:52 -07:00
fanotify: userspace interface for permission responses
2010-07-28 09:59:02 -04:00
vt/console: try harder to print output when panicing
2010-08-10 13:47:40 -07:00
Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-08-06 09:23:07 -07:00
Merge firewire branches to be released post v2.6.35
2010-08-02 10:09:04 +02:00
firewire: add isochronous multichannel reception
2010-07-29 23:09:18 +02:00
flex_array: add helpers to get and put to make pointers easy to use
2010-08-09 20:45:09 -07:00
fs: scale files_lock
2010-08-18 08:35:48 -04:00
fs: fs_struct rwlock to spinlock
2010-08-18 08:35:46 -04:00
fscache: drop references to slow-work
2010-07-22 22:58:58 +02:00
fscache: fix missing kerneldoc annotation
2010-07-11 22:22:23 +02:00
powerpc/5121: move fsl-diu-fb.h to include/linux
2010-08-01 17:06:44 -06:00
Revert "fsnotify: store struct file not struct path"
2010-08-12 14:23:04 -07:00
Revert "fsnotify: store struct file not struct path"
2010-08-12 14:23:04 -07:00
tracing/documentation: Document dynamic ftracer internals
2010-07-21 11:00:25 -04:00
tracing: Reduce latency and remove percpu trace_seq
2010-07-20 22:05:34 -04:00
fuse: add retrieve request
2010-07-12 14:41:40 +02:00
Input: gpio_keys - add hooks to enable/disable device
2010-08-03 19:45:31 -07:00
HID: add HID_QUIRK_HIDINPUT_FORCE
2010-07-19 11:54:16 +02:00
gcc-4.6: mm: fix unused but set warnings
2010-08-09 20:44:58 -07:00
HWPOISON, hugetlb: isolate corrupted hugepage
2010-08-11 09:22:46 +02:00
hwpoison: rename CONFIG
2010-08-11 09:23:22 +02:00
i2c: Multiplexed I2C bus core support
2010-08-11 18:21:02 +02:00
i2c: Multiplexed I2C bus core support
2010-08-11 18:21:02 +02:00
Input: i8042 - mark stubs in i8042.h "static inline"
2010-06-30 01:21:38 -07:00
Revert "net: Make accesses to ->br_port safe for sparse RCU"
2010-07-01 12:45:34 -07:00
Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
2010-08-04 15:31:02 -07:00
Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/davem/net-2.6
2010-07-27 21:01:35 -07:00
inotify: force inotify and fsnotify use same bits
2010-07-28 10:18:49 -04:00
Input: dynamically allocate ABS information
2010-08-02 20:30:04 -07:00
intel_mid: Add Mrst & Mfld DMA Drivers
2010-07-27 23:32:57 -07:00
gpio: Add PMIC GPIO block support
2010-08-03 09:49:09 -04:00
irq: Add new IRQ flag IRQF_NO_SUSPEND
2010-07-29 13:24:57 +02:00
io-mapping: move asm include inside the config option
2010-08-12 11:47:50 +10:00
x86, ioremap: Fix incorrect physical address handling in PAE mode
2010-07-09 11:42:03 -07:00
iommu: inline iommu_num_pages
2010-08-09 20:45:05 -07:00
iommu-api: Extension to check for interrupt remapping
2010-07-19 15:44:25 +02:00
ipv6: Make IP6CB(skb)->nhoff 16-bit.
2010-07-19 22:01:26 -07:00
istallion: use bit ops for the board flags
2010-08-10 13:47:40 -07:00
Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
2010-08-07 13:03:53 -07:00
Merge git://git.infradead.org/mtd-2.6
2010-08-10 11:49:21 -07:00
mfd: Add JZ4740 ADC driver
2010-08-12 11:27:58 +02:00
include: replace unifdef-y with header-y
2010-08-14 22:26:51 +02:00
vt,console,kdb: automatically set kdb LINES variable
2010-08-05 09:22:30 -05:00
Add a dummy printk function for the maintenance of unused printks
2010-08-12 09:51:35 -07:00
kfifo: replace the old non generic API
2010-08-11 08:59:23 -07:00
gdbstub: Implement gdbserial 'p' and 'P' packets
2010-08-05 09:22:21 -05:00
ks8842: Support DMA when accessed via timberdale
2010-07-27 20:48:19 -07:00
ksm: fix ksm swapin time optimization
2010-08-09 20:44:58 -07:00
KVM: Remove memory alias support
2010-08-01 10:47:00 +03:00
KVM: Convert mask notifiers to use irqchip/pin instead of gsi
2010-08-02 06:40:39 +03:00
KVM: Use u64 for frame data types
2010-08-02 06:39:44 +03:00
lglock: introduce special lglock and brlock spin locks
2010-08-18 08:35:48 -04:00
libata: take advantage of cmwq and remove concurrency limitations
2010-07-02 10:59:24 +02:00
Merge branch 'master' into for-linus
2010-07-06 13:45:24 -04:00
AppArmor: basic auditing infrastructure.
2010-08-02 15:35:11 +10:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next-2.6
2010-08-04 11:47:58 -07:00
mbcache: Remove unused features
2010-08-09 16:48:45 -04:00
lmb: rename to memblock
2010-07-14 17:14:00 +10:00
memcg: remove nid and zid argument from mem_cgroup_soft_limit_reclaim()
2010-08-11 08:59:19 -07:00
oom: select task from tasklist for mempolicy ooms
2010-08-09 20:44:56 -07:00
dm: allow autoloading of dm mod
2010-08-12 04:14:05 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs-2.6
2010-08-10 11:26:52 -07:00
gcc-4.6: mm: fix unused but set warnings
2010-08-09 20:44:58 -07:00
vmscan: kill prev_priority completely
2010-08-09 20:45:00 -07:00
param: don't deref arg in __same_type() checks
2010-08-11 23:04:41 +09:30
vfs: remove unused MNT_STRICTATIME
2010-08-11 00:29:47 -04:00
PCI: MSI: Restore read_msi_msg_desc(); add get_cached_msi_msg_desc()
2010-07-30 09:41:39 -07:00
mv643xx_eth: use sw csum for big packets
2010-06-30 13:01:11 -07:00
linux/net.h: fix kernel-doc warnings
2010-07-02 21:59:08 -07:00
Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
2010-08-04 15:31:02 -07:00
net: make netpoll_rx return bool for !CONFIG_NETPOLL
2010-08-10 16:24:41 -07:00
NFSv4: Add support for the RELEASE_LOCKOWNER operation
2010-07-30 14:46:10 -04:00
Merge branch 'nfs-for-2.6.36' of git://git.linux-nfs.org/projects/trondmy/nfs-2.6
2010-08-07 13:19:36 -07:00
NFS: Fix /proc/mount for legacy binary interface
2010-08-06 13:41:39 -04:00
NFSv4: Ensure that we track the NFSv4 lock state in read/write requests.
2010-07-30 14:41:56 -04:00
NFSv4: Add support for the RELEASE_LOCKOWNER operation
2010-07-30 14:46:10 -04:00
nilfs2: reject filesystem with unsupported block size
2010-07-25 23:29:21 +09:00
of: Provide default of_node_to_nid() implementation.
2010-07-30 00:03:58 -06:00
of/address: Clean up function declarations
2010-08-01 01:42:42 -06:00
of/device: Replace struct of_device with struct platform_device
2010-08-06 09:25:50 -06:00
of/gpio: add default of_xlate function if device has a node pointer
2010-07-05 16:14:30 -06:00
of/i2c: Generalize OF support
2010-07-05 16:14:52 -06:00
of/irq: merge irq mapping code
2010-07-05 16:14:25 -06:00
of/device: Replace struct of_device with struct platform_device
2010-08-06 09:25:50 -06:00
of/spi: call of_register_spi_devices() from spi core code
2010-07-30 00:03:59 -06:00
Merge branch 'for-linus' of git://gitorious.org/linux-omap-dss2/linux
2010-08-08 10:02:59 -07:00
memcg: use find_lock_task_mm() in memory cgroups oom
2010-08-11 08:59:19 -07:00
padata: Remove padata_get_cpumask
2010-07-31 19:53:06 +08:00
slub: Use kmem_cache flags to detect if slab is in debugging mode.
2010-07-16 11:13:08 +03:00
Merge branch 'hwpoison' of git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-mce-2.6
2010-08-12 10:15:10 -07:00
vfs: show unreachable paths in getcwd and proc
2010-08-11 00:29:47 -04:00
dmaengine: Driver for Topcliff PCH DMA controller
2010-08-04 14:12:05 -07:00
PCI: disable mmio during bar sizing
2010-07-30 09:29:35 -07:00
watchdog: wdt_pci.c: move ids to pci_ids.h
2010-08-08 18:20:38 +00:00
tmpfs: add accurate compare function to percpu_counter library
2010-08-09 20:44:58 -07:00
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2010-08-06 09:39:22 -07:00
net: support time stamping in phy devices.
2010-07-18 19:15:26 -07:00
Driver core: reduce duplicated code for platform_device creation
2010-08-05 13:53:34 -07:00
plist: Add plist_last
2010-07-19 01:58:48 +02:00
PM / Runtime: Add runtime PM statistics (v3)
2010-07-19 02:01:06 +02:00
pm_qos: Get rid of the allocation in pm_qos_add_request()
2010-07-19 02:00:34 +02:00
PM: describe kernel policy regarding wakeup defaults (v. 2)
2010-07-19 01:58:48 +02:00
PNPACPI: Add support for remote wakeup
2010-07-19 01:58:48 +02:00
hugetlb, rmap: add reverse mapping for hugepage
2010-08-11 09:21:15 +02:00
rlimits: add task_struct to update_rlimit_cpu
2010-07-16 09:48:45 +02:00
ppp: make channel_ops const
2010-08-04 21:53:17 -07:00
net: added a BPF to help drivers detect PTP packets.
2010-07-18 19:15:26 -07:00
quota: Use mark_inode_dirty_sync instead of mark_inode_dirty
2010-07-23 12:50:46 +02:00
mm: implement writeback livelock avoidance using page tagging
2010-08-09 20:44:59 -07:00
rar: Move the RAR driver into the right place as its now clean
2010-08-03 09:48:50 -04:00
rbtree: Undo augmented trees performance damage and regression
2010-07-05 14:43:50 +02:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs-2.6
2010-08-10 11:26:52 -07:00
Fix reiserfs_file_release()
2010-08-09 16:47:27 -04:00
rlimits: redo do_setrlimit to more generic do_prlimit
2010-07-16 09:48:48 +02:00
Merge branch 'hwpoison' of git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-mce-2.6
2010-08-12 10:15:10 -07:00
net: RTA_MARK addition
2010-07-22 13:46:21 -07:00
Add s3c-adc-battery driver
2010-07-23 17:08:41 +04:00
oom: badness heuristic rewrite
2010-08-09 20:45:02 -07:00
Merge branch 'writable_limits' of git://decibel.fi.muni.cz/~xslaby/linux
2010-08-10 12:07:51 -07:00
U6715 16550A serial driver support
2010-08-10 13:47:46 -07:00
U6715 16550A serial driver support
2010-08-10 13:47:46 -07:00
Merge branch 'for-linus' of master.kernel.org:/home/rmk/linux-2.6-arm
2010-08-11 09:13:19 -07:00
hsu: driver for Medfield High Speed UART device
2010-08-10 13:47:46 -07:00
hsu: driver for Medfield High Speed UART device
2010-08-10 13:47:46 -07:00
sh: add a reparent function to DIV6 clocks
2010-08-04 16:12:01 +09:00
tmpfs: make tmpfs scalable with percpu_counter for used blocks
2010-08-09 20:44:58 -07:00
sk_buff: introduce pskb_network_may_pull()
2010-08-04 21:53:14 -07:00
slab: fix caller tracking on !CONFIG_DEBUG_SLAB && CONFIG_TRACING
2010-07-04 19:48:33 +03:00
dma-mapping: rename ARCH_KMALLOC_MINALIGN to ARCH_DMA_MINALIGN
2010-08-11 08:59:21 -07:00
dma-mapping: rename ARCH_KMALLOC_MINALIGN to ARCH_DMA_MINALIGN
2010-08-11 08:59:21 -07:00
dma-mapping: rename ARCH_KMALLOC_MINALIGN to ARCH_DMA_MINALIGN
2010-08-11 08:59:21 -07:00
add f_flags to struct statfs(64)
2010-08-09 16:48:44 -04:00
PM / Suspend: Fix ordering of calls in suspend error paths
2010-07-19 02:00:35 +02:00
memcg: mem_cgroup_shrink_node_zone() doesn't need sc.nodemask
2010-08-11 08:59:19 -07:00
Mark arguments to certain syscalls as being const
2010-08-13 16:53:13 -07:00
sysfs: Remove owner field from sysfs struct attribute
2010-08-05 13:53:35 -07:00
sysrq,kdb: Use __handle_sysrq() for kdb's sysrq function
2010-07-21 19:27:07 -05:00
fs/sysv: v7: adjust sanity checks for some volumes
2010-08-11 08:59:22 -07:00
Mark arguments to certain syscalls as being const
2010-08-13 16:53:13 -07:00
timer: add on-stack deferrable timer interfaces
2010-08-03 09:48:45 -04:00
topology: alternate fix for ia64 tiger_defconfig build breakage
2010-08-09 20:44:57 -07:00
tty: fix fu_list abuse
2010-08-18 08:35:47 -04:00
Merge branch 'master' into for-linus
2010-08-06 10:37:02 -04:00
USB:: fix linux/usb.h kernel-doc warnings
2010-08-10 14:35:44 -07:00
vgaarb: drop vga.h include
2010-08-12 11:47:26 +10:00
V4L/DVB: v4l: Add driver for Samsung S5P SoC video postprocessor
2010-08-08 23:43:01 -03:00
virtio_9p.h: include linux/types.h
2010-08-11 08:59:02 -07:00
Merge branch 'stable/xen-swiotlb-0.8.6' of git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen
2010-08-12 09:09:41 -07:00
vt: Fix warning: statement with no effect due to vt_kern.h
2010-08-10 13:47:44 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
2010-08-07 12:42:58 -07:00
writeback: avoid unnecessary calculation of bdi dirty thresholds
2010-08-12 08:43:29 -07:00
security: move LSM xattrnames to xattr.h
2010-08-02 15:34:57 +10:00
