Johannes Weiner
449dd6984d
mm: keep page cache radix tree nodes in check ...
Previously, page cache radix tree nodes were freed after reclaim emptied
out their page pointers. But now reclaim stores shadow entries in their
place, which are only reclaimed when the inodes themselves are
reclaimed. This is problematic for bigger files that are still in use
after they have a significant amount of their cache reclaimed, without
any of those pages actually refaulting. The shadow entries will just
sit there and waste memory. In the worst case, the shadow entries will
accumulate until the machine runs out of memory.
To get this under control, the VM will track radix tree nodes
exclusively containing shadow entries on a per-NUMA node list. Per-NUMA
rather than global because we expect the radix tree nodes themselves to
be allocated node-locally and we want to reduce cross-node references of
otherwise independent cache workloads. A simple shrinker will then
reclaim these nodes on memory pressure.
A few things need to be stored in the radix tree node to implement the
shadow node LRU and allow tree deletions coming from the list:
1. There is no index available that would describe the reverse path
from the node up to the tree root, which is needed to perform a
deletion. To solve this, encode in each node its offset inside the
parent. This can be stored in the unused upper bits of the same
member that stores the node's height at no extra space cost.
2. The number of shadow entries needs to be counted in addition to the
regular entries, to quickly detect when the node is ready to go to
the shadow node LRU list. The current entry count is an unsigned
int but the maximum number of entries is 64, so a shadow counter
can easily be stored in the unused upper bits.
3. Tree modification needs tree lock and tree root, which are located
in the address space, so store an address_space backpointer in the
node. The parent pointer of the node is in a union with the 2-word
rcu_head, so the backpointer comes at no extra cost as well.
4. The node needs to be linked to an LRU list, which requires a list
head inside the node. This does increase the size of the node, but
it does not change the number of objects that fit into a slab page.
[akpm@linux-foundation.org: export the right function]
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Rik van Riel <riel@redhat.com>
Reviewed-by: Minchan Kim <minchan@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Bob Liu <bob.liu@oracle.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jan Kara <jack@suse.cz>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Luigi Semenzato <semenzato@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Metin Doslu <metin@citusdata.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: Ozgun Erdogan <ozgun@citusdata.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Roman Gushchin <klamm@yandex-team.ru>
Cc: Ryan Mallon <rmallon@gmail.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-04-03 16:21:01 -07:00
Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus
2014-01-30 17:20:32 -08:00
can: allow to change the device mtu for CAN FD capable devices
2014-03-07 09:18:23 +01:00
ceph: remove xattr when null value is given to setxattr()
2014-02-17 12:37:09 -08:00
ARM: OMAP2+: clock: fix clkoutx2 with CLK_SET_RATE_PARENT
2014-02-19 12:07:55 -07:00
extcon: Move OF helper function to extcon core and change function name
2014-03-19 14:41:58 +09:00
fsl /bestcomm
gpio: consumer.h: Move forward declarations outside #ifdef
2014-02-06 10:22:56 +01:00
treewide: Fix typo in Documentation/DocBook
2014-02-19 14:58:17 +01:00
iio:drop IIO_ST macro
2014-02-15 10:58:17 +00:00
sound updates for 3.15-rc1
2014-04-01 15:38:47 -07:00
net/mlx4: Implement vxlan ndo calls
2014-03-28 16:29:35 -04:00
mlx5: Add include of <linux/slab.h> because of kzalloc()/kfree() use
2014-02-13 20:48:02 -08:00
brcmfmac: add BCM4354 SDIO interface support
2014-03-17 13:44:14 -04:00
netfilter: ipset: add forceadd kernel support for hash set types
2014-03-06 09:31:43 +01:00
phy: rename struct omap_control_usb to struct omap_control_phy
2014-03-09 12:45:08 +05:30
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
2014-04-02 16:23:38 -07:00
Merge remote-tracking branches 'regulator/topic/max8973', 'regulator/topic/max8997', 'regulator/topic/max8998', 'regulator/topic/mc13xxx', 'regulator/topic/pfuze100', 'regulator/topic/rc5t583' and 'regulator/topic/s2mps11' into regulator-next
2014-03-26 16:58:11 +00:00
sched/prio: Add 3 macros of MAX_NICE, MIN_NICE and NICE_WIDTH in prio.h
2014-02-22 18:14:13 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
2014-04-02 16:23:38 -07:00
Merge branch 'for-3.14' of git://linux-nfs.org/~bfields/linux
2014-01-30 10:18:43 -08:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
2014-04-02 20:53:45 -07:00
PCI changes for the v3.15 merge window:
2014-04-01 15:14:04 -07:00
ahci_platform: Drop unused ahci_platform_data members
2014-02-22 17:13:01 -05:00
ASoC: atmel_ssc_dai: make option to choose clock
2014-02-12 17:21:22 +00:00
audit: Send replies in the proper network namespace.
2014-02-28 19:44:55 -08:00
bdi: avoid oops on device removal
2014-04-03 16:20:49 -07:00
execve: use 'struct filename *' for executable name passing
2014-02-05 12:54:53 -08:00
block: Fix cloning of discard/write same bios
2014-02-11 08:40:45 -07:00
ext4: atomically set inode->i_flags in ext4_set_inode_flags()
2014-03-30 17:02:06 -07:00
block: remove old blk_iopoll_enabled variable
2014-03-13 09:38:42 -06:00
Merge branch 'for-3.15/core' of git://git.kernel.dk/linux-block
2014-04-01 19:19:15 -07:00
block: Stop abusing csd.list for fifo_time
2014-02-24 14:46:32 -08:00
memblock, bootmem: restore goal for alloc_low
2014-01-30 16:56:54 -08:00
net: phy: add Broadcom BCM7xxx internal PHY driver
2014-02-14 00:27:58 -05:00
crypto: ccp - Move HMAC calculation down to ccp ops file
2014-02-09 09:59:23 +08:00
cgroup: protect modifications to cgroup_idr with cgroup_mutex
2014-02-11 10:38:30 -05:00
Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc
2014-01-30 17:07:18 -08:00
tick: Fixup more fallout from hrtimer broadcast mode
2014-02-09 15:11:47 +01:00
Merge branch 'x86-x32-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2014-04-02 12:51:41 -07:00
compiler/gcc4: Make quirk for asm_volatile_goto() unconditional
2014-02-13 12:34:05 +01:00
connector: add portid to unicast in addition to broadcasting
2014-02-07 15:40:17 -08:00
x86: align x86 arch with generic CPU modalias handling
2014-02-18 12:45:38 -08:00
cpu: add generic support for CPU feature based module autoloading
2014-02-18 12:38:37 -08:00
cpufreq: Make cpufreq_notify_transition & cpufreq_notify_post_transition static
2014-03-26 16:41:41 +01:00
sched/idle: Reorganize the idle loop
2014-03-11 11:52:47 +01:00
mm: optimize put_mems_allowed() usage
2014-04-03 16:20:58 -07:00
cputime: Bring cputime -> nsecs conversion
2014-03-13 15:56:44 +01:00
Driver core / sysfs patches for 3.15-rc1
2014-04-01 16:28:19 -07:00
dma-buf: update debugfs output
2014-02-13 10:08:52 +05:30
Merge branch 'for-linus' of git://git.infradead.org/users/vkoul/slave-dma
2014-01-29 20:27:23 -08:00
drbd: Move string function prototypes from linux/drbd.h to drbd_string.h
2014-02-17 16:50:37 +01:00
drbd: Define the size of res_opts->cpu_mask in a single place
2014-02-17 16:46:48 +01:00
Merge remote-tracking branch 'tip/x86/efi-mixed' into efi-for-mingo
2014-03-05 18:15:37 +00:00
block: Remove useless IPI struct initialization
2014-02-24 14:46:38 -08:00
ethtool: Document the general convention for VLAs in kernel space
2014-02-12 17:33:03 -05:00
extcon: Move OF helper function to extcon core and change function name
2014-03-19 14:41:58 +09:00
get rid of fget_light()
2014-03-10 11:44:42 -04:00
net: filter: rework/optimize internal BPF interpreter's instruction set
2014-03-31 00:45:09 -04:00
firewire: don't use PREPARE_DELAYED_WORK
2014-03-07 10:19:57 -05:00
FMC: show_sdb_tree: dump synthesis/commit ID info
2014-02-28 15:12:09 -08:00
mm + fs: store shadow entries in page cache
2014-04-03 16:21:01 -07:00
driver/memory:Move Freescale IFC driver to a common driver
2014-02-18 12:20:45 -08:00
fanotify: convert access_mutex to spinlock
2014-04-03 16:20:51 -07:00
tracing: Fix array size mismatch in format string
2014-03-20 13:21:05 -04:00
futex: Allow architectures to skip futex_atomic_cmpxchg_inatomic() test
2014-03-03 11:32:08 +01:00
mm: fix GFP_THISNODE callers and clarify
2014-03-10 17:26:19 -07:00
Merge branch 'master' into for-next
2014-02-20 14:54:28 +01:00
genirq: Provide synchronize_hardirq()
2014-02-19 17:22:44 +01:00
HID: hid-sensor-hub: Processing for duplicate physical ids
2014-02-17 17:12:47 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/hid
2014-04-02 16:24:28 -07:00
Merge branches 'for-3.15/multitouch', 'for-3.15/sony' and 'for-3.15/uhid' into for-linus
2014-04-01 19:06:50 +02:00
hrtimer: Rearrange comments in the order struct members are declared
2014-03-20 12:35:45 +01:00
mm: close PageTail race
2014-03-04 07:55:47 -08:00
mm, hugetlb: fix race in region tracking
2014-04-03 16:20:59 -07:00
Drivers: hv: vmbus: Increase the limit on the number of pfns we can handle
2014-03-08 22:15:52 -08:00
Update Jean Delvare's e-mail address
2014-01-29 20:40:08 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
2014-04-02 16:23:38 -07:00
Merge branch 'for-john' of git://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211-next
2014-03-20 11:53:20 -04:00
vlan: make a new function vlan_dev_vlan_proto() and export
2014-03-26 16:41:28 -04:00
lto, workaround: Add workaround for initcall reordering
2014-02-13 20:24:13 -08:00
genirq: Provide irq_wake_thread()
2014-02-19 17:22:44 +01:00
asmlinkage: Make __iowrite32_copy visible
2014-02-13 18:12:04 -08:00
vsprintf: Add support for IORESOURCE_UNSET in %pR
2014-02-26 14:42:09 -07:00
ipc: change kern_ipc_perm.deleted type to bool
2014-01-27 21:02:39 -08:00
ipc,mqueue: remove limits for the amount of system-wide queues
2014-02-25 15:25:45 -08:00
genirq: Add a new IRQCHIP_EOI_THREADED flag
2014-03-14 13:43:33 +01:00
perf/x86: Warn to early_printk() in case irq_work is too slow
2014-02-21 21:49:07 +01:00
net: isdn: use sk_unattached_filter api
2014-03-31 00:45:09 -04:00
Rename TAINT_UNSAFE_SMP to TAINT_CPU_OUT_OF_SPEC
2014-03-20 16:28:09 -07:00
Merge branch 'irq-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2014-04-01 11:22:57 -07:00
kernfs: cache atomic_write_len in kernfs_open_file
2014-03-08 22:08:29 -08:00
kexec/compat: convert to COMPAT_SYSCALL_DEFINE with changing parameter types
2014-03-06 16:30:46 +01:00
treewide: Fix typo in Documentation/DocBook
2014-02-19 14:58:17 +01:00
kmemleak: remove redundant code
2014-04-03 16:20:50 -07:00
KVM: Bump KVM_MAX_IRQ_ROUTES for s390
2014-03-21 13:43:13 +01:00
libata: remove unused ata_sas_port_async_resume() stub
2014-03-19 16:30:23 -04:00
lto: Make asmlinkage __visible
2014-02-13 20:21:59 -08:00
mm: keep page cache radix tree nodes in check
2014-04-03 16:21:01 -07:00
Merge branch 'x86-asmlinkage-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2014-03-31 14:13:25 -07:00
drivers: Introduce MEN Chameleon Bus
2014-02-28 15:36:37 -08:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
2014-04-02 20:53:45 -07:00
mm + fs: store shadow entries in page cache
2014-04-03 16:21:01 -07:00
mm: keep page cache radix tree nodes in check
2014-04-03 16:21:01 -07:00
Driver core / sysfs patches for 3.15-rc1
2014-04-01 16:28:19 -07:00
UAPI: add MPLS label stack definition
2014-03-04 13:51:06 -05:00
ipc: whitespace cleanup
2014-01-27 21:02:39 -08:00
locking/mutexes: Introduce cancelable MCS lock for adaptive spinning
2014-03-11 12:14:56 +01:00
vlan: Warn the user if lowerdev has bad vlan features.
2014-03-28 17:16:51 -04:00
net-gro: restore frag0 optimization
2014-03-31 16:26:40 -04:00
netpoll: Rename netpoll_rx_enable/disable to netpoll_poll_disable/enable
2014-03-29 17:58:37 -04:00
NFS client bugfixes for Linux 3.14
2014-01-31 15:39:07 -08:00
NFSv4: Fix another nfs4_sequence corruptor
2014-03-01 13:51:53 -06:00
mac802154: make csma/cca parameters per-wpan
2014-04-01 16:25:51 -04:00
nvme: don't use PREPARE_WORK
2014-03-07 10:24:49 -05:00
Devicetree changes for v3.15
2014-04-02 14:27:15 -07:00
of/device: Nullify match table in of_match_device() for CONFIG_OF=n
2014-02-05 10:04:37 -06:00
drivers: of: add initialization code for static reserved memory
2014-03-11 17:26:47 +00:00
of: Fix of_graph_parse_endpoint stub for !CONFIG_OF builds
2014-03-07 16:02:46 +01:00
drivers: of: add support for custom reserved memory drivers
2014-03-11 17:27:12 +00:00
swap: add a simple detector for inappropriate swapin readahead
2014-02-06 13:48:51 -08:00
mm + fs: store shadow entries in page cache
2014-04-03 16:21:01 -07:00
mm + fs: prepare for non-page entries in page cache radix trees
2014-04-03 16:21:00 -07:00
ACPI / hotplug / PCI: Rework acpiphp_check_host_bridge()
2014-02-06 17:31:52 +01:00
Merge branch 'pci/resource' into next
2014-03-19 15:11:19 -06:00
SCSI misc on 20140401
2014-04-01 18:49:04 -07:00
net: phy: allow PHY drivers to implement their own software reset
2014-02-17 16:40:09 -05:00
Merge branch 'master' into for-next
2014-02-20 14:54:28 +01:00
Merge branches 'pm-runtime' and 'pm-sleep'
2014-03-20 13:25:54 +01:00
PM / QoS: Add type to dev_pm_qos_add_ancestor_request() arguments
2014-02-11 00:36:00 +01:00
PM: Add pm_runtime_suspend|resume_force functions
2014-03-02 00:18:15 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2014-01-28 08:38:04 -08:00
net: ptp: move PTP classifier in its own file
2014-04-01 16:43:18 -04:00
ptp: introduce programmable pins.
2014-03-21 14:21:13 -04:00
quota: provide function to grab quota structure reference
2014-04-03 16:20:54 -07:00
mm: keep page cache radix tree nodes in check
2014-04-03 16:21:01 -07:00
rcu: Indentation and spacing fixes.
2014-02-17 15:01:52 -08:00
Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2014-03-31 11:05:24 -07:00
rcu: Provide grace-period piggybacking API
2014-03-20 17:12:25 -07:00
rcu: Provide grace-period piggybacking API
2014-03-20 17:12:25 -07:00
Char/Misc driver patches for 3.15-rc1
2014-04-01 16:13:21 -07:00
mm: fix swapops.h:131 bug if remap_file_pages raced migration
2014-03-20 22:09:09 -07:00
rwsem: add rwsem_is_contended
2014-01-28 13:20:09 -08:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2014-04-01 11:00:07 -07:00
net: filter: rework/optimize internal BPF interpreter's instruction set
2014-03-31 00:45:09 -04:00
selinux: add gfp argument to security_xfrm_policy_alloc and fix callers
2014-03-10 08:30:02 +01:00
tty: serial: bcm63xx_uart: define UART_REG_SIZE constant
2014-02-28 16:27:18 -08:00
serial: sh-sci: Add more register documentation
2014-03-17 16:20:49 -07:00
ipc: whitespace cleanup
2014-01-27 21:02:39 -08:00
mm + fs: prepare for non-page entries in page cache radix trees
2014-04-03 16:21:00 -07:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
2014-04-02 20:53:45 -07:00
mm: fix GFP_THISNODE callers and clarify
2014-03-10 17:26:19 -07:00
smp: Rename __smp_call_function_single() to smp_call_function_single_async()
2014-02-24 14:47:15 -08:00
splice: fix unexpected size truncation
2014-01-27 21:02:40 -08:00
spmi: Linux driver framework for SPMI
2014-02-15 11:55:28 -08:00
rcu: Stop tracking FSF's postal address
2014-02-17 15:01:37 -08:00
mm: keep page cache radix tree nodes in check
2014-04-03 16:21:01 -07:00
net: sxgbe: add basic framework for Samsung 10Gb ethernet driver
2014-03-26 16:49:31 -04:00
s390/compat: automatic zero, sign and pointer conversion of syscalls
2014-03-04 09:12:24 +01:00
Revert "sysfs, driver-core: remove unused {sysfs|device}_schedule_callback_owner()"
2014-03-25 20:54:57 -07:00
tcp: switch rtt estimations to usec resolution
2014-02-26 17:08:40 -05:00
rcutorture: Stop generic kthreads in torture_cleanup()
2014-02-23 09:04:27 -08:00
tracing: Do not add event files for modules that fail tracepoints
2014-03-03 21:11:05 -05:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
2014-04-02 20:53:45 -07:00
tty: tty_ldisc.h: Remove duplicate include
2014-02-18 12:50:26 -08:00
net: Replace u64_stats_fetch_begin_bh to u64_stats_fetch_begin_irq
2014-03-14 22:41:36 -04:00
USB patches for 3.15-rc1
2014-04-01 17:06:09 -07:00
uas: Add the posibilty to blacklist uas devices from using the uas driver
2014-03-04 15:38:16 -08:00
mm: vmstat: fix UP zone state accounting
2014-04-03 16:21:00 -07:00
Revert "sched/wait: Suppress Sparse 'variable shadowing' warning"
2014-02-27 12:20:31 +01:00
wl1251: move power GPIO handling into the driver
2014-02-28 14:08:26 -05:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2014-04-01 11:00:07 -07:00
Revert "writeback: do not sync data dirtied after sync start"
2014-02-22 02:02:28 +01:00
zsmalloc: add copyright
2014-01-30 16:56:55 -08:00
