Johannes Weiner
3e32cb2e0a
mm: memcontrol: lockless page counters ...
Memory is internally accounted in bytes, using spinlock-protected 64-bit
counters, even though the smallest accounting delta is a page. The
counter interface is also convoluted and does too many things.
Introduce a new lockless word-sized page counter API, then change all
memory accounting over to it. The translation from and to bytes then only
happens when interfacing with userspace.
The removed locking overhead is noticable when scaling beyond the per-cpu
charge caches - on a 4-socket machine with 144-threads, the following test
shows the performance differences of 288 memcgs concurrently running a
page fault benchmark:
vanilla:
18631648.500498 task-clock (msec) # 140.643 CPUs utilized ( +- 0.33% )
1,380,638 context-switches # 0.074 K/sec ( +- 0.75% )
24,390 cpu-migrations # 0.001 K/sec ( +- 8.44% )
1,843,305,768 page-faults # 0.099 M/sec ( +- 0.00% )
50,134,994,088,218 cycles # 2.691 GHz ( +- 0.33% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
8,049,712,224,651 instructions # 0.16 insns per cycle ( +- 0.04% )
1,586,970,584,979 branches # 85.176 M/sec ( +- 0.05% )
1,724,989,949 branch-misses # 0.11% of all branches ( +- 0.48% )
132.474343877 seconds time elapsed ( +- 0.21% )
lockless:
12195979.037525 task-clock (msec) # 133.480 CPUs utilized ( +- 0.18% )
832,850 context-switches # 0.068 K/sec ( +- 0.54% )
15,624 cpu-migrations # 0.001 K/sec ( +- 10.17% )
1,843,304,774 page-faults # 0.151 M/sec ( +- 0.00% )
32,811,216,801,141 cycles # 2.690 GHz ( +- 0.18% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
9,999,265,091,727 instructions # 0.30 insns per cycle ( +- 0.10% )
2,076,759,325,203 branches # 170.282 M/sec ( +- 0.12% )
1,656,917,214 branch-misses # 0.08% of all branches ( +- 0.55% )
91.369330729 seconds time elapsed ( +- 0.45% )
On top of improved scalability, this also gets rid of the icky long long
types in the very heart of memcg, which is great for 32 bit and also makes
the code a lot more readable.
Notable differences between the old and new API:
- res_counter_charge() and res_counter_charge_nofail() become
page_counter_try_charge() and page_counter_charge() resp. to match
the more common kernel naming scheme of try_do()/do()
- res_counter_uncharge_until() is only ever used to cancel a local
counter and never to uncharge bigger segments of a hierarchy, so
it's replaced by the simpler page_counter_cancel()
- res_counter_set_limit() is replaced by page_counter_limit(), which
expects its callers to serialize against themselves
- res_counter_memparse_write_strategy() is replaced by
page_counter_limit(), which rounds down to the nearest page size -
rather than up. This is more reasonable for explicitely requested
hard upper limits.
- to keep charging light-weight, page_counter_try_charge() charges
speculatively, only to roll back if the result exceeds the limit.
Because of this, a failing bigger charge can temporarily lock out
smaller charges that would otherwise succeed. The error is bounded
to the difference between the smallest and the biggest possible
charge size, so for memcg, this means that a failing THP charge can
send base page charges into reclaim upto 2MB (4MB) before the limit
would have been reached. This should be acceptable.
[akpm@linux-foundation.org: add includes for WARN_ON_ONCE and memparse]
[akpm@linux-foundation.org: add includes for WARN_ON_ONCE, memparse, strncmp, and PAGE_SIZE]
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Vladimir Davydov <vdavydov@parallels.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-12-10 17:41:04 -08:00
net/9p: remove a comment about pref member which doesn't exist
2014-11-06 14:59:19 -05:00
Bluetooth: Fix reason code used for rejecting SCO connections
2014-09-24 14:03:32 +02:00
caif_hsi.h: Remove extern from function prototypes
2013-09-23 16:29:41 -04:00
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
af_iucv: fix recvmsg by replacing skb_pull() function
2013-04-08 17:16:57 -04:00
netfilter: nf_tables: restore synchronous object release from commit/abort
2014-11-12 12:06:24 +01:00
ipv6: make fib6 serial number per namespace
2014-10-07 00:02:30 -04:00
NFC: nci: Add support for proprietary RF Protocols
2014-09-24 02:02:24 +02:00
net: sctp: fix panic on duplicate ASCONF chunks
2014-10-14 12:46:22 -04:00
net_sched: act: hide struct tcf_common from API
2014-02-12 19:23:32 -05:00
6lowpan: remove unused function
2014-07-30 19:28:41 +02:00
net_sched: act: refuse to remove bound action outside
2014-02-12 19:23:32 -05:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2014-09-23 12:09:27 -04:00
ieee802154: add dgram sockopts for security control
2014-05-16 17:23:41 -04:00
af_rxrpc.h: Remove extern from function prototypes
2013-07-31 17:50:01 -07:00
af_unix: improve STREAM behavior with fragmented memory
2013-08-10 01:16:44 -07:00
vsock: Make transport the proto owner
2014-05-05 13:13:50 -04:00
ipsec: Remove obsolete MAX_AH_AUTH_LEN
2014-09-18 10:54:36 +02:00
arp: make arp_invalidate static
2013-12-28 17:02:46 -05:00
ax25.h: Remove extern from function prototypes
2013-07-31 17:50:02 -07:00
sched, net: Fixup busy_loop_us_clock()
2014-01-13 17:39:11 +01:00
cfg80211: allow requesting SMPS mode on ap start
2014-09-11 13:37:02 +02:00
net: optimise csum_replace4()
2014-09-26 16:14:16 -04:00
cipso: cleanup cipso_v4_translate() when !CONFIG_NETLABEL
2013-12-10 17:56:54 -05:00
cgroup: clean up cgroup_subsys names and initialization
2014-02-08 10:36:58 -05:00
net: use ktime_get_ns() and ktime_get_real_ns() helpers
2014-08-22 19:57:23 -07:00
compat.h: Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
net: Move prototype declaration to header file include/net/datalink.h from net/ipx/af_ipx.c
2014-02-09 17:32:50 -08:00
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
Update setapp/getapp prototypes in dcbnl_rtnl_ops to return int instead of u8
2014-07-17 16:02:29 -07:00
net: Move prototype declaration to header file include/net/dn.h from net/decnet/af_decnet.c
2014-02-09 17:32:49 -08:00
dn_dev: add support for IFA_FLAGS nl attribute
2013-12-10 21:50:00 -05:00
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
net: Move prototype declaration to appropriate header file from decnet/af_decnet.c
2014-02-09 17:32:49 -08:00
net: dsa: add includes for ethtool and phy_fixed definitions
2014-10-17 23:54:46 -04:00
xfrm: Generate queueing routes only from route lookup functions
2014-09-16 10:08:49 +02:00
percpu_counter: add @gfp to percpu_counter_init()
2014-09-08 09:51:29 +09:00
net: move pskb_put() to core code
2013-11-07 19:28:58 -05:00
net: ethoc: set up MII management bus clock
2014-02-04 20:19:51 -08:00
fib_rules.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
firewire net, ipv4 arp: Extend hardware address and remove driver-level packet inspection.
2013-03-26 12:32:13 -04:00
ipv4, fib: pass LOOPBACK_IFINDEX instead of 0 to flowi4_iif
2014-04-16 15:05:11 -04:00
net: Add function for parsing the header length out of linear ethernet frames
2014-09-05 17:47:02 -07:00
flowcache: Make flow cache name space aware
2014-02-12 07:02:11 +01:00
garp.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
net: sched: enable per cpu qstats
2014-09-30 01:02:26 -04:00
genetlink: add function genl_has_listeners()
2014-09-19 17:28:26 -04:00
openvswitch: fix a compilation error when CONFIG_INET is not setW!
2014-10-07 00:10:49 -04:00
gre: Call gso_make_checksum
2014-06-04 22:46:38 -07:00
gue: Receive side for Generic UDP Encapsulation
2014-10-03 16:53:33 -07:00
icmp.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
mac80211: propagate STBC / LDPC flags to radiotap
2014-02-06 09:34:58 +01:00
ieee802154: add definitions for link-layer security and header functions
2014-05-15 15:51:42 -04:00
ieee802154, mac802154: implement devkey record option
2014-05-16 17:23:42 -04:00
ipv6: remove aca_lock spinlock from struct ifacaddr6
2014-10-14 13:15:15 -04:00
ipv4: add a sock pointer to ip_queue_xmit()
2014-04-15 12:58:34 -04:00
ipv6: introduce tcp_v6_iif()
2014-10-17 23:48:07 -04:00
net-timestamp: make tcp_recvmsg call ipv6_recv_error for AF_INET6 socks
2014-11-26 15:45:04 -05:00
tcp: avoid possible arithmetic overflows
2014-09-22 16:27:10 -04:00
tunnel: fix RFC number in comment for INET_ECN_decapsulate()
2014-05-07 15:30:52 -04:00
percpu_counter: add @gfp to percpu_counter_init()
2014-09-08 09:51:29 +09:00
net: Use a more standard macro for INET_ADDR_COOKIE
2014-05-14 16:07:23 -04:00
net: remove inet6_reqsk_alloc
2014-06-27 15:53:35 -07:00
inet: move ipv6only in sock_common
2014-07-01 23:46:21 -07:00
inet: remove dead inetpeer sequence code
2014-09-08 16:42:42 -07:00
ipv4: rename ip_options_echo to __ip_options_echo()
2014-09-28 16:35:42 -04:00
net: add gro_compute_pseudo functions
2014-08-24 18:09:23 -07:00
ipv6: make rt_sernum atomic and serial number fields ordinary ints
2014-10-07 00:02:30 -04:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2014-05-24 00:32:30 -04:00
net: unify the pcpu_tstats and br_cpu_netstats as one
2014-01-04 20:10:24 -05:00
ipv4: harden fnhe_hashfun()
2014-09-05 17:40:33 -07:00
openvswitch: Add support for Geneve tunneling.
2014-10-06 00:32:21 -04:00
ipvs: Clean up comment style in ip_vs.h
2014-10-02 18:30:58 +02:00
drivers/net, ipv6: Select IPv6 fragment idents for virtio UFO packets
2014-10-30 20:01:18 -04:00
net: Move prototype declaration to header file include/net/ipx.h from net/ipx/af_ipx.c
2014-02-09 17:32:50 -08:00
iw_handler.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
lapb.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
lib80211: remove unused print_ssid()
2014-10-14 02:18:27 +02:00
llc: make lock static
2014-01-03 20:56:48 -05:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
net: llc: fix order of evaluation in llc_conn_ac_inc_vr_by_1
2014-01-01 22:22:43 -05:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
mac80211: replace SMPS hw flags with wiphy feature bits
2014-09-11 13:37:02 +02:00
mac802154: at86rf230: add hw flags and merge ops
2014-07-07 21:29:24 -07:00
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
ipv6: mld: answer mldv2 queries with mldv1 reports in mldv1 fallback
2014-09-22 16:23:15 -04:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2013-10-01 17:06:14 -04:00
ndisc.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
netfilter: bridge: move br_netfilter out of the core
2014-09-26 18:42:31 +02:00
ipv6: remove rt6i_genid
2014-09-30 14:00:48 -04:00
netevent/netlink.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
netlabel: fix the netlbl_catmap_setlong() dummy function
2014-08-07 20:55:21 -04:00
netlink: fix description of portid
2014-10-16 14:52:35 -04:00
cgroup: clean up cgroup_subsys names and initialization
2014-02-08 10:36:58 -05:00
netrom.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
ieee802154: use ieee802154_addr instead of *_sa variants
2014-03-14 22:15:26 -04:00
p8022.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
ipv6: make IPV6_RECVPKTINFO work for ipv4 datagrams
2014-01-19 19:53:18 -08:00
net: sched: remove tcf_proto from ematch calls
2014-10-06 18:02:32 -04:00
net: sched: avoid costly atomic operation in fq_dequeue()
2014-10-06 00:55:10 -04:00
net: Eliminate no_check from protosw
2014-05-23 16:28:53 -04:00
psnap.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
raw/rawv6.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
raw/rawv6.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
reciprocal_divide: update/correction of the algorithm
2014-01-21 23:17:20 -08:00
regulatory: add NUL to alpha2
2014-08-15 13:51:40 +02:00
inet: includes a sock_common in request_sock
2013-10-10 00:08:07 -04:00
rose.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
ipv4: remove ip_rt_dump from route.c
2014-03-24 12:45:01 -04:00
net: rtnetlink - make create_link take name_assign_type
2014-07-15 16:13:07 -07:00
qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE
2014-10-03 12:37:06 -07:00
scm.h: Remove extern from function prototypes
2013-09-23 01:51:09 -04:00
inetpeer: get rid of ip_id_count
2014-06-02 11:00:41 -07:00
Merge branch 'for-3.18-consistent-ops' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu
2014-10-15 07:48:18 +02:00
mm: memcontrol: lockless page counters
2014-12-10 17:41:04 -08:00
drivers: net: Include new header file in sbni.c
2013-12-19 18:51:20 -05:00
stp.h: Remove extern from function prototypes
2013-09-23 01:51:09 -04:00
tcp: fix build error if IPv6 is not enabled
2014-10-18 13:01:36 -04:00
tcp_memcontrol: Kill struct tcp_memcontrol
2013-10-21 18:43:02 -04:00
ipv6: make IPV6_RECVPKTINFO work for ipv4 datagrams
2014-01-19 19:53:18 -08:00
net: Add a software TSO helper API
2014-05-22 14:57:15 -04:00
udp: Generalize skb_udp_segment
2014-10-01 21:35:51 -04:00
udptunnel: Add SKB_GSO_UDP_TUNNEL during gro_complete.
2014-11-10 15:09:45 -05:00
udplite.h: Remove extern from function prototypes
2013-09-23 16:29:40 -04:00
VSOCK: Move af_vsock.h and vsock_addr.h to include/net
2013-07-27 22:14:06 -07:00
vxlan: Inline vxlan_gso_check().
2014-11-18 15:38:44 -05:00
wext.h: Remove extern from function prototypes
2013-09-23 16:29:40 -04:00
net: treewide: Fix typo found in DocBook/networking.xml
2014-09-05 17:35:28 -07:00
ieee802154: add netlink APIs for smartMAC configuration
2014-02-17 16:42:39 -05:00
x25.h: Remove extern from function prototypes
2013-09-23 16:29:41 -04:00
xfrm: configure policy hash table thresholds by netlink
2014-09-02 13:37:56 +02:00
3e32cb2e0a