android_kernel_motorola_sm6375/mm
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Yu Zhao 3dc679b544 BACKPORT: mm: multi-gen LRU: optimize multiple memcgs
When multiple memcgs are available, it is possible to use generations as a
frame of reference to make better choices and improve overall performance
under global memory pressure.  This patch adds a basic optimization to
select memcgs that can drop single-use unmapped clean pages first.  Doing
so reduces the chance of going into the aging path or swapping, which can
be costly.

A typical example that benefits from this optimization is a server running
mixed types of workloads, e.g., heavy anon workload in one memcg and heavy
buffered I/O workload in the other.

Though this optimization can be applied to both kswapd and direct reclaim,
it is only added to kswapd to keep the patchset manageable.  Later
improvements may cover the direct reclaim path.

While ensuring certain fairness to all eligible memcgs, proportional scans
of individual memcgs also require proper backoff to avoid overshooting
their aggregate reclaim target by too much.  Otherwise it can cause high
direct reclaim latency.  The conditions for backoff are:

1. At low priorities, for direct reclaim, if aging fairness or direct
   reclaim latency is at risk, i.e., aging one memcg multiple times or
   swapping after the target is met.
2. At high priorities, for global reclaim, if per-zone free pages are
   above respective watermarks.
Moto-CRs-Fixed:(CR)

Server benchmark results:
  Mixed workloads:
    fio (buffered I/O): +[19, 21]%
                IOPS         BW
      patch1-8: 1880k        7343MiB/s
      patch1-9: 2252k        8796MiB/s

    memcached (anon): +[119, 123]%
                Ops/sec      KB/sec
      patch1-8: 862768.65    33514.68
      patch1-9: 1911022.12   74234.54

  Mixed workloads:
    fio (buffered I/O): +[75, 77]%
                IOPS         BW
      5.19-rc1: 1279k        4996MiB/s
      patch1-9: 2252k        8796MiB/s

    memcached (anon): +[13, 15]%
                Ops/sec      KB/sec
      5.19-rc1: 1673524.04   65008.87
      patch1-9: 1911022.12   74234.54

  Configurations:
    (changes since patch 6)

    cat mixed.sh
    modprobe brd rd_nr=2 rd_size=56623104

    swapoff -a
    mkswap /dev/ram0
    swapon /dev/ram0

    mkfs.ext4 /dev/ram1
    mount -t ext4 /dev/ram1 /mnt

    memtier_benchmark -S /var/run/memcached/memcached.sock \
      -P memcache_binary -n allkeys --key-minimum=1 \
      --key-maximum=50000000 --key-pattern=P:P -c 1 -t 36 \
      --ratio 1:0 --pipeline 8 -d 2000

    fio -name=mglru --numjobs=36 --directory=/mnt --size=1408m \
      --buffered=1 --ioengine=io_uring --iodepth=128 \
      --iodepth_batch_submit=32 --iodepth_batch_complete=32 \
      --rw=randread --random_distribution=random --norandommap \
      --time_based --ramp_time=10m --runtime=90m --group_reporting &
    pid=$!

    sleep 200

    memtier_benchmark -S /var/run/memcached/memcached.sock \
      -P memcache_binary -n allkeys --key-minimum=1 \
      --key-maximum=50000000 --key-pattern=R:R -c 1 -t 36 \
      --ratio 0:1 --pipeline 8 --randomize --distinct-client-seed

    kill -INT $pid
    wait

Client benchmark results:
  no change (CONFIG_MEMCG=n)

Link: https://lkml.kernel.org/r/20220918080010.2920238-10-yuzhao@google.com
Change-Id: I590780e6381d577d800d4c4551a702047fc31cc7
Signed-off-by: Yu Zhao <yuzhao@google.com>
Acked-by: Brian Geffon <bgeffon@google.com>
Acked-by: Jan Alexander Steffens (heftig) <heftig@archlinux.org>
Acked-by: Oleksandr Natalenko <oleksandr@natalenko.name>
Acked-by: Steven Barrett <steven@liquorix.net>
Acked-by: Suleiman Souhlal <suleiman@google.com>
Tested-by: Daniel Byrne <djbyrne@mtu.edu>
Tested-by: Donald Carr <d@chaos-reins.com>
Tested-by: Holger Hoffstätte <holger@applied-asynchrony.com>
Tested-by: Konstantin Kharlamov <Hi-Angel@yandex.ru>
Tested-by: Shuang Zhai <szhai2@cs.rochester.edu>
Tested-by: Sofia Trinh <sofia.trinh@edi.works>
Tested-by: Vaibhav Jain <vaibhav@linux.ibm.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hillf Danton <hdanton@sina.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michael Larabel <Michael@MichaelLarabel.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: M(CR) Rapoport <rppt@kernel.org>
Cc: M(CR) Rapoport <rppt@linux.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Qi Zheng <zhengqi.arch@bytedance.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit f76c83378851f8e70f032848c4e61203f39480e4)
Bug: 261052246
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
Reviewed-on: https://gerrit.mot.com/2491802
SME-Granted: SME Approvals Granted
SLTApproved: Slta Waiver
Tested-by: Jira Key
Reviewed-by: Xiangpo Zhao <zhaoxp3@motorola.com>
Submit-Approved: Jira Key
2024-04-18 14:45:15 +08:00
..
kasan Merge android11-5.4.233+ (4716ccc) into msm-5.4 2023-05-30 11:16:10 +05:30
backing-dev.c mm: bdi: initialize bdi_min_ratio when bdi is unregistered 2021-12-14 14:49:00 +01:00
balloon_compaction.c
cleancache.c
cma.c Merge android11-5.4.219+ (0ce03d1) into msm-5.4 2023-01-27 00:57:29 +05:30
cma.h
cma_debug.c Revert "Revert "mm: cma: make writeable CMA debugfs optional"" 2020-12-17 15:22:06 +05:30
compaction.c BACKPORT: mm/swap.c: don't pass "enum lru_list" to del_page_from_lru_list() 2024-04-18 14:45:14 +08:00
debug.c
debug_page_ref.c
dmapool.c
early_ioremap.c
fadvise.c
failslab.c
filemap.c Merge android11-5.4.259+ (81334f2) into msm-5.4 2023-12-18 15:40:18 +05:30
frame_vector.c v4l2: don't fall back to follow_pfn() if pin_user_pages_fast() fails 2022-12-08 11:23:06 +01:00
frontswap.c treewide: Remove uninitialized_var() usage 2023-06-09 10:29:01 +02:00
gup.c Merge android11-5.4.233+ (4716ccc) into msm-5.4 2023-05-30 11:16:10 +05:30
gup_benchmark.c
highmem.c
hmm.c BACKPORT: mm/hmm: do not call hmm_vma_walk_hole() while holding a spinlock 2024-04-18 14:45:14 +08:00
huge_memory.c BACKPORT: mm: multi-gen LRU: groundwork 2024-04-18 14:45:14 +08:00
hugetlb.c Merge android11-5.4.233+ (4716ccc) into msm-5.4 2023-05-30 11:16:10 +05:30
hugetlb_cgroup.c
hwpoison-inject.c
init-mm.c
internal.h BACKPORT: mm: multi-gen LRU: exploit locality in rmap 2024-04-18 14:45:14 +08:00
interval_tree.c
Kconfig BACKPORT: mm: multi-gen LRU: minimal implementation 2024-04-18 14:45:14 +08:00
Kconfig.debug
khugepaged.c Merge android11-5.4.233+ (4716ccc) into msm-5.4 2023-05-30 11:16:10 +05:30
kmemleak-test.c
kmemleak.c Revert "mm: kmemleak: take a full lowmem check in kmemleak_*_phys()" 2022-09-15 12:04:49 +02:00
ksm.c Merge android11-5.4.249+ (d57e792) into msm-5.4 2023-09-28 16:45:28 +05:30
list_lru.c mm: list_lru: set shrinker map bit when child nr_items is not zero 2020-12-11 13:23:31 +01:00
maccess.c
madvise.c ANDROID: mm: remove sequence counting when mmap_lock is not exclusively owned 2023-02-16 18:56:04 -08:00
Makefile
memblock.c This is the 5.4.232 stable release 2023-02-22 12:30:48 +00:00
memcontrol.c BACKPORT: mm: multi-gen LRU: support page table walks 2024-04-18 14:45:15 +08:00
memfd.c memfd: fix F_SEAL_WRITE after shmem huge page allocated 2022-03-08 19:07:49 +01:00
memory-failure.c Merge android11-5.4.226+ (d72fdcc) into msm-5.4 2023-02-07 13:23:49 +05:30
memory.c BACKPORT: mm: multi-gen LRU: support page table walks 2024-04-18 14:45:15 +08:00
memory_hotplug.c Merge android11-5.4.161+ (b9d179c) into msm-5.4 2022-02-07 22:29:21 +05:30
mempolicy.c Merge android11-5.4.249+ (d57e792) into msm-5.4 2023-09-28 16:45:28 +05:30
mempool.c
memremap.c BACKPORT: mm: remove superfluous __ClearPageActive() 2024-04-18 14:45:14 +08:00
memtest.c
migrate.c Merge android11-5.4.219+ (0ce03d1) into msm-5.4 2023-01-27 00:57:29 +05:30
mincore.c
mlock.c BACKPORT: mm/swap.c: don't pass "enum lru_list" to del_page_from_lru_list() 2024-04-18 14:45:14 +08:00
mm_init.c BACKPORT: mm: multi-gen LRU: groundwork 2024-04-18 14:45:14 +08:00
mmap.c BACKPORT: FROMLIST: mm: protect free_pgtables with mmap_lock write lock in exit_mmap 2023-08-11 14:59:04 +05:30
mmu_context.c
mmu_gather.c mm/khugepaged: fix GUP-fast interaction by sending IPI 2022-12-14 11:30:42 +01:00
mmu_notifier.c
mmzone.c BACKPORT: mm: multi-gen LRU: groundwork 2024-04-18 14:45:14 +08:00
mprotect.c Merge android11-5.4.219+ (0ce03d1) into msm-5.4 2023-01-27 00:57:29 +05:30
mremap.c ANDROID: Re-enable fast mremap and fix UAF with SPF 2023-02-22 22:23:17 +05:30
msync.c
nommu.c
oom_kill.c Revert "oom_kill.c: futex: delay the OOM reaper to allow time for proper futex cleanup" 2022-04-27 15:56:55 +02:00
OWNERS ANDROID: Add OWNERS files referring to the respective android-mainline OWNERS 2021-04-01 13:45:14 +00:00
page-writeback.c mm: make wait_on_page_writeback() wait for multiple pending writebacks 2023-06-28 10:18:42 +02:00
page_alloc.c disable watermark boost 2024-04-18 14:45:01 +08:00
page_counter.c
page_ext.c mm: fix the page_owner initializing issue for arm32 2020-09-29 12:42:42 +08:00
page_idle.c BACKPORT: mm: don't be stuck to rmap lock on reclaim path 2023-01-13 08:56:32 +00:00
page_io.c mm: zram: use rw_page for zram page but submit_bio for zram wb page 2024-04-18 14:45:03 +08:00
page_isolation.c mm/memory_hotplug: drain per-cpu pages again during memory offline 2020-11-23 17:03:25 +05:30
page_owner.c Merge android11-5.4.86+ (75c93eb) into msm-5.4 2021-04-22 09:44:51 +05:30
page_poison.c
page_vma_mapped.c mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk() 2021-06-30 08:47:55 -04:00
pagewalk.c BACKPORT: mm: pagewalk: add p4d_entry() and pgd_entry() 2024-04-18 14:45:14 +08:00
percpu-internal.h
percpu-km.c
percpu-stats.c
percpu-vm.c
percpu.c treewide: Remove uninitialized_var() usage 2023-06-09 10:29:01 +02:00
pgtable-generic.c mm/thp: fix __split_huge_pmd_locked() on shmem migration entry 2021-06-30 08:47:52 -04:00
process_vm_access.c
readahead.c
rmap.c BACKPORT: mm: multi-gen LRU: exploit locality in rmap 2024-04-18 14:45:14 +08:00
rodata_test.c
shmem.c BACKPORT: mm/swapcache: support to handle the shadow entries 2024-04-18 14:45:14 +08:00
showmem.c
shuffle.c
shuffle.h
slab.c
slab.h mm: kmemleak: slob: respect SLAB_NOLEAKTRACE flag 2021-11-26 10:47:21 +01:00
slab_common.c Merge android11-5.4.147+ (dc8c919) into msm-5.4 2021-11-05 17:27:33 +05:30
slob.c UPSTREAM: mm/sl[uo]b: export __kmalloc_track(_node)_caller 2020-11-02 16:12:00 +00:00
slub.c mm: slub: Declare slab_owner_ops only when SLUB DEBUG is enabled 2024-01-18 01:32:07 -08:00
sparse-vmemmap.c
sparse.c mm/sparse: add the missing sparse_buffer_fini() in error branch 2021-05-14 09:44:32 +02:00
swap.c BACKPORT: mm: multi-gen LRU: minimal implementation 2024-04-18 14:45:14 +08:00
swap_cgroup.c
swap_slots.c
swap_state.c BACKPORT: mm: multi-gen LRU: minimal implementation 2024-04-18 14:45:14 +08:00
swapfile.c BACKPORT: mm/swapcache: support to handle the shadow entries 2024-04-18 14:45:14 +08:00
truncate.c mm/thp: unmap_mapping_page() to fix THP truncate_cleanup_page() 2021-06-30 08:47:53 -04:00
usercopy.c Merge android11-5.4.191+ (375c2e2) into msm-5.4 2022-06-09 11:43:23 +05:30
userfaultfd.c mm: userfaultfd: fix missing cache flush in mcopy_atomic_pte() and __mcopy_atomic() 2022-05-15 19:54:47 +02:00
util.c random: move randomize_page() into mm where it belongs 2022-06-22 14:11:17 +02:00
vmacache.c
vmalloc.c mm: vmalloc: prevent use after free in _vm_unmap_aliases 2021-07-29 20:47:17 -07:00
vmpressure.c
vmscan.c BACKPORT: mm: multi-gen LRU: optimize multiple memcgs 2024-04-18 14:45:15 +08:00
vmstat.c Merge android11-5.4.197+ (3970bc6) into msm-5.4 2022-08-16 16:52:01 +05:30
workingset.c BACKPORT: mm: multi-gen LRU: minimal implementation 2024-04-18 14:45:14 +08:00
z3fold.c mm/z3fold: fix potential memory leak in z3fold_destroy_pool() 2021-07-14 16:53:47 +02:00
zbud.c
zpool.c
zsmalloc.c Merge android11-5.4.197+ (3970bc6) into msm-5.4 2022-08-16 16:52:01 +05:30
zswap.c