When handling speculative page fault, the vma->vm_flags and
vma->vm_page_prot fields are read once the page table lock is released. So
there is no more guarantee that these fields would not change in our back.
They will be saved in the vm_fault structure before the VMA is checked for
changes.
This patch also set the fields in hugetlb_no_page() and
__collapse_huge_page_swapin even if it is not need for the callee.
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Change-Id: I9821f02ea32ef220b57b8bfd817992bbf71bbb1d
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:18
[vinmenon@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
[charante@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
The speculative page fault handler must be protected against anon_vma
changes. This is because page_add_new_anon_rmap() is called during the
speculative path.
In addition, don't try speculative page fault if the VMA don't have an
anon_vma structure allocated because its allocation should be
protected by the mmap_sem.
In __vma_adjust() when importer->anon_vma is set, there is no need to
protect against speculative page faults since speculative page fault
is aborted if the vma->anon_vma is not set.
When calling page_add_new_anon_rmap() vma->anon_vma is necessarily
valid since we checked for it when locking the pte and the anon_vma is
removed once the pte is unlocked. So even if the speculative page
fault handler is running concurrently with do_unmap(), as the pte is
locked in unmap_region() - through unmap_vmas() - and the anon_vma
unlinked later, because we check for the vma sequence counter which is
updated in unmap_page_range() before locking the pte, and then in
free_pgtables() so when locking the pte the change will be detected.
Change-Id: I6c1f3b5c811d1ddd7b3f769082e8bbd40f5b52a0
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:17
[vinmenon@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
If a thread is remapping an area while another one is faulting on the
destination area, the SPF handler may fetch the vma from the RB tree before
the pte has been moved by the other thread. This means that the moved ptes
will overwrite those create by the page fault handler leading to page
leaked.
CPU 1 CPU2
enter mremap()
unmap the dest area
copy_vma() Enter speculative page fault handler
>> at this time the dest area is present in the RB tree
fetch the vma matching dest area
create a pte as the VMA matched
Exit the SPF handler
<data written in the new page>
move_ptes()
> it is assumed that the dest area is empty,
> the move ptes overwrite the page mapped by the CPU2.
To prevent that, when the VMA matching the dest area is extended or created
by copy_vma(), it should be marked as non available to the SPF handler.
The usual way to so is to rely on vm_write_begin()/end().
This is already in __vma_adjust() called by copy_vma() (through
vma_merge()). But __vma_adjust() is calling vm_write_end() before returning
which create a window for another thread.
This patch adds a new parameter to vma_merge() which is passed down to
vma_adjust().
The assumption is that copy_vma() is returning a vma which should be
released by calling vm_raw_write_end() by the callee once the ptes have
been moved.
Change-Id: Icd338ad6e9b3c97b7334d3b8d30a8badfa2a4efa
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:16
[vinmenon@codeaurora.org: changes in vma_merge arguments related
to the anon vma user name which is not suppported upstream.]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
The VMA sequence count has been introduced to allow fast detection of
VMA modification when running a page fault handler without holding
the mmap_sem.
This patch provides protection against the VMA modification done in :
- madvise()
- mpol_rebind_policy()
- vma_replace_policy()
- change_prot_numa()
- mlock(), munlock()
- mprotect()
- mmap_region()
- collapse_huge_page()
- userfaultd registering services
In addition, VMA fields which will be read during the speculative fault
path needs to be written using WRITE_ONCE to prevent write to be split
and intermediate values to be pushed to other CPUs.
Change-Id: Ic36046b7254e538b6baf7144c50ae577ee7f2074
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:15
[vinmenon@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
[charante@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
Wrap the VMA modifications (vma_adjust/unmap_page_range) with sequence
counts such that we can easily test if a VMA is changed.
The unmap_page_range() one allows us to make assumptions about
page-tables; when we find the seqcount hasn't changed we can assume
page-tables are still valid.
The flip side is that we cannot distinguish between a vma_adjust() and
the unmap_page_range() -- where with the former we could have
re-checked the vma bounds against the address.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[Port to 4.12 kernel]
[Build depends on CONFIG_SPECULATIVE_PAGE_FAULT]
[Introduce vm_write_* inline function depending on
CONFIG_SPECULATIVE_PAGE_FAULT]
[Fix lock dependency between mapping->i_mmap_rwsem and vma->vm_sequence by
using vm_raw_write* functions]
[Fix a lock dependency warning in mmap_region() when entering the error
path]
[move sequence initialisation INIT_VMA()]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Change-Id: Ibc23ef3b9dbb80323c0f24cb06da34b4c3a8fa71
Patch-mainline: linux-mm @ 17 Apr 2018 16:33:14
[vinmenon@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
[charante@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
Some VMA struct fields need to be initialized once the VMA structure is
allocated.
Currently this only concerns anon_vma_chain field but some other will be
added to support the speculative page fault.
Instead of spreading the initialization calls all over the code, let's
introduce a dedicated inline function.
Change-Id: I9f6b29dc74055354318b548e2b6b22c37d4c61bb
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:13
[vinmenon@codeaurora.org: trivial merge conflict fixes]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
[charante@codeaurora.org: merge conflict fixes]
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
pte_unmap_same() is making the assumption that the page table are still
around because the mmap_sem is held.
This is no more the case when running a speculative page fault and
additional check must be made to ensure that the final page table are still
there.
This is now done by calling pte_spinlock() to check for the VMA's
consistency while locking for the page tables.
This is requiring passing a vm_fault structure to pte_unmap_same() which is
containing all the needed parameters.
As pte_spinlock() may fail in the case of a speculative page fault, if the
VMA has been touched in our back, pte_unmap_same() should now return 3
cases :
1. pte are the same (0)
2. pte are different (VM_FAULT_PTNOTSAME)
3. a VMA's changes has been detected (VM_FAULT_RETRY)
The case 2 is handled by the introduction of a new VM_FAULT flag named
VM_FAULT_PTNOTSAME which is then trapped in cow_user_page().
If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the
page fault while holding the mmap_sem.
Change-Id: Iaccfa0d877334f4343f8b0ec3400af5070ff5864
Acked-by: David Rientjes <rientjes@google.com>
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:12
[vinmenon@codeaurora.org: trivial merge conflicts]
[vinmenon@codeaurora.org: 5.4: moved PTNOTSAME to mm_types.h]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
When handling page fault without holding the mmap_sem the fetch of the
pte lock pointer and the locking will have to be done while ensuring
that the VMA is not touched in our back.
So move the fetch and locking operations in a dedicated function.
Change-Id: If93ab95b1d22b7195e1c15b57315021f6be7c394
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:11
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
When speculating faults (without holding mmap_sem) we need to validate
that the vma against which we loaded pages is still valid when we're
ready to install the new PTE.
Therefore, replace the pte_offset_map_lock() calls that (re)take the
PTL with pte_map_lock() which can fail in case we find the VMA changed
since we started the fault.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[Port to 4.12 kernel]
[Remove the comment about the fault_env structure which has been
implemented as the vm_fault structure in the kernel]
[move pte_map_lock()'s definition upper in the file]
[move the define of FAULT_FLAG_SPECULATIVE later in the series]
[review error path in do_swap_page(), do_anonymous_page() and
wp_page_copy()]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Change-Id: Id6dfae130fbfdd4bb92aa6415d6f1db7ef833266
[vinmenon@codeaurora.org: fix trivial merge conflicts]
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:10
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
This configuration variable will be used to build the code needed to
handle speculative page fault.
By default it is turned off, and activated depending on architecture
support, SMP and MMU.
Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Suggested-by: David Rientjes <rientjes@google.com>
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Change-Id: I17123124ec0667a0f0af741a740e5219d278620c
Patch-mainline: linux-mm @ Tue, 17 Apr 2018 16:33:07
[vinmenon@codeaurora.org: disable the feature by default]
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
We need to relocate selinux_state to a separate 4k page
to enable EL2 Hypervisor to monitor changes to this
variable using ARM stage 2 MMU. We will avoid getting
page faults from un-related data as the MMU granularity
is configured to 4k.
Change-Id: I7d0e5c9eae0a2a65ca9db73e85283e164e20c11c
Signed-off-by: Preeti Nagar <pnagar@codeaurora.org>
Add snapshot of qpnp-smb5 charger driver from msm-4.19
as of commit 1bb7bbeb1d09("sched: Fix compilation errors
with !WALT").
Change-Id: Ie7c99bea8887dcb97244e43e7984e2a82d1e5526
Signed-off-by: Jishnu Prakash <jprakash@codeaurora.org>
Check whether size of fd is greater than or equal to
length passed by client. If its not checked here, client
might access unmapped address on dsp.
Change-Id: I4e176f32b0f263e2f7bf648e03b4ca904479527b
Acked-by: Deepika Singh <dsi@qti.qualcomm.com>
Signed-off-by: Mohammed Nayeem Ur Rahman <mohara@codeaurora.org>
Two new fields in the struct vm_fault can't easily be excluded using
CONFIG_SPECULATIVE_PAGE_FAULTS. Add them as padding to reduce the
ABI diff between vendors which use SPECULATIVE_PAGE_FAULTS and ACK. This
allows vendors to pick the SPECULATIVE_PAGE_FAULTS feature and still be
ABI compatible with ACK.
Bug: 153715905
Test: build
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
(cherry picked from commit 47e3eb155848872a13ab6ef73a4d0eecac36ce66)
[surenb: kept only struct vm_fault changes and dropped the rest]
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I08dbe38f37fa3e8a0f96f71cb6b2dd8737221da5
Git-commit: a1db93d3a956eae02646e3269f5bb6f3c6fdc7c9
Git-Repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Signed-off-by: Mukesh Ojha <mojha@codeaurora.org>
Add the clock resource ids required to vote from the smd rpm clock
driver.
Change-Id: Ic224d83ea2aacabd26450498b626ebfc687aad04
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Add missing base reference for GPUs to make them consistent
across the list.
Change-Id: Iaaf9ce83fc21d18f5aea573cabe4c99b1454fc76
Signed-off-by: Rajesh Kemisetti <rajeshk@codeaurora.org>
Signed-off-by: Deepak Kumar <dkumar@codeaurora.org>
Update the QCOM whitelist file with additional symbols and
regenerate the snapshot accordingly.
Change-Id: I09c9c91d9d0d1c5bafcb54bd4893f7554c4b8e49
Signed-off-by: Raghavendra Rao Ananta <rananta@codeaurora.org>
As of commit <a5f1397e4b> ("Revert "BACKPORT: tracing: Remove
unnecessary DEBUG_FS dependency""), DEBUG_FS was selected by TRACING.
Remove DEBUG_FS dependency in Tracing kconfig. CONFIG_DEBUG_FS will now
be enabled in QGKI config. Also update the whitelist and abi snapshot.
Change-Id: I8c9e9fb6b97173d6e62f6090712642f18d842888
Signed-off-by: Prakruthi Deepak Heragu <pheragu@codeaurora.org>
Prevent unclocked access if this debugfs parameter is
attempted to be read when USB controller is powered off.
Change-Id: I5a563ea75807d95f0d6d6c5b037db48681d89625
Signed-off-by: Jack Pham <jackp@codeaurora.org>
Ensure that valid memory is allocated for the array of all votables
before an attempt is made to populate it.
Change-Id: I9a0c3e35e345a88560e39d47484348b6c476628d
Signed-off-by: Guru Das Srinagesh <gurus@codeaurora.org>
The allocation code hot-adds the memory from another VM through
the memory hotplug code to create the S1 CPU MMU mappings. The memory
hotplug code can only add memory at a subsection granule, meaning
that all memory requests must be subsection size aligned. Thus,
enforce all allocation requests to be subsection size aligned.
Change-Id: Ia4d3294e36265f0bd956fa42670f2ac8d6974908
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
AMOLED ECM (Embedded Current Measurement) driver helps measure
the display current for OLED panels. Enable it.
Change-Id: I94d729dbfa8b46792afd424333a0bf9c106ffef1
Signed-off-by: Subbaraman Narayanamurthy <subbaram@codeaurora.org>
Client drivers may lend IRQs with knowledge of only their GPIO interrupt
number, which would not directly have an underlying GIC hwirq. Thus,
tweak RM's understanding of IRQs to be aware of IRQ domains. Now, the
irq backed by GPIO will be translated to GIC domain.
Change-Id: I191d9e072b14f4501f5bbeb5d5263f50e0eef8c1
Signed-off-by: Elliot Berman <eberman@codeaurora.org>
If a biased cpu entered shallowest LPM state and
there are no wakeups for it, can stay in the shallowest
state for long.
Program wakeup for the biased CPU to wakeup after the
expected bias window is completed, so that the cpu can
enter a deeper state.
Change-Id: Ic92c779f0f8b1fa85aa8b3afa68d075f8d5d7dd6
Signed-off-by: Srinivas Rao L <lsrao@codeaurora.org>
