The change is for general scheduler improvement.
Change-Id: Ie54730178c19657e541cfba6ab84e075aaa799d7
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The change is for general scheduler improvement.
Change-Id: Ib5d58c23d3a67a81d682e08e58f308e5c659a7f8
Signed-off-by: Kishore Sri venkata Ganesh Bolisetty <bsrivenk@codeaurora.org>
Lets improve performance on builds which don't have WALT enabled.
Change-Id: I0ed4c99a6a36d0c0efcfc211ebf817b281502196
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
* refs/heads/tmp-1d9c43f:
ANDROID: ABI: Update allowed list for QCOM
ANDROID: build_config: drop CONFIG_KASAN_PANIC_ON_WARN
BACKPORT: bpf: add bpf_ktime_get_boot_ns()
ANDROID: ABI: update allowed list for QCOM
BACKPORT: clk: composite: Export clk_hw_register_composite()
ANDROID: PCI: dwc: Export symbols for ep driver
UPSTREAM: futex: Handle faults correctly for PI futexes
UPSTREAM: futex: Simplify fixup_pi_state_owner()
UPSTREAM: futex: Use pi_state_update_owner() in put_pi_state()
UPSTREAM: rtmutex: Remove unused argument from rt_mutex_proxy_unlock()
UPSTREAM: futex: Provide and use pi_state_update_owner()
UPSTREAM: futex: Replace pointless printk in fixup_owner()
UPSTREAM: futex: Ensure the correct return value from futex_lock_pi()
UPSTREAM: Revert "mm/slub: fix a memory leak in sysfs_slab_add()"
Change-Id: I87be53a117320a1855f7671ca07ed7602bef806b
Signed-off-by: Srinivasarao P <spathi@codeaurora.org>
On a device like a cellphone which is constantly suspending
and resuming CLOCK_MONOTONIC is not particularly useful for
keeping track of or reacting to external network events.
Instead you want to use CLOCK_BOOTTIME.
Hence add bpf_ktime_get_boot_ns() as a mirror of bpf_ktime_get_ns()
based around CLOCK_BOOTTIME instead of CLOCK_MONOTONIC.
Signed-off-by: Maciej Żenczykowski <maze@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
(cherry picked from commit 71d19214776e61b33da48f7c1b46e522c7f78221)
Change-Id: Ifd62c410dcc5112fd1a473a7e1f70231ca514bc0
The weak symbols are no longer needed as the WALT code is
moved from techpack to kernel.
Change-Id: I2918ac475d264482b039ee8b65357657c427eb28
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
This change is for general scheduler improvemnts.
Change-Id: If28d00691bc42b62cc4fc96a49d601c7ac5e3e34
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Load_balancer considers only cfs running tasks for finding busiest cpu
to do load balancing. But cpu may be busy with other type tasks (ex: RT),
then that cpu might not selected as busy cpu due to weight vs nr_run
checks fails). And possibley cfs tasks running on that cpu would suffer
till other type tasks finishes or weight checks passes, while other cpus
sitting idle and not able to do load balance.
So, consider all running tasks to check cpu busieness.
Change-Id: Iddf3f668507e20359f6388fc30ff5897d234c902
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
commit 34b1a1ce1458f50ef27c54e28eb9b1947012907a upstream
fixup_pi_state_owner() tries to ensure that the state of the rtmutex,
pi_state and the user space value related to the PI futex are consistent
before returning to user space. In case that the user space value update
faults and the fault cannot be resolved by faulting the page in via
fault_in_user_writeable() the function returns with -EFAULT and leaves
the rtmutex and pi_state owner state inconsistent.
A subsequent futex_unlock_pi() operates on the inconsistent pi_state and
releases the rtmutex despite not owning it which can corrupt the RB tree of
the rtmutex and cause a subsequent kernel stack use after free.
It was suggested to loop forever in fixup_pi_state_owner() if the fault
cannot be resolved, but that results in runaway tasks which is especially
undesired when the problem happens due to a programming error and not due
to malice.
As the user space value cannot be fixed up, the proper solution is to make
the rtmutex and the pi_state consistent so both have the same owner. This
leaves the user space value out of sync. Any subsequent operation on the
futex will fail because the 10th rule of PI futexes (pi_state owner and
user space value are consistent) has been violated.
As a consequence this removes the inept attempts of 'fixing' the situation
in case that the current task owns the rtmutex when returning with an
unresolvable fault by unlocking the rtmutex which left pi_state::owner and
rtmutex::owner out of sync in a different and only slightly less dangerous
way.
Fixes: 1b7558e457 ("futexes: fix fault handling in futex_lock_pi")
Reported-by: gzobqq@gmail.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 34b1a1ce1458f50ef27c54e28eb9b1947012907a)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Iabd44dad5d4b8385b8a20e44d18308708cc3b701
commit f2dac39d93987f7de1e20b3988c8685523247ae2 upstream
Too many gotos already and an upcoming fix would make it even more
unreadable.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit f2dac39d93987f7de1e20b3988c8685523247ae2)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I5a97587c0f696d4f4b849f7f106990a5587fa402
commit 6ccc84f917d33312eb2846bd7b567639f585ad6d upstream
No point in open coding it. This way it gains the extra sanity checks.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 6ccc84f917d33312eb2846bd7b567639f585ad6d)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Ic34513518cd4996b3eb4cb82b669d727be522597
commit 2156ac1934166d6deb6cd0f6ffc4c1076ec63697 upstream
Nothing uses the argument. Remove it as preparation to use
pi_state_update_owner().
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 2156ac1934166d6deb6cd0f6ffc4c1076ec63697)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I126088d93c5f05db654e508dfd4195e5c1696164
commit c5cade200ab9a2a3be9e7f32a752c8d86b502ec7 upstream
Updating pi_state::owner is done at several places with the same
code. Provide a function for it and use that at the obvious places.
This is also a preparation for a bug fix to avoid yet another copy of the
same code or alternatively introducing a completely unpenetratable mess of
gotos.
Originally-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit c5cade200ab9a2a3be9e7f32a752c8d86b502ec7)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I17a67806a035338a55b27eea673d707fc4d8c529
commit 04b79c55201f02ffd675e1231d731365e335c307 upstream
If that unexpected case of inconsistent arguments ever happens then the
futex state is left completely inconsistent and the printk is not really
helpful. Replace it with a warning and make the state consistent.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 04b79c55201f02ffd675e1231d731365e335c307)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I32bd64130d9414f65155ecd80646d01eb4374a60
commit 12bb3f7f1b03d5913b3f9d4236a488aa7774dfe9 upstream
In case that futex_lock_pi() was aborted by a signal or a timeout and the
task returned without acquiring the rtmutex, but is the designated owner of
the futex due to a concurrent futex_unlock_pi() fixup_owner() is invoked to
establish consistent state. In that case it invokes fixup_pi_state_owner()
which in turn tries to acquire the rtmutex again. If that succeeds then it
does not propagate this success to fixup_owner() and futex_lock_pi()
returns -EINTR or -ETIMEOUT despite having the futex locked.
Return success from fixup_pi_state_owner() in all cases where the current
task owns the rtmutex and therefore the futex and propagate it correctly
through fixup_owner(). Fixup the other callsite which does not expect a
positive return value.
Fixes: c1e2f0eaf0 ("futex: Avoid violating the 10th rule of futex")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 12bb3f7f1b03d5913b3f9d4236a488aa7774dfe9)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I6dfb7407620fdf8293edbfbe54463f0c8ff63a57
commit 1e106aa3509b86738769775969822ffc1ec21bf4 upstream.
The exit_pi_state_list() function calls put_pi_state() with IRQs disabled
and is not expecting that IRQs will be enabled inside the function.
Use the _irqsave() variant so that IRQs are restored to the original state
instead of being enabled unconditionally.
Fixes: 153fbd1226 ("futex: Fix more put_pi_state() vs. exit_pi_state_list() races")
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/20201106085205.GA1159983@mwanda
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 2192d905df)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I10bc011b54ffc6d66c0a499ecebfa5b2d96d58ce
commit 9f5d1c336a10c0d24e83e40b4c1b9539f7dba627 upstream.
Gratian managed to trigger the BUG_ON(!newowner) in fixup_pi_state_owner().
This is one possible chain of events leading to this:
Task Prio Operation
T1 120 lock(F)
T2 120 lock(F) -> blocks (top waiter)
T3 50 (RT) lock(F) -> boosts T1 and blocks (new top waiter)
XX timeout/ -> wakes T2
signal
T1 50 unlock(F) -> wakes T3 (rtmutex->owner == NULL, waiter bit is set)
T2 120 cleanup -> try_to_take_mutex() fails because T3 is the top waiter
and the lower priority T2 cannot steal the lock.
-> fixup_pi_state_owner() sees newowner == NULL -> BUG_ON()
The comment states that this is invalid and rt_mutex_real_owner() must
return a non NULL owner when the trylock failed, but in case of a queued
and woken up waiter rt_mutex_real_owner() == NULL is a valid transient
state. The higher priority waiter has simply not yet managed to take over
the rtmutex.
The BUG_ON() is therefore wrong and this is just another retry condition in
fixup_pi_state_owner().
Drop the locks, so that T3 can make progress, and then try the fixup again.
Gratian provided a great analysis, traces and a reproducer. The analysis is
to the point, but it confused the hell out of that tglx dude who had to
page in all the futex horrors again. Condensed version is above.
[ tglx: Wrote comment and changelog ]
Fixes: c1e2f0eaf0 ("futex: Avoid violating the 10th rule of futex")
Reported-by: Gratian Crisan <gratian.crisan@ni.com>
Signed-off-by: Mike Galbraith <efault@gmx.de>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/87a6w6x7bb.fsf@ni.com
Link: https://lore.kernel.org/r/87sg9pkvf7.fsf@nanos.tec.linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 2716e78a64)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I9ab0878641624448cacb1951506b5accb4c92a00
[ Upstream commit 921c7ebd1337d1a46783d7e15a850e12aed2eaa0 ]
If should_futex_fail() returns true in futex_wake_pi(), then the 'ret'
variable is set to -EFAULT and then immediately overwritten. So the failure
injection is non-functional.
Fix it by actually leaving the function and returning -EFAULT.
The Fixes tag is kinda blury because the initial commit which introduced
failure injection was already sloppy, but the below mentioned commit broke
it completely.
[ tglx: Massaged changelog ]
Fixes: 6b4f4bc9cb ("locking/futex: Allow low-level atomic operations to return -EAGAIN")
Signed-off-by: Mateusz Nosek <mateusznosek0@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20200927000858.24219-1-mateusznosek0@gmail.com
Signed-off-by: Sasha Levin <sashal@kernel.org>
(cherry picked from commit 2db7590371)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I22198b7a65630cb5649f3e889810783b147bfada
The scheduler techpack sources are moved to kernel to ease
the development.
The weak symbol definitions for WALT functions are no longer
required to cover the case of compiling kernel without syncing
the scheduler techpack. So remove all the weak symbol references.
Change-Id: Ief85bccd3dceaf60dda44aef9893b4138dc63380
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I288fc9a8858cb29ae9c6993b398a349b4a8c7475
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
We add a vendor hook for util to freq calculation in schedutil,
so we need to do corresponding change for energy calculation.
android_vh_em_pd_energy
adjust energy calculation
Bug: 178021064
Signed-off-by: Yun Hsiang <yun.hsiang@mediatek.com>
Change-Id: Iae772cf07881602eea3f27aeb75fba753e7c2635
Add sched_sync_hint_enable not to enable/disable sync.
Change-Id: Id6bdff3d4b9ea1051d7b55ca4cfd1d802c92fa64
Signed-off-by: Biao Long <blong@codeaurora.org>
Currently, the frequency is calculated by max freq * 1.25 * util / max cap.
Add a vendor hook to adjust the frequency when the calculation
overestimate.
android_vh_map_util_freq
adjust util to freq calculation
Bug: 177845438
Signed-off-by: Yun Hsiang <yun.hsiang@mediatek.com>
Change-Id: I9aa9079f00af7d3380b19f2fe21b75cddd107d15
Minidump ftrace need absolute timestamp of an event, and
not the relative timestamp from buffer start time. So,
do not use latency format for them.
Change-Id: I5ed1aaac62a16e312d8c1d2f3fcbfc2781c4af77
Signed-off-by: Neeraj Upadhyay <neeraju@codeaurora.org>
This symbol is needed for certain vendor modules which cannot rely on
only the sysfs interface for uclamp alone.
Bug: 170697030
Signed-off-by: J. Avila <elavila@google.com>
Change-Id: Ic904ade83a45d259cfc95501e6b81e6c5a0e90a0
(cherry picked from commit 0edac9cb95fb2fa292a4b25ac77b125cde28b674)
This change is for general scheduler improvemnts.
Change-Id: I8e6d7afe12387d86f3721c6fd880812ba86d885e
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: Id19f38a2b9a67b130876cb8df1e88c6a56aa6aad
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I30b3899e191d880e741f922323d0826b5c292498
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I616acbe5bfdb0e1bf88e923d98b4a3d54d26d942
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Record cycle count and kernel time in ns while entering suspend
and while resuming. This will help in matching up the kernel time
with external subsystems and in recording suspend time in cycles.
Change-Id: If22d9880103c7d54087d0faf4c992259e6b97f08
Signed-off-by: Srinivas Ramana <sramana@codeaurora.org>
Signed-off-by: Elliot Berman <eberman@codeaurora.org>
Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>
Print epoch values when the device is enter and exit out of suspend
and resume. These epoch values are useful to know how long the device
is in suspend state. These values can be used to synchronize various
subsystem timestamps and have an unique timestamp to correlate
between various subsystems.
Change-Id: Ibe04b914649db1f2ee0ae748f3950ca646323ef1
Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>
A number of kthread-related functions indirectly take task_struct->pi_lock
while holding worker->lock in the call chain like this:
spin_lock(&worker->lock)
kthread_insert_work
wake_up_process
try_to_wake_up
raw_spin_lock_irqsave(&p->pi_lock, flags)
This lock dependency exists whenever kthread_insert_work is called either
directly or indirectly via __kthread_queue_delayed_work in the following
functions:
kthread_queue_work
kthread_delayed_work_timer_fn
kthread_queue_delayed_work
kthread_flush_work
kthread_mod_delayed_work
This creates possibilities for circular dependencies like the one reported
at [1]. Break this lock dependency by moving task wakeup after worker->lock
has been released.
[1]: https://lore.kernel.org/lkml/CAJuCfpG4NkhpQvZjgXZ_3gm6Hf1QgN_eUOQ8iX9Cv1k9whLwSQ@mail.gmail.com
Reported-by: Ke Wang <ke.wang@unisoc.com>
Reported-by: Shakeel Butt <shakeelb@google.com>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Tested-by: Shakeel Butt <shakeelb@google.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Link: https://lkml.org/lkml/2020/5/4/1148
Cherry picked instead of commit 461daba06bdcb9c7a3f92b9bbd110e1f7d093ffc
as an alternative solution for the lockdep error that does not break KMI.
Bug: 174875976
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I2478847f66c85dca953846cd77955928d690c97c
This change is for general scheduler improvement.
Change-Id: I8f521dfd31ff1c8cc1f16805e4e0ac78228d9383
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
WALT requires both src and dst runqueue locks to be held
during migration. So a double_lock_balance() is added
in move_queued_task() before calling set_task_cpu(). However,
releasing the src rq lock, which is pinned is giving a
lockdep warning. Fix this by unpinning the lock before
calling double_lock_balance.
Change-Id: I64dec0701b3467185cf53f311ebb521c6a822e88
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
