The current implementation of overutilization, aborts energy aware
scheduling if any cpu in the system is over-utilized. This patch introduces
over utilization flag per sched domain level instead of a single flag
system wide. Load balancing is done at the sched domain where any
of the cpu is over utilized. If energy aware scheduling is
enabled and no cpu in a sched domain is overuttilized,
load balancing is skipped for that sched domain and energy aware
scheduling continues at that level.
The implementation takes advantage of the shared sched_domain structure
that is common across all the sched domains at a level. The new flag
introduced is placed in this structure so that all the sched domains the
same level share the flag. In case of an overutilized cpu, the flag gets
set at level1 sched_domain. The flag at the parent sched_domain level gets
set in either of the two following scenarios.
1. There is a misfit task in one of the cpu's in this sched_domain.
2. The total utilization of the domain is greater than the domain capacity
The flag is cleared if no cpu in a sched domain is overutilized.
This implementation still can have corner scenarios with respect to
misfit tasks. For example consider a sched group with n cpus and
n+1 70%utilized tasks. Ideally this is a case for load balance to happen
in a parent sched domain. But neither the total group utilization is
high enough for the load balance to be triggered in the parent domain nor
there is a cpu with a single overutilized task so that aload balance is
triggered in a parent domain. But again this could be a purely academic
sceanrio, as during task wake up these tasks will be placed
more appropriately.
Signed-off-by: Thara Gopinath <thara.gopinath@linaro.org>
Change-Id: I3f327cff4080096a3e58208dd72c9b7f7913cdb2
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Git-commit: addef37808728c719d8c095a75bcf81befdacdaf
Git-repo: https://android.googlesource.com/kernel/common/
[clingutla@codeaurora.org: Resolved trivial merge conflicts]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[satyap@codeaurora.org: port to 5.x and resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 5debecbe7195
("trace: filter out spurious preemption and IRQs disable traces").
Change-Id: I3c9663da1fd89e9e942831fda00a47b4a29ea4e3
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Watchdog gets disabled from other CPUs when isolation is enabled.
Using this_cpu_ptr() would lead to undesired issues, so, switch to
per_cpu_ptr(). While at it, remove warning as it makes no sense
when core isolation is enabled.
Change-Id: Id1a59c4c88044c7570457fb55abb2dcc13de3c11
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Ksfotirqd is a normal priority CFS task. It can experience higher
scheduling latency under heavy load conditions. Currently once
asynchronous softirq processing is deferred to ksoftirqd, softirqs
are not processed further until ksoftirqd task gets a chance to run.
High latencies for softirqs like TIMER, HI TASKLET is not acceptable.
So revert 'commit 4cd13c21b2 ("softirq: Let ksoftirqd do its job")'.
Change-Id: I38a1a88b5f42dd534c65d739dbb7e4321a7904db
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[satyap@codeaurora.org: Fix trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This reverts commit 3c53776e29.
Reverting the change to match what we have on previous Kernels.
Change-Id: I4af64cd7e2c4291dda5f503bf2d74ede459a76c6
[satyap@codeaurora.org: Port to 5.x and fix trivial merge conflict]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
An IRQ affinity is broken during hotplug/isolation when there
are no online and un-isolated CPUs in the current affinity mask.
An online and un-isolated CPU from the irq_default_affinity mask
(i.e /proc/irq/default_smp_affinity) is used as the current
affinity. However Individual IRQs can have their affinity hint
set via irq_set_affinity_hint() API. When such hint is available,
use it instead of the irq_default_affinity which is a system level
setting.
Change-Id: I53a537582ec4e1aed0c59b49f4fd5b6ca7c0c332
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Since 'commit f332a9d53e33 ("softirq: defer softirq processing to
ksoftirqd if CPU is busy with RT")', all softirqs are deferred
to ksoftirqd if one of the potentially long softirqs are pending.
This can significantly delay processing of tasklets and timers,
which are frequently designed to be running at high priority.
Defer only the potentially slow softirqs to ksoftirqd.
Change-Id: I7a2ef7c59749ccf086066c59962ff326786da6f4
Signed-off-by: Björn Davidsson <bjorn.davidsson@sony.com>
[clingutla@codeaurora.org: Renamed local variables and refactored
deferred logic to macro for avoiding potential deadlock.]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
The cpupri_find() returns the candidate CPUs which are running
lower priority than the waking RT task in the lowest_mask. This
contains isolated CPUs as well. Since the energy aware CPU selection
skips isolated CPUs, no target CPU may be found if all unisolated CPUs
are running higher priority RT tasks. In which case, we fallback to
the default CPU selection algorithm and returns an isolated CPU. This
decision is reversed by select_task_rq() and returns an unisolated
CPU that is busy with other RT tasks. This RT task packing is desired
behavior. However, RT push mechanism pushes the packed RT task to
an isolated CPU. This can be avoided by excluding isolated CPUs from
the lowest_mask returned by cpupri_find().
Change-Id: I75486b3935caf496a638d0333565beffc47fe249
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: trivial merge conflict resolution]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Defer the softirq processing to ksoftirqd if a RT task is running
or queued on the current CPU. This complements the RT task placement
algorithm which tries to find a CPU that is not currently busy with
softirqs.
Currently NET_TX, NET_RX, BLOCK and TASKLET softirqs are only deferred
as they can potentially run for long time.
Change-Id: Id7665244af6bbd5a96d9e591cf26154e9eaa860c
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: trivial merge conflict resolution.]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
The scheduling change (bug 31501544) to avoid putting RT threads on cores that
are handling softint's was catching cases where there was no reason
to believe the softint would take a long time, resulting in unnecessary
migration overhead. This patch reduces the migration to cases where
the core has a softint that is actually likely to take a long time,
as opposed to the RCU, SCHED, and TIMER softints that are rather quick.
Bug: 31752786
Change-Id: Ib4e179f1e15c736b2fdba31070494e357e9fbbe2
Git-commit: ce05770bd37b8065b61ef650108ecef2b97b148b
Git-repo: https://android.googlesource.com/kernel/msm
[pkondeti@codeaurora.org: resolved minor merge conflicts]
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 8a6c9078311bc31
("sched: rt: Fix start cpu for RT energy aware task placement").
Change-Id: I850161a6fa845c9a73c67072299476eb3c3d1888
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 89986132507b822
("sched: fair: Improve the scheduler").
Change-Id: I75555a04020478e8d589b6a0fba209501c360dad
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 22d004c08a27
("sched/fair: Improve the scheduler").
Change-Id: I8fc95a4a4650de0dc36bd979d374b9335f6af774
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of
commit 72395648aa0ad18 ("sched: core: Fix usage of cpu core group mask").
Change-Id: I8dc6933a1e3a0835f19c7802930f974d442082a8
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 89986132507b822
("sched: fair: Improve the scheduler").
Change-Id: I9a52e67862ee5086880884128102aa4f8efb04ce
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 5debecbe7195
("trace: filter out spurious preemption and IRQs disable traces").
Change-Id: I222aa448ac68f7365065f62dba9db94925da38a0
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
To debug issues related to task scheduling sched_enq_deq_task
trace point is very useful.
Change-Id: I668702701d5443a4f5b0356634622f071297eb28
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 5debecbe7195
("trace: filter out spurious preemption and IRQs disable traces").
Change-Id: I8fab4084971baadcaa037f40ab549fc073a4b1ea
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 72395648aa0ad18
("sched: core: Fix usage of cpu core group mask").
Change-Id: Icfbe12f59502c115b010ab53c110f232173c89a5
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
The idle-state of each cpu is currently pointed to by rq->idle_state but
there isn't any information in the struct cpuidle_state that can used to
look up the idle-state energy model data stored in struct
sched_group_energy. For this purpose is necessary to store the idle
state index as well. Ideally, the idle-state data should be unified.
cc: Ingo Molnar <mingo@redhat.com>
cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Morten Rasmussen <morten.rasmussen@arm.com>
Change-Id: Ib3d1178512735b0e314881f73fb8ccff5a69319f
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
(cherry picked from commit a732c97420e109956c20f34c70b91e6d06f5df31)
[ Fixed trivial cherry-pick conflict in kernel/sched/sched.h ]
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
[satyap@codeaurora.org: port to msm-5.x from msm-4.19 and add
WALT check around idle_state_idx usage]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of 'commit 5debecbe7195
("trace: filter out spurious preemption and IRQs disable traces")'.
Change-Id: I5b7bcbbc1da5dffb89932dfda392029159d17859
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Sleeping for a second adds unnecessary latency to hotplugging a cpu
(cpu_up). Reduce the wait time down to 1 ms which is in line with
earlier kernel versions. The 1 second value was caused due to an
incorrect merge conflict resolution during kernel upgrade.
Change-Id: Ice58147fff4d9b997def823436510ce23b02eb8e
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[satyap@codeaurora.org: trivial merge conflict resolution]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Currently EAS gets disabled when the system has only symmetric CPUs.
For example, either the system is SMP or all big or little CPUs are
hotplugged out on a b.L system. EAS features are required for better
power and performance on systems with symmetric CPUs. So don't disable
EAS on symmetric systems. This is done by assigning sd_asym_cpucapacity
per-cpu pointer to the only available sched domain.
Change-Id: If08a162f652a6a73884481ebc60e940373dbf9e2
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
As freq increases freq/power should decrease, indicating nonlinearly
higher power is required for running at higher frequencies.
If freq/power increases, the cost associated with that
freq will be lower than its previous one, causing the energy
calculations to choose a cpu with that frequency over a cpu with
the previous lesser freq.
For some frequencies, if their voltage increment from the
previous ones is very small (or is the same), we could end up with
higher freq/power ratio. This is primarily because
of_dev_pm_opp_get_cpu_power() returns power in mW and looses precision.
But instead of addressing it, enforce the same cost as the previous
frequency.
The energy evaluation code prefers the previous cpu when the energy costs
are same as other candidates. By keeping the energy costs same in these
situations we are increasing the likely hood of selecting prev cpu even
if it results in a slight freq bump. In general, selecting prev cpu is
beneficial because it avoids warming up the caches at a different cpu.
Change-Id: Ic66ab18ba65f2917b73d9fbe92a39b743b9839a0
Signed-off-by: Abhijeet Dharmapurikar <adharmap@codeaurora.org>
On systems with asymmetric cpu capacities, a skewed load distribution
might yield better throughput than balancing load per group capacity.
For example, running compute intensive tasks on high capacity cpus while
leaving low capacity cpus idle. So we let load-balance back off if the
busiest group isn't really overloaded.
cc: Ingo Molnar <mingo@redhat.com>
cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Morten Rasmussen <morten.rasmussen@arm.com>
Change-Id: I8b08a0fa73f357a9972324bc76cec3912fe293cf
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Git-commit: c42f9795e6a0c0e9d3cc6e1fe02fa60829ceb23a
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
[clingutla@codeaurora.org: Took partial commit '5494e2edf59c2 ("ANDROID:
sched: Consider misfit tasks when load-balancing")', which defined
function group_similar_cpu_capacity.]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
This reverts commit 9659e1ee. The change leaves a possibility of
scheduling dl task on offline cpu. The original commit text suggests
that it should not be a problem since migrate_tasks() will take care of
it, but there can be delay between marking cpu as offline and
migrate_tasks() call since they are called on different hotplug
notifications-offline happens on CPUHP_TEARDOWN_CPU while
migrate_tasks() gets called on CPUHP_AP_SCHED_STARTING. And in
between this duration there is a chance the dl task can get scheduled on
this offline cpu and lead to BUG_ON. Hence revert the change.
Change-Id: I6f4ea10d2ce4e83e12d36b9b90b1b630ffe4fc74
[pujag@codeaurora.org: fix minor conflicts due to reorg of the code]
Signed-off-by: Puja Gupta <pujag@codeaurora.org>
[satyap@codeaurora.org: resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
As deadline tasks doesn't have priority, their priority is set
to -1. If there are any deadline tasks in the system, chrome/tracing
doesn't show results as expected as it can't recognize tasks with
priority -1. Fix it by using 150 as priority for deadline tasks.
Change-Id: I49b43f77b97ebfe6d79a7c9e9ec346ea60e79768
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
When cpusets are enabled, the rebuilding of the scheduling domains
is deferred to a workqueue context. Make sure that the work is
completed before proceeding to the next hotplug. Otherwise scheduler
observes an inconsistent view of online and offline CPUs in the
root domain. If the online CPUs are still stuck in the offline i.e
default domain, those CPUs would not be visible when scheduling
happens on from other CPUs in the root domain. This results in
packing tasks more on certain CPUs while leaving the other CPUs
idle. Also RT_RUNTIME_SHARE feature i.e borrowing RT runtime from
the other CPUs would not work correctly. Because the CPUs in the
root domain can't borrow the runtime from the online CPUs but stuck
in the offline/default domain.
Change-Id: I93c51d2abc8a9c099b62f1d2cbb9095d3a788bda
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: fix trivial merge conflict]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
While the feature TTWU_QUEUE has the advantage of reducing cache
bouncing of runqueue locks, it has the side effect that runqueue
statistics are not updated until the remote CPU has a chance to
enqueue the task. Since there is no upper bound on the amount of
time it can take the remote CPU to enqueue the task, several
sequential wakeups can result in suboptimal task placement based
on the stale statistics. Turn off the feature as the cost of
sub-optimal placement is much higher than the cost of cache bouncing
spinlocks for msm based systems.
Change-Id: I0b85c0225237b2bc44f54934769f5e3750c0f3d6
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[satyap@codeaurora.org: fix trivial merge conflict]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Introduce PANIC_ON_SCHED_BUG to catch issues related to scheduler
like calling sleepable functions inside atomic context etc.
Change-Id: I9f1fe96772b317b17bd0e7423a48023d3cf141c7
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Add function to migrate timer that will be used by later patch set.
Change-Id: I2d4fbe312fe635273c7386aba8378937f4b0ac15
Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
[ohaugan@codeaurora.org: Updated commit text and fixed trivial merge conflict]
Git-commit: 3633b88d8fcb4273807574c27c328b6908a741e5
Git-repo: git://git.linaro.org/people/mike.holmes/santosh.shukla/lng-isol.git
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
[rameezmustafa@codeaurora.org: Port to msm-4.9]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
An hrtimer may be pinned to a CPU but inactive, so it is no longer valid
to test the hrtimer.state struct member as having no bits set when inactive.
Changed the test function to mask out the HRTIMER_STATE_PINNED bit when
checking for inactive state.
Change-Id: Icb4156f4b7cb10138ea6c5cd5b762e35dc25428d
Signed-off-by: Gary S. Robertson <gary.robertson@linaro.org>
[ohaugan@codeaurora.org: Port to 4.4]
Git-commit: 902e4d4eb0d2158d2792166221a72a829caecf07
Git-repo: git://git.linaro.org/people/mike.holmes/santosh.shukla/lng-isol.git
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
[satyap@codeaurora.org: Port to 4.19 and fix merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
To isolate CPUs (isolate from hrtimers) from sysfs using cpusets, we need some
support from the hrtimer core. i.e. A routine hrtimer_quiesce_cpu() which would
migrate away all the unpinned hrtimers, but shouldn't touch the pinned ones.
This patch creates this routine.
Change-Id: Icf93622bdeed9c6c28eb1aca1637bc5b9522a533
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
[forward port to 3.18]
Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
[ohaugan@codeaurora.org: Port to 4.4]
Git-commit: d4d50a0ddc35e58ee95137ba4d14e74fea8b682f
Git-repo: git://git.linaro.org/people/mike.holmes/santosh.shukla/lng-isol.git
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
[rameezmustafa@codeaurora.org: Port to msm-4.9]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[satyap@codeaurora.org: Port to msm-4.19 and fix merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
'Pinned' information would be required in migrate_hrtimers() now, as we can
migrate non-pinned timers away without a hotplug (i.e. with cpuset.quiesce). And
so we may need to identify pinned timers now, as we can't migrate them.
This patch reuses the timer->state variable for setting this flag as there were
enough number of free bits available in this variable. And there is no point
increasing size of this struct by adding another field.
Change-Id: Id2cdd74903598e31fab6999c4cbab9c2fd17fe23
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
[forward port to 3.18]
Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
[ohaugan@codeaurora.org: Port to 4.4]
Git-commit: 62feaf1ed0b64c04868d143d8bdb92d60dc3189b
Git-repo: git://git.linaro.org/people/mike.holmes/santosh.shukla/lng-isol.git
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
[markivx: Port to 4.14, fix merge conflicts due to lack of timer stats code]
Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>
[satyap@codeaurora.org: Port to 4.19, fix merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
To isolate CPUs (isolate from timers) from sysfs using cpusets, we need some
support from the timer core. i.e. A routine timer_quiesce_cpu() which would
migrates away all the unpinned timers, but shouldn't touch the pinned ones.
This patch creates this routine.
Change-Id: I4795aeffd1be41830ac295a1e2e8f36fb05e98b7
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
[forward port to 3.18]
Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
[ohaugan@codeaurora.org: Port to 4.4. Fixes for compilation error]
Git-commit: 313910b70ea0c73f8789d9189c11e1f339080646
Git-repo: git://git.linaro.org/people/mike.holmes/santosh.shukla/lng-isol.git
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
[rameezmustafa@codeaurora.org: Port to msm-4.9 and rebase to change
patch dependency order.]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[markivx: Port to 4.14, fix merge conflicts]
Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>
[satyap@codeaurora.org: Port to 4.19 and resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Enable compilation of the fastrpc driver on Lahaina, which
provides a communication mechanism that allows clients to
make remote method invocations across processor boundary
to applications, compute and sensors DSP processors.
Change-Id: I2a43104e260b3e129e3447840f3180843c10ce3a
Signed-off-by: Edgar Flores <edgarf@codeaurora.org>
Guard the code against ABI breakage using IS_ENABLED Helper macro.
Guarded code will be compiled when config is set to m or y.
CRs-Fixed: 2579332
Change-Id: Ic1a5ab22c8ced2c7b527bf5086c2c7b14fe24919
Signed-off-by: himta ram <hram@codeaurora.org>
