The find_best_target() looks for the higher capacity CPU when
the task is boosted. However the current code does not consider
the boost while deciding the start_cpu. This may result in
unnecessary traversal of the lower capacity CPU group in
find_best_target().
Change-Id: I8153482e15d2c9ea8971415edb9dbd54e10e11d0
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The following tracepoints were added to include/trace/events/power.h
but not exported:
ERROR: "__tracepoint_memlat_dev_update" [drivers/devfreq/governor_memlat.ko]
ERROR: "__tracepoint_memlat_dev_meas" [drivers/devfreq/governor_memlat.ko]
ERROR: "__tracepoint_cache_hwmon_update" [drivers/devfreq/governor_cache_hwmon.ko]
ERROR: "__tracepoint_cache_hwmon_meas" [drivers/devfreq/governor_cache_hwmon.ko]
ERROR: "__tracepoint_bw_hwmon_update" [drivers/devfreq/governor_bw_hwmon.ko]
ERROR: "__tracepoint_bw_hwmon_meas" [drivers/devfreq/governor_bw_hwmon.ko]
Change-Id: Ic0dedbadbee6533cba9ae312ab764873ab72a342
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
When WALT & UCLAMP are enabled, compilation is broken.
Change-Id: Id0b23f1d0fd0131f390c062076bddad5104ddd51
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Even though best_energy_cpu is idle and which would consume
same amount of energy as prev cpu, due to 6% check with prev cpu,
we end up choosing prev cpu over best_energy_cpu for the tasks.
This leads to increase in runnables. Choose best_energy_cpu when
it is idle and shallowest c-state over prev cpu in such scenarios
to avoid runnables.
Change-Id: Ib9eee263640bcd43bd80f8f777a8312585991bc5
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
If both wakee and waker are in top-app,don't place wakee on waker
cpu, because most of this case waker is not going to sleep,this
will cause waker preempted and lead high runnable time,so let
scheduler to select the best cpu in this case.
Change-Id: If6eb3c0acb202197e9dd90205c581398ac049649
Signed-off-by: Biao Long <blong@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I7c3a073fa68d715dd3bdbed11e4f83e82046a3c0
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I7a981c0aa4826ae448d4de224966dec7e21dab86
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This reverts commit 1e6b1214f107503 ("FROMLIST: sched/topology:
Make Energy Aware Scheduling depend on schedutil").
Right now, EAS gets disabled when governor set to anything other
than schedutil which affects scheduling decisions. To make sure
we always get better scheduling decions, EAS needs to be supported
irrespective of the governor.
Change-Id: I788faaacd477b3ff93ea9787c18a1235c8c3eb0b
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
[shalagra@codeaurora.org: Fixed merge conflicts in port to 5.x]
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 2508aa54fb9e
(sched: Improve the scheduler").
Change-Id: I193900db684515f6585371b7c58d5b9c361c2e6b
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Add snapshot of runqueue stats module as of commit <e56fe938419e>
(Merge "uapi: Introduce V4L2_FLAG_CVPMETADATA_SKIP flag").
Added #ifdefery to allow rq_stats to actually be optional.
Change-Id: I3352a472c5b0ad61cc04317a1d6f26549bfcdc04
Signed-off-by: Elliot Berman <eberman@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I5fbadf248c0bfe27bc761686de7a925cec2e4163
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
[satyap@codeaurora.org: port to 5.4 and fix trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I33e9ec890f8b54d673770d5d02dba489a8e08ce7
Signed-off-by: Sai Harshini Nimmala <snimmala@codeaurora.org>
[satyap@codeaurora.org: port to 5.4 and resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
If this runqueue have iowait task and still force newly
idle load balance, will make iowait task performance
worse. Need to do the average idle time check if
there is iowait task on this runqueue to have upcoming
iowait task have enough cpu resource.
Change-Id: I8e4dd440edbbd14d5170fbcf358bdaf5cc5b26e8
Signed-off-by: Maria Yu <aiquny@codeaurora.org>
Signed-off-by: Biao Long <blong@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I7e6cc760a6add9ebdab7d67a176559bc640f7790
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[satyap@codeaurora.org: port to 5.4 and minor update for WALT]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
We often see traces that gold cpus goes to idle even when silver
has big tasks. This is because avg_idle checks does not allow
the gold cpus to enter idle balance. This is NOT okay when
silver CPUs have big tasks.
Change-Id: Ice00d43c70be105b5bc3ad09caa6047aa98d7402
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Runmin Wang <runminw@codeaurora.org>
[satyap@codeaurora.org: port to 5.4 and add WALT checks]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Dup ftrace event traffic and writes to trace_marker file from
userspace to STM. Also dup trace printk traffic to STM. This
allows Linux tracing and log data to be correlated with other
data transported over STM.
Change-Id: Ie9a2e1a8c9ea542226f387e9207f3d736d9187ab
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
[spjoshi@codeaurora.org: 3.18 code fixup]
Signed-off-by: Sarangdhar Joshi <spjoshi@codeaurora.org>
[mittals@codeaurora.org: 4.4 code fixup]
Signed-off-by: Shashank Mittal <mittals@codeaurora.org>
[sadesai@codeaurora.org: 4.8 code fixup]
Signed-off-by: Satyajit Desai <sadesai@codeaurora.org>
Signed-off-by: Rama Aparna Mallavarapu <aparnam@codeaurora.org>
Signed-off-by: Tingwei Zhang <tingwei@codeaurora.org>
The current derivation of pfns/page structures has a fair
amount of complexity in it, and can be simplified. For example,
there need not be completely separate cases for pages that are
all remapped into the vmalloc space, but were allocated
differently (i.e. atomic allocations vs contiguous allocations).
So, simplify the handling for those, and leverage iommu_dma_mmap()
and iommu_dma_get_sgtable(), as there is nothing fastmap-specific
with the way pages are mapped into userspace and how we construct
an sg-table respectively.
Change-Id: Ia5725a93ee84f2c5e7ccc1d6e62f828338e83bd7
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
The generic atomic pool is initialized at the postcore
initcall level. However, the same function is called to
initialize the atomic pool for fastmap at a later level,
and that function makes it so that the generic pool is
clobbered, and we start using the same atomic pool for
both fastmap and allocations outside of fastmap.
Fix this by adding support in the atomic pool framework
to create and utilize different atomic pools, and creating
a private atomic pool for fastmap.
Change-Id: I934774f44cc25ff66157be7920895618cf76b812
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I15c01c20691899053e0aa760923ba0db644f2b73
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: port to 5.4 and resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I9216f9316e2bad067c10762de8d67912826b7bc7
Signed-off-by: Maria Yu <aiquny@codeaurora.org>
Co-developed-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[pkondeti@codeaurora.org: skip_cpu argument is implemented for fbt]
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: port to 5.4 and fix trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This patch adds a parameter to select_task_rq, sibling_count_hint
allowing the caller, where it has this information, to inform the
sched_class the number of tasks that are being woken up as part of
the same event.
The wake_q mechanism is one case where this information is available.
select_task_rq_fair can then use the information to detect that it
needs to widen the search space for task placement in order to avoid
overloading the last-level cache domain's CPUs.
* * *
The reason I am investigating this change is the following use case
on ARM big.LITTLE (asymmetrical CPU capacity): 1 task per CPU, which
all repeatedly do X amount of work then
pthread_barrier_wait (i.e. sleep until the last task finishes its X
and hits the barrier). On big.LITTLE, the tasks which get a "big" CPU
finish faster, and then those CPUs pull over the tasks that are still
running:
v CPU v ->time->
-------------
0 (big) 11111 /333
-------------
1 (big) 22222 /444|
-------------
2 (LITTLE) 333333/
-------------
3 (LITTLE) 444444/
-------------
Now when task 4 hits the barrier (at |) and wakes the others up,
there are 4 tasks with prev_cpu=<big> and 0 tasks with
prev_cpu=<little>. want_affine therefore means that we'll only look
in CPUs 0 and 1 (sd_llc), so tasks will be unnecessarily coscheduled
on the bigs until the next load balance, something like this:
v CPU v ->time->
------------------------
0 (big) 11111 /333 31313\33333
------------------------
1 (big) 22222 /444|424\4444444
------------------------
2 (LITTLE) 333333/ \222222
------------------------
3 (LITTLE) 444444/ \1111
------------------------
^^^
underutilization
So, I'm trying to get want_affine = 0 for these tasks.
I don't _think_ any incarnation of the wakee_flips mechanism can help
us here because which task is waker and which tasks are wakees
generally changes with each iteration.
However pthread_barrier_wait (or more accurately FUTEX_WAKE) has the
nice property that we know exactly how many tasks are being woken, so
we can cheat.
It might be a disadvantage that we "widen" _every_ task that's woken in
an event, while select_idle_sibling would work fine for the first
sd_llc_size - 1 tasks.
IIUC, if wake_affine() behaves correctly this trick wouldn't be
necessary on SMP systems, so it might be best guarded by the presence
of SD_ASYM_CPUCAPACITY?
* * *
Final note..
In order to observe "perfect" behaviour for this use case, I also had
to disable the TTWU_QUEUE sched feature. Suppose during the wakeup
above we are working through the work queue and have placed tasks 3
and 2, and are about to place task 1:
v CPU v ->time->
--------------
0 (big) 11111 /333 3
--------------
1 (big) 22222 /444|4
--------------
2 (LITTLE) 333333/ 2
--------------
3 (LITTLE) 444444/ <- Task 1 should go here
--------------
If TTWU_QUEUE is enabled, we will not yet have enqueued task
2 (having instead sent a reschedule IPI) or attached its load to CPU
2. So we are likely to also place task 1 on cpu 2. Disabling
TTWU_QUEUE means that we enqueue task 2 before placing task 1,
solving this issue. TTWU_QUEUE is there to minimise rq lock
contention, and I guess that this contention is less of an issue on
big.LITTLE systems since they have relatively few CPUs, which
suggests the trade-off makes sense here.
Signed-off-by: Brendan Jackman <brendan.jackman@arm.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Josef Bacik <josef@toxicpanda.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Matt Fleming <matt@codeblueprint.co.uk>
( - Applied from https://patchwork.kernel.org/patch/9895261/
- Fixed trivial conflict in kernel/sched/core.c
- Fixed select_task_rq_idle, now in kernel/sched/idle.c
- Fixed trivial conflict in select_task_rq_fair )
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Change-Id: I3cfc4bf48c3d7feef969db4d22449f4fbb4f795d
[satyap@codeaurora.org: port to 5.4 and fix trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This change is for general scheduler improvement.
Change-Id: I7a84c7f7444d57f31e0859c4d39450d23421eb73
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
The current code uses the rq->cpu_capacity_orig while sorting
the min/mid/max CPUs while rebuilding the scheduler domains. This
rq->cpu_capacity_orig is subjected to changed when the scaling max
frequency is clipped for that CPU's frequency domain. Since we don't
recompute the min/mid/max CPUs when the capacity is changed, the
sorting becomes incorrect later when the frequency limits are lifted.
The task CPU selection algorithm depends heavily on min/mid/max CPUs
and changing them on the fly results in incorrect task placement.
Hence use the true capacity of CPUs while sorting the min/mid/max
CPUs. This means that the sorting gets changed only when CPUs are
hotplugged out, otherwise these reflect the correct topology
all the time.
The max_cpu_capacity struct in root domain also maintains the
max CPU and its capacity. Since we now use true capacity, there
is no need to compute this on the fly. So move this evaluation from
the periodic load balancer to the scheduler domain rebuilding.
The rq->cpu_capacity_orig is still subjected to change upon frequency
or thermal limits. So we still identify the cases where tasks not
fitting on the max/mid CPUs when the capacity is reduced.
Change-Id: I42735accd079b2ece1eb58d1ebcf322d454e33a2
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: trivial changes to arch_scale_cpu_capacity()]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
When tasks move across cpusets, the current affinity settings
are lost. Cache the task affinity and restore it during cpuset
migration. The restoring happens only when the cached affinity
is subset of the current cpuset settings.
Change-Id: I6c2ec1d5e3d994e176926d94b9e0cc92418020cc
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: fix trivial merge conflicts and
replace cs->cpus_requested with cs->cpus_allowed]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Extend sched_load_balance trace event to print overutilization
status of the current balancing sched domain.
Change-Id: Iae14e580294a0ab38394b4e8d660d0c5dd14327d
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
It is quite a common occurrence that we have a misfit task (a task
running on a CPU which is close to the capacity limit of that CPU)
in the system AND we have recently done a load balance operation.
Normal load-balance rate limiting means we can leave those tasks on
the restrictive CPU for a few hundred ms before we move them.
Inspired by the work Leo Yan did on balancing misfits.
This patch reduces the balance interval to one jiffy when there are such
tasks running.
There appears to be no impact on interactive performance from this
patch.
Change-Id: I71d74734d3bd6bca80476c0f466ba48914b7ac02
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Git-commit: 84c88fc37932db51b87af8df3cedc621cb5c6e97
Git-repo: https://android.googlesource.com/kernel/common
[clingutla@codeaurora.org: Took only partial part from commit
'84c88fc37932("ANDROID: Combined EAS Load Balance Tweaks")',
and skip using 1 jiffy for busy balancing]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
The current implementation of overutilization, aborts energy aware
scheduling if any cpu in the system is over-utilized. This patch introduces
over utilization flag per sched domain level instead of a single flag
system wide. Load balancing is done at the sched domain where any
of the cpu is over utilized. If energy aware scheduling is
enabled and no cpu in a sched domain is overuttilized,
load balancing is skipped for that sched domain and energy aware
scheduling continues at that level.
The implementation takes advantage of the shared sched_domain structure
that is common across all the sched domains at a level. The new flag
introduced is placed in this structure so that all the sched domains the
same level share the flag. In case of an overutilized cpu, the flag gets
set at level1 sched_domain. The flag at the parent sched_domain level gets
set in either of the two following scenarios.
1. There is a misfit task in one of the cpu's in this sched_domain.
2. The total utilization of the domain is greater than the domain capacity
The flag is cleared if no cpu in a sched domain is overutilized.
This implementation still can have corner scenarios with respect to
misfit tasks. For example consider a sched group with n cpus and
n+1 70%utilized tasks. Ideally this is a case for load balance to happen
in a parent sched domain. But neither the total group utilization is
high enough for the load balance to be triggered in the parent domain nor
there is a cpu with a single overutilized task so that aload balance is
triggered in a parent domain. But again this could be a purely academic
sceanrio, as during task wake up these tasks will be placed
more appropriately.
Signed-off-by: Thara Gopinath <thara.gopinath@linaro.org>
Change-Id: I3f327cff4080096a3e58208dd72c9b7f7913cdb2
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Git-commit: addef37808728c719d8c095a75bcf81befdacdaf
Git-repo: https://android.googlesource.com/kernel/common/
[clingutla@codeaurora.org: Resolved trivial merge conflicts]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[satyap@codeaurora.org: port to 5.x and resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This snapshot is taken from msm-4.19 as of commit 5debecbe7195
("trace: filter out spurious preemption and IRQs disable traces").
Change-Id: I3c9663da1fd89e9e942831fda00a47b4a29ea4e3
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Watchdog gets disabled from other CPUs when isolation is enabled.
Using this_cpu_ptr() would lead to undesired issues, so, switch to
per_cpu_ptr(). While at it, remove warning as it makes no sense
when core isolation is enabled.
Change-Id: Id1a59c4c88044c7570457fb55abb2dcc13de3c11
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Ksfotirqd is a normal priority CFS task. It can experience higher
scheduling latency under heavy load conditions. Currently once
asynchronous softirq processing is deferred to ksoftirqd, softirqs
are not processed further until ksoftirqd task gets a chance to run.
High latencies for softirqs like TIMER, HI TASKLET is not acceptable.
So revert 'commit 4cd13c21b2 ("softirq: Let ksoftirqd do its job")'.
Change-Id: I38a1a88b5f42dd534c65d739dbb7e4321a7904db
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[satyap@codeaurora.org: Fix trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
This reverts commit 3c53776e29.
Reverting the change to match what we have on previous Kernels.
Change-Id: I4af64cd7e2c4291dda5f503bf2d74ede459a76c6
[satyap@codeaurora.org: Port to 5.x and fix trivial merge conflict]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
An IRQ affinity is broken during hotplug/isolation when there
are no online and un-isolated CPUs in the current affinity mask.
An online and un-isolated CPU from the irq_default_affinity mask
(i.e /proc/irq/default_smp_affinity) is used as the current
affinity. However Individual IRQs can have their affinity hint
set via irq_set_affinity_hint() API. When such hint is available,
use it instead of the irq_default_affinity which is a system level
setting.
Change-Id: I53a537582ec4e1aed0c59b49f4fd5b6ca7c0c332
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Since 'commit f332a9d53e33 ("softirq: defer softirq processing to
ksoftirqd if CPU is busy with RT")', all softirqs are deferred
to ksoftirqd if one of the potentially long softirqs are pending.
This can significantly delay processing of tasklets and timers,
which are frequently designed to be running at high priority.
Defer only the potentially slow softirqs to ksoftirqd.
Change-Id: I7a2ef7c59749ccf086066c59962ff326786da6f4
Signed-off-by: Björn Davidsson <bjorn.davidsson@sony.com>
[clingutla@codeaurora.org: Renamed local variables and refactored
deferred logic to macro for avoiding potential deadlock.]
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
The cpupri_find() returns the candidate CPUs which are running
lower priority than the waking RT task in the lowest_mask. This
contains isolated CPUs as well. Since the energy aware CPU selection
skips isolated CPUs, no target CPU may be found if all unisolated CPUs
are running higher priority RT tasks. In which case, we fallback to
the default CPU selection algorithm and returns an isolated CPU. This
decision is reversed by select_task_rq() and returns an unisolated
CPU that is busy with other RT tasks. This RT task packing is desired
behavior. However, RT push mechanism pushes the packed RT task to
an isolated CPU. This can be avoided by excluding isolated CPUs from
the lowest_mask returned by cpupri_find().
Change-Id: I75486b3935caf496a638d0333565beffc47fe249
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: trivial merge conflict resolution]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Defer the softirq processing to ksoftirqd if a RT task is running
or queued on the current CPU. This complements the RT task placement
algorithm which tries to find a CPU that is not currently busy with
softirqs.
Currently NET_TX, NET_RX, BLOCK and TASKLET softirqs are only deferred
as they can potentially run for long time.
Change-Id: Id7665244af6bbd5a96d9e591cf26154e9eaa860c
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: trivial merge conflict resolution.]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
The scheduling change (bug 31501544) to avoid putting RT threads on cores that
are handling softint's was catching cases where there was no reason
to believe the softint would take a long time, resulting in unnecessary
migration overhead. This patch reduces the migration to cases where
the core has a softint that is actually likely to take a long time,
as opposed to the RCU, SCHED, and TIMER softints that are rather quick.
Bug: 31752786
Change-Id: Ib4e179f1e15c736b2fdba31070494e357e9fbbe2
Git-commit: ce05770bd37b8065b61ef650108ecef2b97b148b
Git-repo: https://android.googlesource.com/kernel/msm
[pkondeti@codeaurora.org: resolved minor merge conflicts]
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: resolve trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
