blkcg: add blkg_[rw]stat->aux_cnt and replace cfq_group->dead_stats with it

cgroup stats are local to each cgroup and doesn't propagate to
ancestors by default.  When recursive stats are necessary, the sum is
calculated over all the descendants.  This initially was for backward
compatibility to support both group-local and recursive stats but this
mode of operation makes general sense as stat update is much hotter
thafn reporting those stats.

This however ends up losing recursive stats when a child is removed.
To work around this, cfq-iosched adds its stats to its parent
cfq_group->dead_stats which is summed up together when calculating
recursive stats.

It's planned that the core stats will be moved to blkcg_gq, so we want
to move the mechanism for keeping track of the stats of dead children
from cfq to blkcg core.  This patch adds blkg_[rw]stat->aux_cnt which
are atomic64_t's keeping track of auxiliary counts which are excluded
when reading local counts but included for recursive.

blkg_[rw]stat_merge() which were used by cfq to implement dead_stats
are replaced by blkg_[rw]stat_add_aux(), and cfq now forwards stats of
a dead cgroup to the aux counts of parent->stats instead of separate
->dead_stats.

This will also help making blkg_[rw]stats per-cpu.

Signed-off-by: Tejun Heo <[email protected]>
Cc: Vivek Goyal <[email protected]>
Signed-off-by: Jens Axboe <[email protected]>
This commit is contained in:
Tejun Heo 2015-08-18 14:55:21 -07:00 • committed by Jens Axboe
commit e6269c4454
3 changed files with 57 additions and 66 deletions

View file

@ -584,7 +584,7 @@ EXPORT_SYMBOL_GPL(blkg_prfill_rwstat);
* @off: offset to the blkg_stat in @pd
*
* Collect the blkg_stat specified by @off from @pd and all its online
* descendants and return the sum. The caller must be holding the queue
* descendants and their aux counts. The caller must be holding the queue
* lock for online tests.
*/
u64 blkg_stat_recursive_sum(struct blkg_policy_data *pd, int off)
@ -602,7 +602,8 @@ u64 blkg_stat_recursive_sum(struct blkg_policy_data *pd, int off)
struct blkg_stat *stat = (void *)pos_pd + off;
if (pos_blkg->online)
sum += blkg_stat_read(stat);
sum += blkg_stat_read(stat) +
atomic64_read(&stat->aux_cnt);
}
rcu_read_unlock();
@ -616,7 +617,7 @@ EXPORT_SYMBOL_GPL(blkg_stat_recursive_sum);
* @off: offset to the blkg_stat in @pd
*
* Collect the blkg_rwstat specified by @off from @pd and all its online
* descendants and return the sum. The caller must be holding the queue
* descendants and their aux counts. The caller must be holding the queue
* lock for online tests.
*/
struct blkg_rwstat blkg_rwstat_recursive_sum(struct blkg_policy_data *pd,
@ -642,7 +643,8 @@ struct blkg_rwstat blkg_rwstat_recursive_sum(struct blkg_policy_data *pd,
tmp = blkg_rwstat_read(rwstat);
for (i = 0; i < BLKG_RWSTAT_NR; i++)
sum.cnt[i] += tmp.cnt[i];
sum.cnt[i] += tmp.cnt[i] +
atomic64_read(&rwstat->aux_cnt[i]);
}
rcu_read_unlock();