Martin KaFai Lau
6ac99e8f23
bpf: Introduce bpf sk local storage ...
After allowing a bpf prog to
- directly read the skb->sk ptr
- get the fullsock bpf_sock by "bpf_sk_fullsock()"
- get the bpf_tcp_sock by "bpf_tcp_sock()"
- get the listener sock by "bpf_get_listener_sock()"
- avoid duplicating the fields of "(bpf_)sock" and "(bpf_)tcp_sock"
into different bpf running context.
this patch is another effort to make bpf's network programming
more intuitive to do (together with memory and performance benefit).
When bpf prog needs to store data for a sk, the current practice is to
define a map with the usual 4-tuples (src/dst ip/port) as the key.
If multiple bpf progs require to store different sk data, multiple maps
have to be defined. Hence, wasting memory to store the duplicated
keys (i.e. 4 tuples here) in each of the bpf map.
[ The smallest key could be the sk pointer itself which requires
some enhancement in the verifier and it is a separate topic. ]
Also, the bpf prog needs to clean up the elem when sk is freed.
Otherwise, the bpf map will become full and un-usable quickly.
The sk-free tracking currently could be done during sk state
transition (e.g. BPF_SOCK_OPS_STATE_CB).
The size of the map needs to be predefined which then usually ended-up
with an over-provisioned map in production. Even the map was re-sizable,
while the sk naturally come and go away already, this potential re-size
operation is arguably redundant if the data can be directly connected
to the sk itself instead of proxy-ing through a bpf map.
This patch introduces sk->sk_bpf_storage to provide local storage space
at sk for bpf prog to use. The space will be allocated when the first bpf
prog has created data for this particular sk.
The design optimizes the bpf prog's lookup (and then optionally followed by
an inline update). bpf_spin_lock should be used if the inline update needs
to be protected.
BPF_MAP_TYPE_SK_STORAGE:
-----------------------
To define a bpf "sk-local-storage", a BPF_MAP_TYPE_SK_STORAGE map (new in
this patch) needs to be created. Multiple BPF_MAP_TYPE_SK_STORAGE maps can
be created to fit different bpf progs' needs. The map enforces
BTF to allow printing the sk-local-storage during a system-wise
sk dump (e.g. "ss -ta") in the future.
The purpose of a BPF_MAP_TYPE_SK_STORAGE map is not for lookup/update/delete
a "sk-local-storage" data from a particular sk.
Think of the map as a meta-data (or "type") of a "sk-local-storage". This
particular "type" of "sk-local-storage" data can then be stored in any sk.
The main purposes of this map are mostly:
1. Define the size of a "sk-local-storage" type.
2. Provide a similar syscall userspace API as the map (e.g. lookup/update,
map-id, map-btf...etc.)
3. Keep track of all sk's storages of this "type" and clean them up
when the map is freed.
sk->sk_bpf_storage:
------------------
The main lookup/update/delete is done on sk->sk_bpf_storage (which
is a "struct bpf_sk_storage"). When doing a lookup,
the "map" pointer is now used as the "key" to search on the
sk_storage->list. The "map" pointer is actually serving
as the "type" of the "sk-local-storage" that is being
requested.
To allow very fast lookup, it should be as fast as looking up an
array at a stable-offset. At the same time, it is not ideal to
set a hard limit on the number of sk-local-storage "type" that the
system can have. Hence, this patch takes a cache approach.
The last search result from sk_storage->list is cached in
sk_storage->cache[] which is a stable sized array. Each
"sk-local-storage" type has a stable offset to the cache[] array.
In the future, a map's flag could be introduced to do cache
opt-out/enforcement if it became necessary.
The cache size is 16 (i.e. 16 types of "sk-local-storage").
Programs can share map. On the program side, having a few bpf_progs
running in the networking hotpath is already a lot. The bpf_prog
should have already consolidated the existing sock-key-ed map usage
to minimize the map lookup penalty. 16 has enough runway to grow.
All sk-local-storage data will be removed from sk->sk_bpf_storage
during sk destruction.
bpf_sk_storage_get() and bpf_sk_storage_delete():
------------------------------------------------
Instead of using bpf_map_(lookup|update|delete)_elem(),
the bpf prog needs to use the new helper bpf_sk_storage_get() and
bpf_sk_storage_delete(). The verifier can then enforce the
ARG_PTR_TO_SOCKET argument. The bpf_sk_storage_get() also allows to
"create" new elem if one does not exist in the sk. It is done by
the new BPF_SK_STORAGE_GET_F_CREATE flag. An optional value can also be
provided as the initial value during BPF_SK_STORAGE_GET_F_CREATE.
The BPF_MAP_TYPE_SK_STORAGE also supports bpf_spin_lock. Together,
it has eliminated the potential use cases for an equivalent
bpf_map_update_elem() API (for bpf_prog) in this patch.
Misc notes:
----------
1. map_get_next_key is not supported. From the userspace syscall
perspective, the map has the socket fd as the key while the map
can be shared by pinned-file or map-id.
Since btf is enforced, the existing "ss" could be enhanced to pretty
print the local-storage.
Supporting a kernel defined btf with 4 tuples as the return key could
be explored later also.
2. The sk->sk_lock cannot be acquired. Atomic operations is used instead.
e.g. cmpxchg is done on the sk->sk_bpf_storage ptr.
Please refer to the source code comments for the details in
synchronization cases and considerations.
3. The mem is charged to the sk->sk_omem_alloc as the sk filter does.
Benchmark:
---------
Here is the benchmark data collected by turning on
the "kernel.bpf_stats_enabled" sysctl.
Two bpf progs are tested:
One bpf prog with the usual bpf hashmap (max_entries = 8192) with the
sk ptr as the key. (verifier is modified to support sk ptr as the key
That should have shortened the key lookup time.)
Another bpf prog is with the new BPF_MAP_TYPE_SK_STORAGE.
Both are storing a "u32 cnt", do a lookup on "egress_skb/cgroup" for
each egress skb and then bump the cnt. netperf is used to drive
data with 4096 connected UDP sockets.
BPF_MAP_TYPE_HASH with a modifier verifier (152ns per bpf run)
27: cgroup_skb name egress_sk_map tag 74f56e832918070b run_time_ns 58280107540 run_cnt 381347633
loaded_at 2019-04-15T13:46:39-0700 uid 0
xlated 344B jited 258B memlock 4096B map_ids 16
btf_id 5
BPF_MAP_TYPE_SK_STORAGE in this patch (66ns per bpf run)
30: cgroup_skb name egress_sk_stora tag d4aa70984cc7bbf6 run_time_ns 25617093319 run_cnt 390989739
loaded_at 2019-04-15T13:47:54-0700 uid 0
xlated 168B jited 156B memlock 4096B map_ids 17
btf_id 6
Here is a high-level picture on how are the objects organized:
sk
┌──────┐
│ │
│ │
│ │
│*sk_bpf_storage─────▶ bpf_sk_storage
└──────┘ ┌───────┐
┌───────────┤ list │
│ │ │
│ │ │
│ │ │
│ └───────┘
│
│ elem
│ ┌────────┐
├─▶│ snode │
│ ├────────┤
│ │ data │ bpf_map
│ ├────────┤ ┌─────────┐
│ │map_node│◀─┬─────┤ list │
│ └────────┘ │ │ │
│ │ │ │
│ elem │ │ │
│ ┌────────┐ │ └─────────┘
└─▶│ snode │ │
├────────┤ │
bpf_map │ data │ │
┌─────────┐ ├────────┤ │
│ list ├───────▶│map_node│ │
│ │ └────────┘ │
│ │ │
│ │ elem │
└─────────┘ ┌────────┐ │
┌─▶│ snode │ │
│ ├────────┤ │
│ │ data │ │
│ ├────────┤ │
│ │map_node│◀─┘
│ └────────┘
│
│
│ ┌───────┐
sk └──────────│ list │
┌──────┐ │ │
│ │ │ │
│ │ │ │
│ │ └───────┘
│*sk_bpf_storage───────▶bpf_sk_storage
└──────┘
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2019-04-27 09:07:04 -07:00
libceph: wait for latest osdmap in ceph_monc_blacklist_add()
2019-03-20 16:27:40 +01:00
ARM: SoC driver updates for 5.1
2019-03-06 09:41:12 -08:00
irqchip updates for 5.1, take #2
2019-03-21 12:30:54 +01:00
mailbox: ZynqMP IPI mailbox controller
2019-03-06 20:34:20 -06:00
- New Drivers
2019-03-08 10:02:58 -08:00
mlx5-updates-2019-04-02
2019-04-08 14:31:25 -07:00
mmc: core: Add discard support to sd
2019-02-28 09:16:12 +01:00
netfilter: make two functions static
2019-04-08 23:28:33 +02:00
Here's more than a handful of clk driver fixes for changes that came in
2019-04-13 14:33:56 -07:00
regulator: core: Add set/get_current_limit helpers for regmap users
2019-03-03 23:45:27 +00:00
ptrace: take into account saved_sigmask in PTRACE{GET,SET}SIGMASK
2019-03-29 10:01:37 -07:00
Revert "SUNRPC: Micro-optimise when the task is known not to be sleeping"
2019-04-11 15:41:14 -04:00
wusb: Remove unnecessary static function ckhdid_printf
2019-03-01 20:53:41 +01:00
device-dax for 5.1
2019-03-16 13:05:32 -07:00
appletalk: Fix potential NULL pointer dereference in unregister_snap_client
2019-03-15 11:25:48 -07:00
writeback: fix inode cgroup switching comment
2019-03-05 21:07:21 -08:00
mm: convert PG_balloon to PG_offline
2019-03-05 21:07:14 -08:00
block: don't use for-inside-for in bio_for_each_segment_all
2019-04-08 08:19:41 -06:00
include/linux/bitops.h: set_mask_bits() to return old value
2019-03-07 18:32:00 -08:00
include/linux/bitrev.h: fix constant bitrev
2019-04-05 16:02:30 -10:00
blk-mq: introduce blk_mq_complete_request_sync()
2019-04-10 09:57:33 -06:00
block: add BIO_NO_PAGE_REF flag
2019-03-18 10:44:48 -06:00
block: add BLK_MQ_POLL_CLASSIC for hybrid poll and return EINVAL for unexpected value
2019-03-20 14:02:07 -06:00
bpf: Add file_pos field to bpf_sysctl ctx
2019-04-12 13:54:58 -07:00
bpf: Introduce bpf sk local storage
2019-04-27 09:07:04 -07:00
bpf: Introduce bpf sk local storage
2019-04-27 09:07:04 -07:00
bpf: remove global variables
2019-04-23 01:50:43 +02:00
net: phy: bcm54xx: Encode link speed and activity into LEDs
2019-03-26 11:24:47 -07:00
bpf: allow for key-less BTF in array map
2019-04-09 17:05:46 -07:00
build_bug.h: add wrapper for _Static_assert
2019-03-07 18:31:59 -08:00
block: don't use for-inside-for in bio_for_each_segment_all
2019-04-08 08:19:41 -06:00
audit/stable-5.1 PR 20190305
2019-03-07 12:20:11 -08:00
Merge branch 'for-5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
2019-03-07 10:11:41 -08:00
include/linux/compaction.h: fix potential build error
2019-03-05 21:07:20 -08:00
init/calibrate.c: provide proper prototype
2019-03-07 18:32:01 -08:00
dm: add support to directly boot to a mapped device
2019-03-05 14:53:50 -05:00
driver core: remove BUS_ATTR()
2019-03-18 07:07:30 +01:00
virtio: fixes, cleanups
2019-03-10 12:47:57 -07:00
dynamic_debug: refactor dynamic_pr_debug and friends
2019-03-07 18:32:00 -08:00
arm64 updates for 5.1:
2019-03-10 10:17:23 -07:00
vfs: Add configuration parser helpers
2019-02-28 03:28:53 -05:00
net: pass net_device argument to the eth_get_headlen
2019-04-23 18:36:34 +02:00
f2fs: correct spelling mistake
2019-03-12 18:59:17 -07:00
fs: add fget_many() and fput_many()
2019-02-28 08:24:23 -07:00
bpf: Add file_pos field to bpf_sysctl ctx
2019-04-12 13:54:58 -07:00
mm: rid swapoff of quadratic complexity
2019-03-05 21:07:18 -08:00
fs: stream_open - opener for stream-like files so that read and write can run simultaneously without deadlock
2019-04-06 07:01:55 -10:00
vfs: Implement logging through fs_context
2019-02-28 03:29:37 -05:00
vfs: Add configuration parser helpers
2019-02-28 03:28:53 -05:00
generic radix trees
2019-03-12 10:04:02 -07:00
genetlink: make policy common to family
2019-03-22 10:38:23 -04:00
mm: shuffle GFP_* flags
2019-03-05 21:07:16 -08:00
mm/hmm: convert to use vm_fault_t
2019-03-12 10:04:00 -07:00
include/linux/hugetlb.h: convert to use vm_fault_t
2019-03-29 10:01:37 -07:00
bridge: broute: make broute a real ebtables table
2019-04-12 01:47:50 +02:00
ip: fix ip_mc_may_pull() return value
2019-03-09 19:52:47 -08:00
ipv4: Move IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN to helper
2019-03-29 10:48:03 -07:00
Merge branches 'iommu/fixes', 'arm/msm', 'arm/tegra', 'arm/mediatek', 'x86/vt-d', 'x86/amd', 'hyper-v' and 'core' into next
2019-03-01 11:24:51 +01:00
net: ipv6: add socket option IPV6_ROUTER_ALERT_ISOLATE
2019-03-03 21:05:10 -08:00
genirq: Fix typo in comment of IRQD_MOVE_PCNTXT
2019-03-21 11:52:37 +01:00
This is the bulk of GPIO changes for the v5.1 cycle:
2019-03-08 10:09:53 -08:00
time: Introduce jiffies64_to_msecs()
2019-04-08 22:56:14 +02:00
kasan: fix kasan_check_read/write definitions
2019-03-05 21:07:13 -08:00
Linux 5.1-rc2
2019-03-28 10:58:28 +01:00
printk: Remove no longer used LOG_PREFIX.
2019-03-04 13:42:05 +01:00
linux/kernel.h: Use parentheses around argument in u64_to_user_ptr()
2019-04-03 11:43:49 +02:00
Merge branch 'work.mount' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-03-12 14:08:19 -07:00
mm: reuse only-pte-mapped KSM page in do_wp_page()
2019-03-05 21:07:15 -08:00
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-03-06 08:14:05 -08:00
KVM: fix spectrev1 gadgets
2019-04-16 15:38:07 +02:00
device-dax for 5.1
2019-03-16 13:05:32 -07:00
linux/kernel.h: split *_MAX and *_MIN macros into <linux/limits.h>
2019-03-07 18:31:59 -08:00
include/linux/list.h: fix list_is_first() kernel-doc
2019-03-29 10:01:37 -07:00
locking/lockdep: Shrink struct lock_class_key
2019-02-28 07:55:53 +01:00
Merge branch 'work.mount' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-03-12 14:08:19 -07:00
lib/lzo: separate lzo-rle from lzo
2019-03-07 18:32:03 -08:00
net: mdio: rename mdio_device reset to reset_gpio
2019-04-18 17:42:54 -07:00
memblock: remove memblock_{set,clear}_region_flags
2019-03-12 10:04:02 -07:00
mm: writeback: use exact memcg dirty counts
2019-04-05 16:02:31 -10:00
xen: fixes and features for 5.1-rc1
2019-03-11 17:08:14 -07:00
net: mii: Fix PAUSE cap advertisement from linkmode_adv_to_lcl_adv_t() helper
2019-03-27 22:49:06 -07:00
Merge branch 'page-refs' (page ref overflow)
2019-04-14 15:09:40 -07:00
mm: fix vm_fault_t cast in VM_FAULT_GET_HINDEX()
2019-04-05 16:02:31 -10:00
mm: remove zone_lru_lock() function, access ->lru_lock directly
2019-03-05 21:07:21 -08:00
platform-drivers-x86 for v5.1-1
2019-03-10 13:16:37 -07:00
A fairly routine cycle for docs - lots of typo fixes, some new documents,
2019-03-09 09:56:17 -08:00
net: rework SIOCGSTAMP ioctl handling
2019-04-19 14:07:40 -07:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2019-04-17 11:26:25 -07:00
netfilter: replace NF_NAT_NEEDED with IS_ENABLED(CONFIG_NF_NAT)
2019-04-08 23:02:52 +02:00
netfilter: nf_tables: merge route type into core
2019-04-08 23:01:42 +02:00
NFSv4.2: Add client support for the generic 'layouterror' RPC call
2019-03-01 16:20:16 -05:00
NFSv4.2: Add client support for the generic 'layouterror' RPC call
2019-03-01 16:20:16 -05:00
NFSv4.2: Add client support for the generic 'layouterror' RPC call
2019-03-01 16:20:16 -05:00
numa: make "nr_online_nodes" unsigned int
2019-03-05 21:07:20 -08:00
nvmet: fix discover log page when offsets are used
2019-04-11 17:28:30 +02:00
of: mark early_init_dt_alloc_reserved_memory_arch static
2019-02-28 11:40:49 -06:00
mm: better document PG_reserved
2019-03-05 21:07:19 -08:00
mm/hotplug: fix offline undo_isolate_page_range()
2019-03-29 10:01:37 -07:00
filemap: kill page_cache_read usage in filemap_fault
2019-03-15 11:21:25 -07:00
Revert "parport: daisy: use new parport device model"
2019-03-25 14:49:00 -07:00
PCI/ATS: Add inline to pci_prg_resp_pasid_required()
2019-02-28 11:09:29 +01:00
IOMMU Updates for Linux v5.1
2019-03-10 12:29:52 -07:00
locking/percpu-rwsem: Remove preempt_disable variants
2019-02-28 07:55:37 +01:00
perf/core improvements and fixes:
2019-02-28 08:29:50 +01:00
net: phy: improve link partner capability detection
2019-04-08 15:18:00 -07:00
include/linux/pid.h: remove next_pidmap() declaration
2019-03-07 18:31:59 -08:00
Merge branch 'page-refs' (page ref overflow)
2019-04-14 15:09:40 -07:00
This is the bulk of GPIO changes for the v5.1 cycle:
2019-03-08 10:09:53 -08:00
PM-runtime: Call pm_runtime_active|suspended_time() from sysfs
2019-03-07 11:23:17 +01:00
Power management updates for 5.1-rc1
2019-03-06 12:59:46 -08:00
PM / wakeup: Drop wakeup_source_drop()
2019-03-12 09:43:00 +01:00
Drop flex_arrays
2019-03-12 10:04:03 -07:00
Printk changes for 5.1
2019-03-09 09:22:42 -08:00
signal: add pidfd_send_signal() syscall
2019-03-05 17:03:53 +01:00
ptrace: Remove maxargs from task_current_syscall()
2019-04-04 09:17:15 -04:00
include/linux/relay.h: fix percpu annotation in struct rchan
2019-03-07 18:32:03 -08:00
rhashtable: use bit_spin_locks to protect hash bucket.
2019-04-07 19:12:12 -07:00
rhashtable: use BIT(0) for locking.
2019-04-12 17:34:45 -07:00
tracing: kdb: Fix ftdump to not sleep
2019-03-13 09:46:10 -04:00
sbitmap: trivial - update comment for sbitmap_deferred_clear_bit
2019-03-22 11:01:02 -06:00
audit/stable-5.1 PR 20190305
2019-03-07 12:20:11 -08:00
Merge branch 'work.mount' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-03-12 14:08:19 -07:00
mm: rid swapoff of quadratic complexity
2019-03-05 21:07:18 -08:00
inet: switch IP ID generator to siphash
2019-03-27 14:29:26 -07:00
bpf: support BPF_PROG_QUERY for BPF_FLOW_DISSECTOR attach_type
2019-04-25 23:49:06 +02:00
mm: add support for kmem caches in DMA32 zone
2019-03-29 10:01:37 -07:00
include/linux/slub_def.h: comment fixes
2019-03-05 21:07:14 -08:00
net: add documentation to socket.c
2019-03-15 15:29:47 -07:00
lib/string.c: implement a basic bcmp
2019-04-05 16:02:30 -10:00
include/linux/swap.h: use offsetof() instead of custom __swapoffset macro
2019-03-14 14:36:20 -07:00
virtio: fixes, cleanups
2019-03-10 12:47:57 -07:00
pidfd patches for v5.1-rc1
2019-03-16 13:47:14 -07:00
tee: add cancellation support to client interface
2019-02-28 13:49:29 +01:00
bpf: add writable context for raw tracepoints
2019-04-26 19:04:19 -07:00
include/linux/types.h: use "unsigned int" instead of "unsigned"
2019-03-07 18:31:59 -08:00
io_uring-20190323
2019-03-23 10:25:12 -07:00
virt: vbox: Implement passing requestor info to the host for VirtualBox 6.0.x
2019-03-28 01:55:18 +09:00
drm: move i915_kick_out_vgacon to vgaarb
2019-03-07 10:34:37 +01:00
virtio: Introduce virtio_max_dma_size()
2019-03-06 11:19:15 -05:00
virtio: hint if callbacks surprisingly might sleep
2019-03-06 11:19:57 -05:00
virtio: Honour 'may_reduce_num' in vring_create_virtqueue
2019-04-08 17:05:52 -04:00
platform/x86: wmi: move struct wmi_device_id to mod_devicetable.h
2019-03-07 08:46:07 -08:00
Driver core patches for 5.1-rc1
2019-03-06 14:52:48 -08:00
