Hannes Frederic Sowa
809fa972fd
reciprocal_divide: update/correction of the algorithm
...
Jakub Zawadzki noticed that some divisions by reciprocal_divide()
were not correct [1][2], which he could also show with BPF code
after divisions are transformed into reciprocal_value() for runtime
invariance which can be passed to reciprocal_divide() later on;
reverse in BPF dump ended up with a different, off-by-one K in
some situations.
This has been fixed by Eric Dumazet in commit aee636c480
("bpf: do not use reciprocal divide"). This follow-up patch
improves reciprocal_value() and reciprocal_divide() to work in
all cases by using Granlund and Montgomery method, so that also
future use is safe and without any non-obvious side-effects.
Known problems with the old implementation were that division by 1
always returned 0 and some off-by-ones when the dividend and divisor
where very large. This seemed to not be problematic with its
current users, as far as we can tell. Eric Dumazet checked for
the slab usage, we cannot surely say so in the case of flex_array.
Still, in order to fix that, we propose an extension from the
original implementation from commit 6a2d7a955d resp. [3][4],
by using the algorithm proposed in "Division by Invariant Integers
Using Multiplication" [5], Torbjörn Granlund and Peter L.
Montgomery, that is, pseudocode for q = n/d where q, n, d is in
u32 universe:
1) Initialization:
int l = ceil(log_2 d)
uword m' = floor((1<<32)*((1<<l)-d)/d)+1
int sh_1 = min(l,1)
int sh_2 = max(l-1,0)
2) For q = n/d, all uword:
uword t = (n*m')>>32
q = (t+((n-t)>>sh_1))>>sh_2
The assembler implementation from Agner Fog [6] also helped a lot
while implementing. We have tested the implementation on x86_64,
ppc64, i686, s390x; on x86_64/haswell we're still half the latency
compared to normal divide.
Joint work with Daniel Borkmann.
[1] http://www.wireshark.org/~darkjames/reciprocal-buggy.c
[2] http://www.wireshark.org/~darkjames/set-and-dump-filter-k-bug.c
[3] https://gmplib.org/~tege/division-paper.pdf
[4] http://homepage.cs.uiowa.edu/~jones/bcd/divide.html
[5] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1.2556
[6] http://www.agner.org/optimize/asmlib.zip
Reported-by: Jakub Zawadzki <darkjames-ws@darkjames.pl>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Austin S Hemmelgarn <ahferroin7@gmail.com>
Cc: linux-kernel@vger.kernel.org
Cc: Jesse Gross <jesse@nicira.com>
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: Andy Gospodarek <andy@greyhouse.net>
Cc: Veaceslav Falico <vfalico@redhat.com>
Cc: Jay Vosburgh <fubar@us.ibm.com>
Cc: Jakub Zawadzki <darkjames-ws@darkjames.pl>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-01-21 23:17:20 -08:00
..
9p
for-linus-3.12-merge minor 9p fixes and tweaks for 3.12 merge window
2013-09-11 12:34:13 -07:00
bluetooth
Bluetooth: Add quirk for disabling Delete Stored Link Key command
2014-01-04 20:10:40 +02:00
caif
caif_hsi.h: Remove extern from function prototypes
2013-09-23 16:29:41 -04:00
irda
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
iucv
af_iucv: fix recvmsg by replacing skb_pull() function
2013-04-08 17:16:57 -04:00
netfilter
netfilter: nf_tables: rename nft_do_chain_pktinfo() to nft_do_chain()
2014-01-09 20:17:16 +01:00
netns
ipv6: add flowlabel_consistency sysctl
2014-01-19 17:12:31 -08:00
nfc
Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next into for-davem
2014-01-17 14:43:17 -05:00
phonet
sctp
sctp: remove macros sctp_bh_[un]lock_sock
2014-01-21 18:41:36 -08:00
tc_act
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
act_api.h
net_sched: act: export tcf_hash_search() instead of tcf_hash_lookup()
2014-01-21 14:43:16 -08:00
addrconf.h
neigh: ipv6: respect default values set before an address is assigned to device
2013-12-09 20:56:12 -05:00
af_ieee802154.h
af_rxrpc.h
af_rxrpc.h: Remove extern from function prototypes
2013-07-31 17:50:01 -07:00
af_unix.h
af_unix: improve STREAM behavior with fragmented memory
2013-08-10 01:16:44 -07:00
af_vsock.h
VSOCK: Move af_vsock.h and vsock_addr.h to include/net
2013-07-27 22:14:06 -07:00
ah.h
arp.h
arp: make arp_invalidate static
2013-12-28 17:02:46 -05:00
atmclip.h
ax25.h
ax25.h: Remove extern from function prototypes
2013-07-31 17:50:02 -07:00
ax88796.h
busy_poll.h
net: add cpu_relax to busy poll loop
2013-08-28 17:45:48 -04:00
cfg80211-wext.h
cfg80211.h
cfg80211: Add a function to get the number of supported channels
2014-01-09 14:24:24 +01:00
checksum.h
net: checksum: fix warning in skb_checksum
2013-11-04 15:27:08 -05:00
cipso_ipv4.h
cipso: cleanup cipso_v4_translate() when !CONFIG_NETLABEL
2013-12-10 17:56:54 -05:00
cls_cgroup.h
net: net_cls: move cgroupfs classid handling into core
2014-01-03 23:41:41 +01:00
codel.h
net: introduce reciprocal_scale helper and convert users
2014-01-21 23:17:20 -08:00
compat.h
compat.h: Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
datalink.h
dcbevent.h
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
dcbnl.h
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
dn.h
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
dn_dev.h
dn_dev: add support for IFA_FLAGS nl attribute
2013-12-10 21:50:00 -05:00
dn_fib.h
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
dn_neigh.h
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
dn_nsp.h
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
dn_route.h
decnet (dn*.h): Remove extern from function prototypes
2013-09-20 14:49:32 -04:00
dsa.h
dsfield.h
dst.h
net: Add utility functions to clear rxhash
2013-12-17 16:36:21 -05:00
dst_ops.h
esp.h
net: move pskb_put() to core code
2013-11-07 19:28:58 -05:00
ethoc.h
fib_rules.h
fib_rules.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
firewire.h
firewire net, ipv4 arp: Extend hardware address and remove driver-level packet inspection.
2013-03-26 12:32:13 -04:00
flow.h
net: Remove FLOWI_FLAG_CAN_SLEEP
2013-12-06 07:24:39 +01:00
flow_keys.h
flow_dissector: factor out the ports extraction in skb_flow_get_ports
2013-10-03 15:36:37 -04:00
garp.h
garp.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
gen_stats.h
gen_stats.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
genetlink.h
genl: Add genlmsg_new_unicast() for unicast message allocation
2014-01-06 15:51:53 -08:00
gre.h
gre_offload: statically build GRE offloading support
2014-01-06 20:28:34 -05:00
gro_cells.h
icmp.h
icmp.h: Remove extern from function prototypes
2013-09-20 14:49:33 -04:00
ieee80211_radiotap.h
mac80211: add radiotap flag and handling for 5/10 MHz
2013-07-16 09:58:05 +03:00
ieee802154.h
ieee802154_netdev.h
ieee802154/nl-mac.c: make some MLME operations optional
2013-04-08 12:00:16 -04:00
if_inet6.h
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2014-01-18 00:55:41 -08:00
inet6_connection_sock.h
inet*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
inet6_hashtables.h
ipv6: split inet6_ehashfn to hash functions per compilation unit
2013-10-19 19:45:34 -04:00
inet_common.h
inet*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
inet_connection_sock.h
inet*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
inet_ecn.h
net: Correct comparisons and calculations using skb->tail and skb-transport_header
2013-05-28 23:49:07 -07:00
inet_frag.h
inet: remove old fragmentation hash initializing
2013-10-23 17:01:41 -04:00
inet_hashtables.h
tcp/dccp: remove twchain
2013-10-08 23:19:24 -04:00
inet_sock.h
inet: convert inet_ehash_secret and ipv6_hash_secret to net_get_random_once
2013-10-19 19:45:35 -04:00
inet_timewait_sock.h
ipv6: tcp: fix flowlabel value in ACK messages send from TIME_WAIT
2014-01-17 17:56:33 -08:00
inetpeer.h
ipv4: remove unused function
2013-12-28 17:03:20 -05:00
ip.h
ipv6: make IPV6_RECVPKTINFO work for ipv4 datagrams
2014-01-19 19:53:18 -08:00
ip6_checksum.h
net: fix build errors if ipv6 is disabled
2013-10-09 13:04:03 -04:00
ip6_fib.h
ipv6: remove prune parameter for fib6_clean_all
2014-01-02 03:30:35 -05:00
ip6_route.h
IPv6: add the option to use anycast addresses as source addresses in echo reply
2014-01-07 15:51:39 -05:00
ip6_tunnel.h
net: unify the pcpu_tstats and br_cpu_netstats as one
2014-01-04 20:10:24 -05:00
ip_fib.h
ip*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
ip_tunnels.h
ipv4: fix a dst leak in tunnels
2014-01-17 18:36:39 -08:00
ip_vs.h
netfilter: push reasm skb through instead of original frag skbs
2013-11-11 00:19:35 -05:00
ipcomp.h
ipconfig.h
ipv6.h
ipv6: protect protocols not handling ipv4 from v4 connection/bind attempts
2014-01-21 16:59:19 -08:00
ipx.h
ipx.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
iw_handler.h
iw_handler.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
lapb.h
lapb.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
lib80211.h
hostap: Don't use create_proc_read_entry()
2013-04-29 15:41:56 -04:00
llc.h
llc: make lock static
2014-01-03 20:56:48 -05:00
llc_c_ac.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc_c_ev.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc_c_st.h
llc_conn.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc_if.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc_pdu.h
net: llc: fix order of evaluation in llc_conn_ac_inc_vr_by_1
2014-01-01 22:22:43 -05:00
llc_s_ac.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc_s_ev.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
llc_s_st.h
llc_sap.h
llc*.h: Remove extern from function prototypes
2013-09-21 14:01:38 -04:00
mac80211.h
mac80211: handle MMPDUs at EOSP correctly
2014-01-10 09:50:02 +01:00
mac802154.h
mac802154: correct a typo in ieee802154_alloc_device() prototype
2013-10-21 18:56:23 -04:00
mip6.h
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
mld.h
net: ipv6: mld: get rid of MLDV2_MRC and simplify calculation
2013-09-04 14:53:20 -04:00
mrp.h
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2013-10-01 17:06:14 -04:00
ndisc.h
ndisc.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
neighbour.h
neigh: use NEIGH_VAR_INIT in ndo_neigh_setup functions.
2014-01-16 11:31:58 -08:00
net_namespace.h
netfilter: nf_tables: complete net namespace support
2013-10-14 18:00:59 +02:00
net_ratelimit.h
netdma.h
netevent.h
netevent/netlink.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
netlabel.h
include/net/: Fix FSF address in file headers
2013-12-06 12:37:56 -05:00
netlink.h
netevent/netlink.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
netprio_cgroup.h
net: netprio: rename config to be more consistent with cgroup configs
2014-01-03 23:41:42 +01:00
netrom.h
netrom.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
nexthop.h
nl802154.h
p8022.h
p8022.h: Remove extern from function prototypes
2013-09-21 14:01:39 -04:00
ping.h
ipv6: make IPV6_RECVPKTINFO work for ipv4 datagrams
2014-01-19 19:53:18 -08:00
pkt_cls.h
net_sched: optimize tcf_match_indev()
2014-01-13 11:50:15 -08:00
pkt_sched.h
pkt_sched: give visibility to mq slave qdiscs
2013-12-09 19:54:47 -05:00
protocol.h
net: Add GRO support for UDP encapsulating protocols
2014-01-21 18:05:04 -08:00
psnap.h
psnap.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
raw.h
raw/rawv6.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
rawv6.h
raw/rawv6.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
red.h
reciprocal_divide: update/correction of the algorithm
2014-01-21 23:17:20 -08:00
regulatory.h
cfg80211: make regulatory_hint() remove REGULATORY_CUSTOM_REG
2014-01-13 14:46:58 -05:00
request_sock.h
inet: includes a sock_common in request_sock
2013-10-10 00:08:07 -04:00
rose.h
rose.h: Remove extern from function prototypes
2013-09-23 01:51:08 -04:00
route.h
ipv4: introduce ip_dst_mtu_maybe_forward and protect forwarding path against pmtu spoofing
2014-01-13 11:22:54 -08:00
rtnetlink.h
netlink: cleanup rntl_af_register
2014-01-01 23:42:19 -05:00
sch_generic.h
net_sched: add struct net pointer to tcf_proto_ops->dump
2014-01-13 11:50:14 -08:00
scm.h
scm.h: Remove extern from function prototypes
2013-09-23 01:51:09 -04:00
secure_seq.h
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
2013-10-01 17:06:14 -04:00
slhc_vj.h
snmp.h
sock.h
Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next
2014-01-05 20:18:50 -05:00
Space.h
drivers: net: Include new header file in sbni.c
2013-12-19 18:51:20 -05:00
stp.h
stp.h: Remove extern from function prototypes
2013-09-23 01:51:09 -04:00
tcp.h
tcp: make local functions static
2013-12-29 16:34:24 -05:00
tcp_memcontrol.h
tcp_memcontrol: Kill struct tcp_memcontrol
2013-10-21 18:43:02 -04:00
tcp_states.h
timewait_sock.h
transp_v6.h
ipv6: make IPV6_RECVPKTINFO work for ipv4 datagrams
2014-01-19 19:53:18 -08:00
udp.h
udp: Remove unnecessary semicolon from do{}while (0) macro
2013-11-07 02:14:33 -05:00
udplite.h
udplite.h: Remove extern from function prototypes
2013-09-23 16:29:40 -04:00
vsock_addr.h
VSOCK: Move af_vsock.h and vsock_addr.h to include/net
2013-07-27 22:14:06 -07:00
vxlan.h
net: Add GRO support for vxlan traffic
2014-01-21 18:05:04 -08:00
wext.h
wext.h: Remove extern from function prototypes
2013-09-23 16:29:40 -04:00
wimax.h
wimax.h: Remove extern from function prototypes
2013-09-23 16:29:41 -04:00
wpan-phy.h
x25.h
x25.h: Remove extern from function prototypes
2013-09-23 16:29:41 -04:00
x25device.h
xfrm.h
{pktgen, xfrm} Introduce xfrm_state_lookup_byspi for pktgen
2014-01-03 07:29:12 +01:00