From 699841d492ca0229b652efc977be695a1831687d Mon Sep 17 00:00:00 2001 From: Vadim Solomin Date: Tue, 4 Dec 2018 23:41:20 +0300 Subject: [PATCH 1/2] net: aquantia: Atlantic forwarding driver support Squash all the atlantic forwarding driver changes. commit ea48303fcc27f56438c4585aac06bcdbf1996cd2 ("atlantic-forwarding driver v1.1.11") commit d5b0f4c48788abb3bb11dd1b3e70db111a20390c ("atlantic-forwarding driver v1.1.10") commit 67474e4efc0d0a70286b4c3def3d39e16e344406 ("atlantic-forwarding driver v1.1.9") commit c6141c29b8b25fd345c7e5b0dd3e186735258000 ("atlantic-forwarding driver v1.1.8") commit 683002bd1eefbaf23af264878f3409e79ba0b522 ("atlantic-forwarding driver v1.1.7") commit dc69fcd7640bb6727d886b69db0d49ae60d00cf1 ("atlantic-forwarding driver v1.1.6") commit 274a6d9d013241c4a8d7cc89c270e0458735362d ("atlantic-forwarding driver v1.1.5") commit 5be5557abd1f19f510d4b32df46bf196f68e8a4e ("atlantic-forwarding driver v1.1.4") commit 24b77eaf64ca1399208f8e8666410821992a171a ("atlantic-forwarding driver v1.1.3") commit 714828615d0657982fbde6fa8503008a53554147 ("atlantic-forwarding driver v1.1.2") commit 7edd86bf7b8a734372f11cdec7c11bb2411c2b8c ("atlantic-forwarding driver v1.1.1") commit c4882a2abafe947a4420819831f4c2d14ebadbc5 ("atlantic-forwarding driver v1.1.0") commit 0f9c9e19e318bbb2180273c80f513f7128a9caf4 ("atlantic-forwarding driver v1.0.31") commit ffcc939dc8c3b0ee4792070929365ac4d1723583 ("atlantic-forwarding driver v1.0.30") commit 674c4e44c501eed10a57e6f52e58c749f07a4639 ("atlantic-forwarding driver v1.0.29") commit ef58204f631fc46cd096d3be916dc383e5c8baec ("atlantic-forwarding driver v1.0.28") commit 76593bccda07e3dca7093462f5eee4b79ddb7981 ("atlantic-forwarding driver v1.0.27") commit e0e81b671f50044633de0738a8f2975466e2da45 ("atlantic-forwarding driver v1.0.26") commit 6a8d1d677a69c550a9eccd03a841f187f0d42dd4 ("atlantic-forwarding driver v1.0.25") commit 1d997ae0cb919b3f57dde7efcbf971f22761c2fe ("atlantic-forwarding driver v1.0.24") commit 233592c1f59eccb87b1609f342826de383fde073 ("atlantic-forwarding driver v1.0.23") commit c0260b607c91b5d10be3d939c9adaa48f8ea162d ("atlantic-forwarding driver v1.0.22") commit 27608814e0000857b338f5f0f43942db5171fe08 ("atlantic-forwarding driver v1.0.21") commit 216c6111a776632e67bf9f7e013b720120521d9d ("atlantic-forwarding driver v1.0.20") commit 4227f74b8af94e5457c0efc165be03429c67437c ("atlantic-forwarding driver v1.0.19") commit 834fb1c3851d5316e94a908767e81de341ca3f3f ("atlantic-forwarding driver v1.0.18") commit 1bae566ed23376e38d84af02939e9f39cb5ba47e ("atlantic-forwarding driver v1.0.17") commit 5b7bb561633dfec8a9a52cbb35c202f40c474cf4 ("atlantic-forwarding driver v1.0.16") commit abd3119fadbecba1de363c75aaaf958ec99dd2b9 ("atlantic-forwarding driver v1.0.15") Change-Id: I3d002d33ad8734bd99629e24e311aa852b074f1c Signed-off-by: Dmitry Bogdanov Signed-off-by: Mark Starovoytov Signed-off-by: Nikita Danilov Signed-off-by: Vadim Solomin Signed-off-by: Igor Russkikh Signed-off-by: Jinesh K. Jayakumar Git-commit: ea48303fcc27f56438c4585aac06bcdbf1996cd2 Git-repo: https://github.com/aquantia/linux-4.14-atlantic-forwarding Signed-off-by: Lakshit Tyagi --- drivers/net/ethernet/aquantia/Kconfig | 8 + drivers/net/ethernet/aquantia/Makefile | 1 + .../Documentation/networking/atlantic.txt | 283 ++ .../ethernet/aquantia/atlantic-fwd/Kconfig | 52 + .../ethernet/aquantia/atlantic-fwd/Makefile | 30 + .../ethernet/aquantia/atlantic-fwd/atl2_fw.c | 968 +++++ .../ethernet/aquantia/atlantic-fwd/atl2_fw.h | 690 ++++ .../aquantia/atlantic-fwd/atl_common.h | 446 +++ .../aquantia/atlantic-fwd/atl_compat.c | 139 + .../aquantia/atlantic-fwd/atl_compat.h | 495 +++ .../ethernet/aquantia/atlantic-fwd/atl_desc.h | 159 + .../aquantia/atlantic-fwd/atl_drviface.h | 421 +++ .../aquantia/atlantic-fwd/atl_ethtool.c | 3108 +++++++++++++++++ .../aquantia/atlantic-fwd/atl_ethtool.h | 25 + .../ethernet/aquantia/atlantic-fwd/atl_fw.c | 1215 +++++++ .../ethernet/aquantia/atlantic-fwd/atl_fw.h | 304 ++ .../ethernet/aquantia/atlantic-fwd/atl_fwd.c | 905 +++++ .../ethernet/aquantia/atlantic-fwd/atl_fwd.h | 408 +++ .../aquantia/atlantic-fwd/atl_fwdnl.c | 1815 ++++++++++ .../aquantia/atlantic-fwd/atl_fwdnl.h | 122 + .../aquantia/atlantic-fwd/atl_fwdnl_params.c | 150 + .../aquantia/atlantic-fwd/atl_fwdnl_params.h | 21 + .../ethernet/aquantia/atlantic-fwd/atl_hw.c | 1602 +++++++++ .../ethernet/aquantia/atlantic-fwd/atl_hw.h | 382 ++ .../aquantia/atlantic-fwd/atl_hw_ptp.c | 247 ++ .../aquantia/atlantic-fwd/atl_hw_ptp.h | 31 + .../aquantia/atlantic-fwd/atl_hwmon.c | 214 ++ .../ethernet/aquantia/atlantic-fwd/atl_log.h | 77 + .../aquantia/atlantic-fwd/atl_macsec.c | 1738 +++++++++ .../aquantia/atlantic-fwd/atl_macsec.h | 139 + .../ethernet/aquantia/atlantic-fwd/atl_main.c | 1073 ++++++ .../ethernet/aquantia/atlantic-fwd/atl_mdio.h | 28 + .../ethernet/aquantia/atlantic-fwd/atl_ptp.c | 1328 +++++++ .../ethernet/aquantia/atlantic-fwd/atl_ptp.h | 65 + .../ethernet/aquantia/atlantic-fwd/atl_regs.h | 224 ++ .../ethernet/aquantia/atlantic-fwd/atl_ring.c | 2199 ++++++++++++ .../ethernet/aquantia/atlantic-fwd/atl_ring.h | 235 ++ .../aquantia/atlantic-fwd/atl_ring_desc.h | 55 + .../aquantia/atlantic-fwd/atl_stats.h | 87 + .../aquantia/atlantic-fwd/atl_trace.c | 16 + .../aquantia/atlantic-fwd/atl_trace.h | 274 ++ .../aquantia/atlantic-fwd/atlfwdtool/Makefile | 36 + .../atlantic-fwd/atlfwdtool/atlfwd_args.c | 222 ++ .../atlantic-fwd/atlfwdtool/atlfwd_args.h | 38 + .../atlantic-fwd/atlfwdtool/atlfwd_ctx.h | 33 + .../atlantic-fwd/atlfwdtool/atlfwd_msg.c | 72 + .../atlantic-fwd/atlfwdtool/atlfwd_msg.h | 22 + .../atlantic-fwd/atlfwdtool/atlfwd_reply.c | 82 + .../atlantic-fwd/atlfwdtool/atlfwd_reply.h | 20 + .../atlantic-fwd/atlfwdtool/atlfwdtool.c | 418 +++ .../macsec/MSS_Egress_registers.h | 79 + .../macsec/MSS_Ingress_registers.h | 84 + .../aquantia/atlantic-fwd/macsec/macsec_api.c | 2976 ++++++++++++++++ .../aquantia/atlantic-fwd/macsec/macsec_api.h | 334 ++ .../atlantic-fwd/macsec/macsec_struct.h | 920 +++++ .../aquantia/atlantic-fwd/release_notes.txt | 352 ++ 56 files changed, 27467 insertions(+) create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/Documentation/networking/atlantic.txt create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/Kconfig create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/Makefile create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_common.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_desc.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_drviface.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_hwmon.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_log.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_main.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_mdio.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_regs.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring_desc.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_stats.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/Makefile create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_ctx.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwdtool.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Egress_registers.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Ingress_registers.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.c create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_struct.h create mode 100644 drivers/net/ethernet/aquantia/atlantic-fwd/release_notes.txt diff --git a/drivers/net/ethernet/aquantia/Kconfig b/drivers/net/ethernet/aquantia/Kconfig index 350a48e4f124..32f1e8dc82c4 100644 --- a/drivers/net/ethernet/aquantia/Kconfig +++ b/drivers/net/ethernet/aquantia/Kconfig @@ -23,4 +23,12 @@ config AQTION ---help--- This enables the support for the aQuantia AQtion(tm) Ethernet card. +config AQFWD + tristate "aQuantia Forwarding driver" + depends on PCI && (X86_64 || ARM64) + ---help--- + This enables the support for forwarding driver for the aQuantia AQtion(tm) Ethernet card. + +source "drivers/net/ethernet/aquantia/atlantic-fwd/Kconfig" + endif # NET_VENDOR_AQUANTIA diff --git a/drivers/net/ethernet/aquantia/Makefile b/drivers/net/ethernet/aquantia/Makefile index c4e7d01ea650..363339bffe9b 100644 --- a/drivers/net/ethernet/aquantia/Makefile +++ b/drivers/net/ethernet/aquantia/Makefile @@ -4,3 +4,4 @@ # obj-$(CONFIG_AQTION) += atlantic/ +obj-$(CONFIG_AQFWD) += atlantic-fwd/ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/Documentation/networking/atlantic.txt b/drivers/net/ethernet/aquantia/atlantic-fwd/Documentation/networking/atlantic.txt new file mode 100644 index 000000000000..5956851c04ed --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/Documentation/networking/atlantic.txt @@ -0,0 +1,283 @@ + + + +Supported hardware +------------------ + +AQC-107 / -108 / -109, revisions B0 / B1. FW versions 1.x / 2.x / 3.x +supported + + + +Features / acceleration +----------------------- + + - Message-signalled interrupts + - Hardware VLAN tag offloads + - Hardware checksum offloads: + - IPv4 header checksums + - UDP / TCP checksums with IPv4 and IPv6 + - Hardware LSO for TCP v4/v6 + - Hardware LRO for TCP v4/v6 + - Coalescing TCP segments with TCP options not supported due to a + HW limitation + - Software GRO using the Linux NAPI GRO interface + - Hardware Rx header splitting + - RSS over up to 8 queues using MSI-X + - Linear skb receive mode + - N-tuple Rx filters + - Runtime power management + - Hardware MACSEC offload + + +Building and installation +------------------------- + +1. Unpack the archive into a directory of your choice. + +2. Change to that directory and compile the module: + $ make [KDIR=] + + Explicitly specifying KDIR is only necessary when the headers can't + be found automatically via the + /lib/modules/`uname -r`/{build,source} symlinks + +3. Load the module + # insmod atlnew.ko [] + +4. Unload the module + # rmmod atlnew + + + +Device configuration +-------------------- + + # ip link set up dev + # ip addr add / brd + dev + +Add a VLAN ID: + # ip link add link name . type vlan id + # ip link set up dev . + +Remove a VLAN ID: + # ip link del dev . + + + +Module parameters +----------------- + +Module parameters may be specified at module load time on the insmod +command line or via modprobe.conf files. Some parameters marked as +[RW] below can also be changed at run time via files in +/sys/module/atlnew/parameters directory + + - msi = < 1 | 0 > (default: 1) [RO] + Enable use of MSI interrupts. + + - rx_refill_batch = (default: 16) [RW] + Number of cleaned Rx ring descriptors to trigger Rx refill + + - tx_clean_budget = (default: 256) [RW] + Max number of processed Tx descriptors to clean in one pass of + NAPI poll + + - max_queues = (default: 8) [RO] + Max number of RSS queues. The actual number of queues will the + minimum of this parameter and the number of logical CPUs in + the system. + + - rx_linear = <0 | 1> (default: 0) + Enables linear skb receive mode. Frames larger than + ATL_RX_BUF_SIZE defined in atl_ring.h will be dropped. + + - min_intr_delay (default: 10) [RW] + When an interrupt is about to be asserted on an LRO-enabled + queue, LRO requires some lead time to close and flush + in-progress flows. This parameter sets the LRO interrupt delay + and provides the lower boundary for interrupt coalescing + delays settable via ethtool. When modified, the new value + takes effect on interface down/up cycle or on setting an + interrupt coalescing delay via ethtool. + + - rx_mod_hyst (default: 10) [RW] + - tx_mod_hyst (default: 10) [RW] + Each ring has two interrupt coalescing timers. When an + interrupt-causing event happens, they're initialized with + values called minimum and maximum respectively. The former + timer is restarted each time another such event happens, while + the latter keeps running. When either timer reaches zero, the + interrupt is reported to the host. The minimum value is set + via ethtool. These module parameters set the difference + between the maximum and minimum values for Rx and Tx + paths. When modified, the new max values go into effect upon + interface down/up cycle or upon modifying ethtool interrupt + coalescing settings. + +- keep_link = < 1 | 0 > (default: 0) [RO] + When set to 1, the ethernet link is established at PCI probe + time, and is not dropped on ifdown. This allows the rings + allocated to forwarding engines to handle traffic even when + linux network interface is down. + +- sleep_delay = (default: 10000) [RW] + Delay in ms after link goes down to suspend the device if runtime power + management is turned on. + +- macsec_bridge = < 1 | 0 > (default: 1) [RO] + When set to 0, several SecY can be created, macsec will filter by source mac + address to apply apropriate SecY. + By default, only 1 SecY can be created, macsec will handle all packets. + +- rx/tx_ring_size = <8..8184> (default: 4096) [RO] + Default size for the rings. + +VLAN filters +------------ + +Sixteen VLAN filters are programmable via the ethtool -N interface using +locations 0 through 15. The only flow-type supported is ether and only +VLAN id can be matched. + +For VLAN filters only, it's possible not to specify an explicit +filter location. In this case an appropriate location will be assigned +automatically. + +These filters are also used for HW filter acceleration for Linux VLAN +subdevices. Filters automatically added by Linux VLAN code can be +overridden using ethtool subject to restrictions described below: + +* A special ring number 32 is used in ethtool interface to signify + 'any ring'. An ethtool filter rule with ring set to 32 accepts a + packet leaving the ring to be determined using the standard RSS + mechanism. VLAN code adds rules in this manner. Such a rule can be + modified using ethool to set a specific destination ring. + +* There can't be two filters with the same VID. An attempt to add a + duplicate filter using an explicit location different from the + location of existing filter with the same VID will fail. When the + location is not specified, the existing filter will be modified. + +* An attempt to change the VID of a filter added by VLAN code using an + explicit location will fail. Either use a different explicit + location or no location. + +* An attempt to delete a rule created by VLAN code will return + success, but the rule will remain in place. If a specific ring was + set in the rule, it will be reset to 'any ring'. + +* If a VLAN sub-device is added when there're no unused filters, VLAN + promiscuous mode will be enabled. This will result in all VIDs being + accepted. Filters can still be used to direct specific VIDs to + specific rings. + +* Even when all 16 filters are used, attempting to add a rule with a + new VID via ethtool will succeed as long as at least one existing + filter was set by VLAN code and still has 'any ring' as + destination. This filter will be re-used for the new rule and VLAN + promiscuous mode will be enabled. + +* When a filter becomes available as a result of a deletion of a VLAN + subdevice or a deletion of an ethtool filter rule, and a VLAN + promiscuous mode is enabled, the vacated filter will be re-used for + another enabled VLAN subdevice. This may result in VLAN promiscuous + mode being turned off if that VLAN subdevice was the last one with + no filter allocated. + +Ethertype filters +----------------- + +Fifteen ethertype filters are programmable via the ethtool -N +interface using locations 16 through 30. The only flow-type supported +is ether and only the ethertype field can be matched. + +N-tuple filters +--------------- + +Eight IPv4 n-tuple filters are supported, programmable using ethtool +-N command with filter locations 32 through 39. + +Fields that can be matched are source / destination IP addresses, and, +for TCP / UDP / SCTP protocols, additionally source / destination +ports can be matched. Partial (masked) matches are not supported, a +mask for each field must consist of either all ones or all zeroes. + +Each aligned group of four filters (locations 32 through 35 or 36 +through 39) can alternatively be used as one IPv6 filter, using filter +locations 32 or 36. + + + +Testing forwarding performance +------------------------------ + +Since this is a design for a router, the best way to test would be +between our aquantia port and another high speed port on the same +DUT. However, in the situation where the Aquantia interface is the +only high speed one, we have devised the following testing strategy +to emulate these two ports on same DUT. + +One of the simplest ways is to set up two VLAN interfaces and forward +traffic between them as in the following example: + +Set up two VLANs with examlple ids 42 and 43 on the Aquantia +interface, assign the switch port connected to it to both of those +VLANs. +# ip link set up dev +# ip link add link name .42 type vlan id 42 +# ip link add link name .43 type vlan id 43 +# ip link set up dev .42 +# ip link set up dev .43 +# ip addr add 10.42.42.1/24 brd + dev .42 +# ip addr add 10.42.43.1/24 brd + dev .43 +# sysctl net.ipv4.ip_forward=1 + +This sets up two VLAN interfaces with VLAN ids 42 and 43 and enables +IP routing. Two hosts should be set up to generate and terminate the +traffic: + +On a host on the 42 VLAN with IP 10.42.42.xx run + $ iperf3 -s +On a host on the 43 VLAN run + $ iperf3 -c 10.42.42.xx + +Another possibility is to use a single host running Linux as a link +partner both generating and terminating the traffic. Two VLAN +sub-interfaces should be configured on a high-speed port on that host, +belonging to VLAN 42 / IP network 10.42.42/24 and VLAN 43 / IP network +10.42.43/24 respectively. + +Since, by default, traffic addressed to one of the local IPs is +received locally, a special configuration of Linux policy routing is +required to make such traffic actually go out on the wire to the DUT, +as described below, where stands for the chosen network interface: + +# ip rule add table local prio 100 +# ip rule del table local prio 0 +# ip rule add iif lo table 42 prio 42 +# ip link set up dev + +# ip link add link name .42 type vlan id 42 +# ip link add link name .43 type vlan id 43 +# ip link set up dev .42 +# ip link set up dev .43 +# ip addr add 10.42.42.10/24 brd + dev .42 +# ip addr add 10.42.43.10/24 brd + dev .43 + +# ip route add 10.42.42.10 via 10.42.43.1 table 42 +# ip route add 10.42.43.10 via 10.42.42.1 table 42 + +This routes the locally-originated traffic, that is addressed to the local IP +of one of the VLAN interfaces, out via the other VLAN interface, instead of +being immediately looped back locally. + + +$ iperf3 -s + +$ iperf3 -c 10.42.42.10 + +The following command can be used to restore the default routing +behavior: + +# ip route flush table 42 diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/Kconfig b/drivers/net/ethernet/aquantia/atlantic-fwd/Kconfig new file mode 100644 index 000000000000..d124a01865c0 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/Kconfig @@ -0,0 +1,52 @@ +# +# Aquantia atlantic-forwarding driver config +# + +if AQFWD + +config ATLFWD_FWD + bool "Enable forwarding-engine API" + default n + help + + Say Y to enable the forwarding-engine API + +config ATLFWD_FWD_RXBUF + int + prompt "Rx buffer space for forwarding engine's rings" if ATLFWD_FWD + range 0 320 + default 160 if ATLFWD_FWD + default 0 + help + + Amount of Rx buffer to reserve for the forwarding-engine + rings. This sets the default value of the fwd_rx_buf_reserve + module option. + + Value in kiB, 0 to 320, defaults to 160 if forwarding-engine + API enabled, 0 otherwise. + +config ATLFWD_FWD_TXBUF + int + prompt "Tx buffer space for forwarding engine's rings" if ATLFWD_FWD + range 0 160 + default 80 if ATLFWD_FWD + default 0 + help + + Amount of Tx buffer to reserve for the forwarding-engine + rings. This sets the default value of the fwd_tx_buf_reserve + module option. + + Value in kiB, 0 to 320, defaults to 80 if forwarding-engine + API enabled, 0 otherwise. + +config ATLFWD_FWD_NETLINK + bool "Enable netlink control socket for forwarding engine" + default n + depends on ATLFWD_FWD + help + + Expose forwarding engine APIs over the netlink socket. + +endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/Makefile b/drivers/net/ethernet/aquantia/atlantic-fwd/Makefile new file mode 100644 index 000000000000..e8e82bcd95a2 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/Makefile @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Atlantic Network Driver +# +# Copyright(c) 2014-2019 aQuantia Corporation. +# Copyright(c) 2019-2020 Marvell International Ltd. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. + +obj-$(CONFIG_AQFWD) += atlantic-fwd.o + +atlantic-fwd-objs := atl_fw.o \ + atl2_fw.o \ + atl_hw.o \ + atl_main.o \ + atl_ring.o \ + atl_ethtool.o \ + atl_trace.o \ + atl_compat.o \ + atl_hwmon.o \ + atl_ptp.o \ + atl_hw_ptp.o + +atlantic-fwd-$(CONFIG_ATLFWD_FWD) += atl_fwd.o +atlantic-fwd-$(CONFIG_ATLFWD_FWD_NETLINK) += atl_fwdnl.o \ + atl_fwdnl_params.o +atlantic-fwd-$(CONFIG_MACSEC) += atl_macsec.o macsec/macsec_api.o + +ccflags-y += -I$(src) diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.c new file mode 100644 index 000000000000..a5f0ca04bb09 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.c @@ -0,0 +1,968 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include + +#include "atl_common.h" +#include "atl_hw.h" +#include "atl2_fw.h" +#include "atl_fw.h" + +#define ATL2_FW_READ_TRY_MAX 1000 + +#define atl2_shared_buffer_write(HW, ITEM, VARIABLE) \ + atl2_mif_shared_buf_write(HW,\ + (offsetof(struct fw_interface_in, ITEM) / sizeof(u32)),\ + (u32 *)&VARIABLE, sizeof(VARIABLE) / sizeof(u32)) + +#define atl2_shared_buffer_get(HW, ITEM, VARIABLE) \ + atl2_mif_shared_buf_get(HW, \ + (offsetof(struct fw_interface_in, ITEM) / sizeof(u32)),\ + (u32 *)&VARIABLE, \ + sizeof(VARIABLE) / sizeof(u32)) + +/* This should never be used on non atomic fields, + * treat any > u32 read as non atomic. + */ +#define atl2_shared_buffer_read(HW, ITEM, VARIABLE) \ +{\ + BUILD_BUG_ON_MSG((offsetof(struct fw_interface_out, ITEM) % \ + sizeof(u32)) != 0,\ + "Non aligned read " # ITEM);\ + BUILD_BUG_ON_MSG(sizeof(VARIABLE) > sizeof(u32),\ + "Non atomic read " # ITEM);\ + atl2_mif_shared_buf_read(HW, \ + (offsetof(struct fw_interface_out, ITEM) / sizeof(u32)),\ + (u32 *)&VARIABLE, \ + sizeof(VARIABLE) / sizeof(u32));\ +} + +static void atl2_mif_shared_buf_get(struct atl_hw *hw, int offset, + u32 *data, int len) +{ + int i; + + for (i = 0; i < len; i++) + data[i] = atl_read(hw, ATL2_MIF_SHARED_BUFFER_IN(offset + i)); +} + +static void atl2_mif_shared_boot_buf_write(struct atl_hw *hw, int offset, + u32 *data, int len) +{ + int i; + + for (i = 0; i < len / sizeof(u32); i++) + atl_write(hw, ATL2_MIF_SHARED_BUFFER_BOOT(i), data[i]); +} + +static void atl2_mif_shared_buf_write(struct atl_hw *hw, int offset, + u32 *data, int len) +{ + int i; + + for (i = 0; i < len; i++) + atl_write(hw, ATL2_MIF_SHARED_BUFFER_IN(offset + i), + data[i]); +} + +static void atl2_mif_shared_buf_read(struct atl_hw *hw, int offset, + u32 *data, int len) +{ + int i; + + for (i = 0; i < len; i++) + data[i] = atl_read(hw, ATL2_MIF_SHARED_BUFFER_OUT(offset + i)); +} + +static void atl2_mif_host_finished_write_set(struct atl_hw *hw, u32 finish) +{ + atl_write_bits(hw, ATL2_MIF_HOST_FINISHED_WRITE, 0, 1, finish); +} + +static u32 atl2_mif_mcp_finished_read_get(struct atl_hw *hw) +{ + return atl_read(hw, ATL2_MIF_MCP_FINISHED_READ) & 1; +} + +#define atl2_shared_buffer_read_safe(HW, ITEM, DATA) \ + _atl2_shared_buffer_read_safe((HW), \ + (offsetof(struct fw_interface_out, ITEM) / sizeof(u32)),\ + sizeof(((struct fw_interface_out *)0)->ITEM) / sizeof(u32),\ + (DATA)) + +/* There are two 16bit transaction counters. The one is incremented at start of + * changing non atomic value, the other one - at the end. So we need to wait for + * ma moment when they are equal each other at the reading non atomic memory and + * at the end of read. + */ +static int _atl2_shared_buffer_read_safe(struct atl_hw *hw, + uint32_t offset, uint32_t dwords, + void *data) +{ + struct transaction_counter_s tid1, tid2; + int cnt = 0; + + do { + do { + atl2_shared_buffer_read(hw, transaction_id, tid1); + cnt++; + if (cnt > ATL2_FW_READ_TRY_MAX) + return -ETIME; + if (tid1.transaction_cnt_a != tid1.transaction_cnt_b) + udelay(1); + } while (tid1.transaction_cnt_a != tid1.transaction_cnt_b); + + atl2_mif_shared_buf_read(hw, offset, (u32 *)data, dwords); + + atl2_shared_buffer_read(hw, transaction_id, tid2); + + cnt++; + //TODO: no much sense to handle these errors + // therefore put WARN here to make it visible + if (cnt > ATL2_FW_READ_TRY_MAX) + return -ETIME; + } while (tid2.transaction_cnt_a != tid2.transaction_cnt_b || + tid1.transaction_cnt_a != tid2.transaction_cnt_a); + + return 0; +} + +static inline int atl2_shared_buffer_finish_ack(struct atl_hw *hw) +{ + u32 val; + int err = 0; + + atl2_mif_host_finished_write_set(hw, 1U); + + busy_wait(1000, udelay(100), val, + atl2_mif_mcp_finished_read_get(hw), + val != 0); + + if (val != 0) + err = -ETIME; + WARN(err, "atl2_shared_buffer_finish_ack"); + + return err; +} + +static int atl2_fw_get_filter_caps(struct atl_hw *hw) +{ + struct atl_nic *nic = container_of(hw, struct atl_nic, hw); + struct filter_caps_s filter_caps; + u32 tag_top; + int err; + + err = atl2_shared_buffer_read_safe(hw, filter_caps, &filter_caps); + if (err) + return err; + + hw->art_base_index = filter_caps.rslv_tbl_base_index * 8; + hw->art_available = filter_caps.rslv_tbl_count * 8; + if (hw->art_available == 0) + hw->art_available = 128; + nic->rxf_flex.available = 1; + nic->rxf_flex.base_index = filter_caps.flexible_filter_mask >> 1; + nic->rxf_mac.base_index = filter_caps.l2_filters_base_index; + nic->rxf_mac.available = filter_caps.l2_filter_count; + nic->rxf_etype.base_index = filter_caps.ethertype_filter_base_index; + nic->rxf_etype.available = filter_caps.ethertype_filter_count; + nic->rxf_etype.tag_top = + (nic->rxf_etype.available >= ATL2_RPF_ETYPE_TAGS) ? + (ATL2_RPF_ETYPE_TAGS) : (ATL2_RPF_ETYPE_TAGS >> 1); + nic->rxf_vlan.base_index = filter_caps.vlan_filter_base_index; + /* 0 - no tag, 1 - reserved for vlan-filter-offload filters */ + tag_top = (filter_caps.vlan_filter_count == ATL_VLAN_FLT_NUM) ? + (ATL_VLAN_FLT_NUM - 2) : + (ATL_VLAN_FLT_NUM / 2 - 2); + nic->rxf_vlan.available = min_t(u32, filter_caps.vlan_filter_count - 2, + tag_top); + nic->rxf_ntuple.l3_v4_base_index = filter_caps.l3_ip4_filter_base_index; + nic->rxf_ntuple.l3_v4_available = min_t(u32, + filter_caps.l3_ip4_filter_count, + ATL_NTUPLE_FLT_NUM - 1); + nic->rxf_ntuple.l3_v6_base_index = filter_caps.l3_ip6_filter_base_index; + nic->rxf_ntuple.l3_v6_available = filter_caps.l3_ip6_filter_count; + nic->rxf_ntuple.l4_base_index = filter_caps.l4_filter_base_index; + nic->rxf_ntuple.l4_available = min_t(u32, filter_caps.l4_filter_count, + ATL_NTUPLE_FLT_NUM - 1); + + return 0; +} + +static int __atl2_fw_wait_init(struct atl_hw *hw) +{ + struct request_policy_s request_policy; + struct link_control_s link_control; + uint32_t mtu; + int err; + + BUILD_BUG_ON_MSG(sizeof(struct link_options_s) != 0x4, + "linkOptions invalid size"); + BUILD_BUG_ON_MSG(sizeof(struct thermal_shutdown_s) != 0x4, + "thermalShutdown invalid size"); + BUILD_BUG_ON_MSG(sizeof(struct sleep_proxy_s) != 0x958, + "sleepProxy invalid size"); + BUILD_BUG_ON_MSG(sizeof(struct pause_quanta_s) != 0x18, + "pauseQuanta invalid size"); + BUILD_BUG_ON_MSG(sizeof(struct cable_diag_control_s) != 0x4, + "cableDiagControl invalid size"); + BUILD_BUG_ON_MSG(sizeof(struct statistics_s) != 0x70, + "statistics_s invalid size"); + + + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, mtu) != 0, + "mtu invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, mac_address) != 0x8, + "macAddress invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, + link_control) != 0x10, + "linkControl invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, + link_options) != 0x18, + "linkOptions invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, + thermal_shutdown) != 0x20, + "thermalShutdown invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, sleep_proxy) != 0x28, + "sleepProxy invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, + pause_quanta) != 0x984, + "pauseQuanta invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_in, + cable_diag_control) != 0xA44, + "cableDiagControl invalid offset"); + + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, version) != 0x04, + "version invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, link_status) != 0x14, + "linkStatus invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + wol_status) != 0x18, + "wolStatus invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + mac_health_monitor) != 0x610, + "macHealthMonitor invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + phy_health_monitor) != 0x620, + "phyHealthMonitor invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + cable_diag_status) != 0x630, + "cableDiagStatus invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + device_link_caps) != 0x648, + "deviceLinkCaps invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + sleep_proxy_caps) != 0x650, + "sleepProxyCaps invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + lkp_link_caps) != 0x660, + "lkpLinkCaps invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, core_dump) != 0x668, + "coreDump invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, stats) != 0x700, + "stats invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, filter_caps) != 0x774, + "filter_caps invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, + management_status) != 0x78c, + "management_status invalid offset"); + BUILD_BUG_ON_MSG(offsetof(struct fw_interface_out, trace) != 0x800, + "trace invalid offset"); + + err = atl2_fw_get_filter_caps(hw); + if (err) + return err; + + atl2_shared_buffer_get(hw, link_control, link_control); + link_control.mode = ATL2_HOST_MODE_ACTIVE; + atl2_shared_buffer_write(hw, link_control, link_control); + + atl2_shared_buffer_get(hw, mtu, mtu); + mtu = ATL_MAX_MTU + ETH_FCS_LEN + ETH_HLEN; + atl2_shared_buffer_write(hw, mtu, mtu); + + atl2_shared_buffer_get(hw, request_policy, request_policy); + request_policy.bcast.accept = 1; + request_policy.bcast.queue_or_tc = 1; + request_policy.bcast.rx_queue_tc_index = 0; + request_policy.mcast.accept = 1; + request_policy.mcast.queue_or_tc = 1; + request_policy.mcast.rx_queue_tc_index = 0; + request_policy.promisc.queue_or_tc = 1; + request_policy.promisc.rx_queue_tc_index = 0; + atl2_shared_buffer_write(hw, request_policy, request_policy); + + return atl2_shared_buffer_finish_ack(hw); +} + +int atl2_fw_set_filter_policy(struct atl_hw *hw, bool promisc, bool allmulti) +{ + struct request_policy_s request_policy; + + atl2_shared_buffer_get(hw, request_policy, request_policy); + + request_policy.promisc.all = promisc; + request_policy.mcast.promisc = allmulti; + + atl2_shared_buffer_write(hw, request_policy, request_policy); + return atl2_shared_buffer_finish_ack(hw); +} + +static int atl2_fw_deinit(struct atl_hw *hw) +{ + struct link_control_s link_control; + int err = 0; + + atl2_shared_buffer_get(hw, link_control, link_control); + link_control.mode = ATL2_HOST_MODE_SHUTDOWN; + + atl2_shared_buffer_write(hw, link_control, link_control); + err = atl2_shared_buffer_finish_ack(hw); + + return err; +} + +static inline unsigned int atl_link_adv(struct atl_link_state *lstate) +{ + struct atl_hw *hw = container_of(lstate, struct atl_hw, link_state); + + if (lstate->thermal_throttled + && hw->thermal.flags & atl_thermal_throttle) + /* FW doesn't provide raw LP's advertized rates, only + * the rates adverized both by us and LP. Here we + * advertize not just the throttled_to rate, but also + * all the lower rates as well. That way if LP changes + * or dynamically starts to adverize a lower rate than + * throttled_to, we will notice that in + * atl_thermal_check() and switch to that lower + * rate there. + */ + return (lstate->advertized & ATL_EEE_MASK) | + (BIT(lstate->throttled_to + 1) - 1); + + return lstate->advertized; +} + +static bool atl2_fw_set_link_needed(struct atl_link_state *lstate) +{ + struct atl_fc_state *fc = &lstate->fc; + bool ret = false; + + if (fc->req != fc->prev_req) + ret = true; + + if (atl_link_adv(lstate) != lstate->prev_advertized) + ret = true; + + return ret; +} + +static void atl2_set_rate(struct atl_hw *hw, + struct link_options_s *link_options) +{ + unsigned int adv = atl_link_adv(&hw->link_state); + + link_options->rate_10M_hd = !!(adv & BIT(atl_link_type_idx_10m_half)); + link_options->rate_100M_hd = !!(adv & BIT(atl_link_type_idx_100m_half)); + link_options->rate_1G_hd = !!(adv & BIT(atl_link_type_idx_1g_half)); + link_options->rate_10M = !!(adv & BIT(atl_link_type_idx_10m)); + link_options->rate_100M = !!(adv & BIT(atl_link_type_idx_100m)); + link_options->rate_1G = !!(adv & BIT(atl_link_type_idx_1g)); + link_options->rate_2P5G = !!(adv & BIT(atl_link_type_idx_2p5g)); + link_options->rate_5G = !!(adv & BIT(atl_link_type_idx_5g)); + link_options->rate_10G = !!(adv & BIT(atl_link_type_idx_10g)); + link_options->rate_N5G = link_options->rate_5G; + link_options->rate_N2P5G = link_options->rate_2P5G; +} + +static void atl2_set_eee(struct atl_link_state *lstate, + struct link_options_s *link_options, bool eee_enabled) +{ + uint32_t eee_advertized = lstate->advertized >> ATL_EEE_BIT_OFFT; + + if (!eee_enabled) { + link_options->eee_100M = 0; + link_options->eee_1G = 0; + link_options->eee_2P5G = 0; + link_options->eee_5G = 0; + link_options->eee_10G = 0; + return; + } + + link_options->eee_100M = !!(eee_advertized & BIT(atl_link_type_idx_100m)); + link_options->eee_1G = !!(eee_advertized & BIT(atl_link_type_idx_1g)); + link_options->eee_2P5G = !!(eee_advertized & BIT(atl_link_type_idx_2p5g)); + link_options->eee_5G = !!(eee_advertized & BIT(atl_link_type_idx_5g)); + link_options->eee_10G = !!(eee_advertized & BIT(atl_link_type_idx_10g)); +} + +/* fw lock must be held */ +static void __atl2_fw_set_link(struct atl_hw *hw) +{ + struct atl_link_state *lstate = &hw->link_state; + struct link_options_s link_options; + int err = 0; + + atl2_shared_buffer_get(hw, link_options, link_options); + + link_options.pause_rx = link_options.pause_tx = 0; + if (lstate->fc.req & atl_fc_rx) { + link_options.pause_rx = 1; + link_options.pause_tx = 1; + } + if (lstate->fc.req & atl_fc_tx) + link_options.pause_tx ^= 1; + + link_options.link_up = 1; + if (lstate->force_off) + link_options.link_up = 0; + + atl2_set_rate(hw, &link_options); + atl2_set_eee(lstate, &link_options, lstate->eee_enabled); + + atl2_shared_buffer_write(hw, link_options, link_options); + err = atl2_shared_buffer_finish_ack(hw); + + lstate->prev_advertized = atl_link_adv(lstate); + lstate->fc.prev_req = lstate->fc.req; +} + +static void atl2_fw_set_link(struct atl_hw *hw, bool force) +{ + if (!force && !atl2_fw_set_link_needed(&hw->link_state)) + return; + + atl_lock_fw(hw); + __atl2_fw_set_link(hw); + atl_unlock_fw(hw); +} + +static u32 a2_fw_caps_to_mask(struct device_link_caps_s *link_caps) +{ + u32 supported = 0; + + if (link_caps->rate_10G) + supported |= BIT(atl_link_type_idx_10g); + if (link_caps->rate_5G) + supported |= BIT(atl_link_type_idx_5g); + if (link_caps->rate_2P5G) + supported |= BIT(atl_link_type_idx_2p5g); + if (link_caps->rate_1G) + supported |= BIT(atl_link_type_idx_1g); + if (link_caps->rate_100M) + supported |= BIT(atl_link_type_idx_100m); + if (link_caps->rate_10M) + supported |= BIT(atl_link_type_idx_10m); + if (link_caps->rate_1G_hd) + supported |= BIT(atl_link_type_idx_1g_half); + if (link_caps->rate_100M_hd) + supported |= BIT(atl_link_type_idx_100m_half); + if (link_caps->rate_10M_hd) + supported |= BIT(atl_link_type_idx_10m_half); + + if (link_caps->eee_10G) + supported |= BIT(atl_link_type_idx_10g) << ATL_EEE_BIT_OFFT; + if (link_caps->eee_5G) + supported |= BIT(atl_link_type_idx_5g) << ATL_EEE_BIT_OFFT; + if (link_caps->eee_2P5G) + supported |= BIT(atl_link_type_idx_2p5g) << ATL_EEE_BIT_OFFT; + if (link_caps->eee_1G) + supported |= BIT(atl_link_type_idx_1g) << ATL_EEE_BIT_OFFT; + if (link_caps->eee_100M) + supported |= BIT(atl_link_type_idx_100m) << ATL_EEE_BIT_OFFT; + + return supported; +} + +static u32 a2_fw_lkp_to_mask(struct lkp_link_caps_s *lkp_link_caps) +{ + u32 rate = 0; + + if (lkp_link_caps->rate_10G) + rate |= BIT(atl_link_type_idx_10g); + if (lkp_link_caps->rate_5G) + rate |= BIT(atl_link_type_idx_5g); + if (lkp_link_caps->rate_2P5G) + rate |= BIT(atl_link_type_idx_2p5g); + if (lkp_link_caps->rate_1G) + rate |= BIT(atl_link_type_idx_1g); + if (lkp_link_caps->rate_100M) + rate |= BIT(atl_link_type_idx_100m); + if (lkp_link_caps->rate_10M) + rate |= BIT(atl_link_type_idx_10m); + if (lkp_link_caps->rate_1G_hd) + rate |= BIT(atl_link_type_idx_1g_half); + if (lkp_link_caps->rate_100M_hd) + rate |= BIT(atl_link_type_idx_100m_half); + if (lkp_link_caps->rate_10M_hd) + rate |= BIT(atl_link_type_idx_10m_half); + + if (lkp_link_caps->eee_10G) + rate |= BIT(atl_link_type_idx_10g) << ATL_EEE_BIT_OFFT; + if (lkp_link_caps->eee_5G) + rate |= BIT(atl_link_type_idx_5g) << ATL_EEE_BIT_OFFT; + if (lkp_link_caps->eee_2P5G) + rate |= BIT(atl_link_type_idx_2p5g) << ATL_EEE_BIT_OFFT; + if (lkp_link_caps->eee_1G) + rate |= BIT(atl_link_type_idx_1g) << ATL_EEE_BIT_OFFT; + if (lkp_link_caps->eee_100M) + rate |= BIT(atl_link_type_idx_100m) << ATL_EEE_BIT_OFFT; + + return rate; +} + +static int __atl2_fw_update_link_status(struct atl_hw *hw) +{ + struct atl_link_state *lstate = &hw->link_state; + struct link_status_s link_status; + struct lkp_link_caps_s lkp_link_caps; + enum atl_fc_mode fc = atl_fc_none; + + atl2_shared_buffer_read(hw, link_status, link_status); + + switch (link_status.link_rate) { + case ATL2_FW_LINK_RATE_10G: + lstate->link = &atl_link_types[atl_link_type_idx_10g]; + break; + case ATL2_FW_LINK_RATE_5G: + lstate->link = &atl_link_types[atl_link_type_idx_5g]; + break; + case ATL2_FW_LINK_RATE_2G5: + lstate->link = &atl_link_types[atl_link_type_idx_2p5g]; + break; + case ATL2_FW_LINK_RATE_1G: + lstate->link = link_status.duplex ? + &atl_link_types[atl_link_type_idx_1g] : + &atl_link_types[atl_link_type_idx_1g_half]; + break; + case ATL2_FW_LINK_RATE_100M: + lstate->link = link_status.duplex ? + &atl_link_types[atl_link_type_idx_100m] : + &atl_link_types[atl_link_type_idx_100m_half]; + break; + case ATL2_FW_LINK_RATE_10M: + lstate->link = link_status.duplex ? + &atl_link_types[atl_link_type_idx_10m] : + &atl_link_types[atl_link_type_idx_10m_half]; + break; + default: + lstate->link = NULL; + } + + if (link_status.pause_rx) + fc |= atl_fc_rx; + if (link_status.pause_tx) + fc |= atl_fc_tx; + lstate->fc.cur = fc; + lstate->eee = link_status.eee; + + atl2_shared_buffer_read(hw, lkp_link_caps, lkp_link_caps); + + lstate->lp_advertized = a2_fw_lkp_to_mask(&lkp_link_caps); + + return 0; +} + +static struct atl_link_type *atl2_fw_check_link(struct atl_hw *hw) +{ + struct atl_link_type *link; + struct atl_link_state *lstate = &hw->link_state; + struct phy_health_monitor_s phy_health_monitor; + int ret = 0; + + memset(&phy_health_monitor, 0, sizeof(phy_health_monitor)); + + atl_lock_fw(hw); + + __atl2_fw_update_link_status(hw); + + ret = atl2_shared_buffer_read_safe(hw, phy_health_monitor, + &phy_health_monitor); + atl_unlock_fw(hw); + + atl_thermal_check(hw, phy_health_monitor.phy_hot_warning); + + /* Thermal check might have reset link due to throttling */ + link = lstate->link; + + return link; +} + +/* fw lock must be held */ +static int __atl2_fw_get_link_caps(struct atl_hw *hw) +{ + struct device_link_caps_s device_link_caps; + struct atl_mcp *mcp = &hw->mcp; + unsigned int supported = 0; + int ret = 0; + + + atl2_shared_buffer_read(hw, device_link_caps, device_link_caps); + + mcp->wdog_disabled = true; + + supported = a2_fw_caps_to_mask(&device_link_caps); + + hw->link_state.supported = supported; + hw->link_state.lp_lowest = fls(supported) - 1; + mcp->caps_low = atl_fw2_wake_on_link_force; + + return ret; +} + +static int atl2_fw_restart_aneg(struct atl_hw *hw) +{ + struct link_options_s link_options; + int ret = 0; + + atl2_shared_buffer_get(hw, link_options, link_options); + + link_options.link_renegotiate = 1; + + atl2_shared_buffer_write(hw, link_options, link_options); + ret = atl2_shared_buffer_finish_ack(hw); + + link_options.link_renegotiate = 0; + atl2_shared_buffer_write(hw, link_options, link_options); + + return ret; +} + +static void atl2_fw_set_default_link(struct atl_hw *hw) +{ + struct atl_link_state *lstate = &hw->link_state; + + lstate->autoneg = true; + lstate->advertized = hw->link_state.supported; + lstate->fc.req = atl_fc_full; + lstate->eee_enabled = 0; + lstate->advertized &= ~ATL_EEE_MASK; +} + +static int atl2_fw_get_phy_temperature(struct atl_hw *hw, int *temp) +{ + struct phy_health_monitor_s phy_health_monitor; + int ret = 0; + + atl_lock_fw(hw); + + ret = atl2_shared_buffer_read_safe(hw, phy_health_monitor, + &phy_health_monitor); + + *temp = (int8_t)(phy_health_monitor.phy_temperature) * 1000; + + atl_unlock_fw(hw); + + return ret; +} + +static int atl2_fw_get_mac_addr(struct atl_hw *hw, uint8_t *mac) +{ + struct mac_address_s mac_address; + int err = 0; + + atl2_shared_buffer_get(hw, mac_address, mac_address); + + ether_addr_copy(mac, (u8 *)mac_address.mac_address); + + return err; +} + +/* fw lock must be held */ +static int __atl2_fw_get_hbeat(struct atl_hw *hw, uint16_t *hbeat) +{ + struct phy_health_monitor_s phy_health_monitor; + int ret = 0; + + ret = atl2_shared_buffer_read_safe(hw, phy_health_monitor, + &phy_health_monitor); + + *hbeat = phy_health_monitor.phy_heart_beat; + + return ret; +} + +static int atl2_fw_set_phy_loopback(struct atl_nic *nic, u32 mode) +{ + struct device_link_caps_s device_link_caps; + bool on = !!(nic->priv_flags & BIT(mode)); + struct atl_hw *hw = &nic->hw; + struct link_options_s link_options; + int ret = 0; + + atl_lock_fw(hw); + + atl2_shared_buffer_read(hw, device_link_caps, device_link_caps); + atl2_shared_buffer_get(hw, link_options, link_options); + + switch (mode) { + case ATL_PF_LPB_INT_PHY: + if (!device_link_caps.internal_loopback) { + ret = -EOPNOTSUPP; + goto unlock; + } + + link_options.internal_loopback = on; + break; + case ATL_PF_LPB_EXT_PHY: + if (!device_link_caps.external_loopback) { + ret = -EOPNOTSUPP; + goto unlock; + } + link_options.external_loopback = on; + break; + } + + atl2_shared_buffer_write(hw, link_options, link_options); + ret = atl2_shared_buffer_finish_ack(hw); + +unlock: + atl_unlock_fw(hw); + + return ret; +} + +static int atl2_fw_enable_wol(struct atl_hw *hw, unsigned int wol_mode) +{ + struct link_options_s link_options; + struct link_control_s link_control; + struct wake_on_lan_s wake_on_lan; + struct mac_address_s mac_address; + int ret = 0; + + atl_lock_fw(hw); + + atl2_shared_buffer_get(hw, sleep_proxy, wake_on_lan); + + if (wol_mode & atl_fw_wake_on_link) { + wake_on_lan.wake_on_link_up = 1; + if (test_bit(ATL_ST_UP, &hw->state)) + wake_on_lan.restore_link_before_wake = 1; + } + + if (wol_mode & atl_fw_wake_on_link_rtpm) { + wake_on_lan.wake_on_link_up = 1; + } + + if (wol_mode & atl_fw_wake_on_magic) { + wake_on_lan.wake_on_magic_packet = 1; + if (test_bit(ATL_ST_UP, &hw->state)) + wake_on_lan.restore_link_before_wake = 1; + } + + ether_addr_copy(mac_address.mac_address, hw->mac_addr); + + atl2_shared_buffer_write(hw, mac_address, mac_address); + atl2_shared_buffer_write(hw, sleep_proxy, wake_on_lan); + atl2_shared_buffer_get(hw, link_control, link_control); + link_control.mode = ATL2_HOST_MODE_SLEEP_PROXY; + atl2_shared_buffer_write(hw, link_control, link_control); + + atl2_shared_buffer_get(hw, link_options, link_options); + link_options.link_up = 1; + atl2_shared_buffer_write(hw, link_options, link_options); + + ret = atl2_shared_buffer_finish_ack(hw); + + atl_unlock_fw(hw); + return ret; +} + +static int atl2_fw_update_thermal(struct atl_hw *hw) +{ + bool enable = !!(hw->thermal.flags & atl_thermal_monitor); + struct phy_health_monitor_s phy_health_monitor; + struct thermal_shutdown_s thermal_shutdown; + int ret = 0; + + memset(&phy_health_monitor, 0, sizeof(phy_health_monitor)); + + atl_lock_fw(hw); + + atl2_shared_buffer_get(hw, thermal_shutdown, thermal_shutdown); + thermal_shutdown.shutdown_enable = enable; + thermal_shutdown.shutdown_temp_threshold = hw->thermal.crit; + thermal_shutdown.warning_hot_tempThreshold = hw->thermal.high; + thermal_shutdown.warning_cold_temp_threshold = hw->thermal.low; + atl2_shared_buffer_write(hw, thermal_shutdown, thermal_shutdown); + ret = atl2_shared_buffer_finish_ack(hw); + + atl2_shared_buffer_read_safe(hw, phy_health_monitor, + &phy_health_monitor); + atl_unlock_fw(hw); + + /* Thresholds might have changed, recheck state. */ + atl_thermal_check(hw, phy_health_monitor.phy_hot_warning); + + return ret; +} + +static int atl2_fw_set_pad_stripping(struct atl_hw *hw, bool on) +{ + struct link_control_s link_control; + int err = 0; + + atl_lock_fw(hw); + + atl2_shared_buffer_get(hw, link_control, link_control); + link_control.enable_frame_padding_removal_rx = on; + atl2_shared_buffer_write(hw, link_control, link_control); + err = atl2_shared_buffer_finish_ack(hw); + + atl_unlock_fw(hw); + + return err; +} + +#define ATL2_FW_CFG_DUMP_SIZE (4 + (sizeof(struct link_options_s) * 3) + \ + (sizeof(struct link_control_s) * 3)) +static int atl2_fw_dump_cfg(struct atl_hw *hw) +{ + if (!hw->mcp.fw_cfg_dump) + hw->mcp.fw_cfg_dump = devm_kzalloc(&hw->pdev->dev, + ATL2_FW_CFG_DUMP_SIZE, + GFP_KERNEL); + if (!hw->mcp.fw_cfg_dump) + return -ENOMEM; + + /* save link configuration */ + hw->mcp.fw_cfg_dump[0] = ((2 * 3 * 4) << 16) | 3; + hw->mcp.fw_cfg_dump[1] = offsetof(struct fw_interface_in, link_control); + atl2_shared_buffer_get(hw, link_control, hw->mcp.fw_cfg_dump[2]); + hw->mcp.fw_cfg_dump[3] = ~0; + + hw->mcp.fw_cfg_dump[4] = offsetof(struct fw_interface_in, link_options); + atl2_shared_buffer_get(hw, link_options, hw->mcp.fw_cfg_dump[5]); + hw->mcp.fw_cfg_dump[6] = ~0; + + return 0; +} + +static void atl2_confirm_buffer_write(struct atl_hw *hw, + uint32_t offset, uint32_t len) +{ + struct data_buffer_status_s buffer_status; + + buffer_status.data_offset = offset; + buffer_status.data_length = len; + atl2_shared_buffer_write(hw, data_buffer_status, buffer_status); +} + +static int atl2_fw_restore_cfg(struct atl_hw *hw) +{ + uint32_t req_adr = atl_read(hw, ATL2_MIF_BOOT_READ_REQ_ADR); + uint32_t req_len = atl_read(hw, ATL2_MIF_BOOT_READ_REQ_LEN); + + if (req_adr == ATL2_ITI_ADDRESS_START) { + struct fw_iti_hdr iti_header; + + memset(&iti_header, 0, sizeof(iti_header)); + iti_header.instuction_bitmask = BIT(2); + iti_header.iti[0].type = 2; + iti_header.iti[0].length = ATL2_FW_CFG_DUMP_SIZE; + atl2_mif_shared_boot_buf_write(hw, 0, (void *)&iti_header, + sizeof(iti_header)); + atl2_confirm_buffer_write(hw, req_adr, req_len); + + atl2_mif_host_finished_write_set(hw, 1U); + + return 0; + } else if (req_adr == ATL2_ITI_ADDRESS_BLOCK_1) { + atl2_confirm_buffer_write(hw, req_adr, req_len); + atl2_mif_shared_boot_buf_write(hw, 0, + (void *)hw->mcp.fw_cfg_dump, ATL2_FW_CFG_DUMP_SIZE); + atl2_mif_host_finished_write_set(hw, 1U); + return 0; + } + + atl_dev_err("FW requests invalid address %#x", req_adr); + + return -EFAULT; +} + +static int atl2_fw_set_mediadetect(struct atl_hw *hw, bool on) +{ + struct link_options_s link_options; + + atl2_shared_buffer_get(hw, link_options, link_options); + + link_options.low_power_autoneg = on; + + atl2_shared_buffer_write(hw, link_options, link_options); + + return atl2_shared_buffer_finish_ack(hw); +} + +static int atl2_fw_unsupported(struct atl_hw *hw) +{ + return -EOPNOTSUPP; +} + +int atl2_get_fw_version(struct atl_hw *hw, u32 *fw_version) +{ + struct mac_version_t mac_version; + + atl2_shared_buffer_read(hw, version.mac, mac_version); + *fw_version = mac_version.major << 24 | mac_version.minor << 16 | + mac_version.build; + + return 0; +} + +static struct atl_fw_ops atl2_fw_ops = { + .__wait_fw_init = __atl2_fw_wait_init, + .deinit = atl2_fw_deinit, + .set_link = atl2_fw_set_link, + .check_link = atl2_fw_check_link, + .__get_link_caps = __atl2_fw_get_link_caps, + .restart_aneg = atl2_fw_restart_aneg, + .set_default_link = atl2_fw_set_default_link, + .get_phy_temperature = atl2_fw_get_phy_temperature, + .set_mediadetect = atl2_fw_set_mediadetect, + .send_macsec_req = (void *)atl2_fw_unsupported, + .set_pad_stripping = atl2_fw_set_pad_stripping, + .get_mac_addr = atl2_fw_get_mac_addr, + .__get_hbeat = __atl2_fw_get_hbeat, + .set_phy_loopback = atl2_fw_set_phy_loopback, + .enable_wol = atl2_fw_enable_wol, + .dump_cfg = (void *)atl2_fw_dump_cfg, + .restore_cfg = atl2_fw_restore_cfg, + .update_thermal = atl2_fw_update_thermal, + }; + +int atl2_fw_init(struct atl_hw *hw) +{ + struct atl_mcp *mcp = &hw->mcp; + int ret; + + atl2_get_fw_version(hw, &mcp->fw_rev); + + mcp->ops = &atl2_fw_ops; + atl_dev_dbg("Detect ATL2FW %x\n", mcp->fw_rev); + + ret = mcp->ops->__wait_fw_init(hw); + if (ret) + return ret; + + mcp->fw_stat_addr = 0; + mcp->rpc_addr = 0; + + ret = mcp->ops->__get_hbeat(hw, &mcp->phy_hbeat); + if (ret) + return ret; + mcp->next_wdog = jiffies + 2 * HZ; + + ret = mcp->ops->__get_link_caps(hw); + if (ret) + return ret; + + return ret; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.h new file mode 100644 index 000000000000..5db61f37253f --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl2_fw.h @@ -0,0 +1,690 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL2_FW_H_ +#define _ATL2_FW_H_ + +/* Start of HW byte packed interface declaration */ +#pragma pack(push, 1) + +/* F W A P I */ + +struct link_options_s { + uint32_t link_up:1; + uint32_t link_renegotiate:1; + uint32_t minimal_link_speed:1; + uint32_t internal_loopback:1; + uint32_t external_loopback:1; + uint32_t rate_10M_hd:1; + uint32_t rate_100M_hd:1; + uint32_t rate_1G_hd:1; + + + uint32_t rate_10M:1; + uint32_t rate_100M:1; + uint32_t rate_1G:1; + uint32_t rate_2P5G:1; + uint32_t rate_N2P5G:1; + uint32_t rate_5G:1; + uint32_t rate_N5G:1; + uint32_t rate_10G:1; + + uint32_t eee_100M:1; + uint32_t eee_1G:1; + uint32_t eee_2P5G:1; + uint32_t eee_5G:1; + uint32_t eee_10G:1; + uint32_t rsvd3:2; + uint32_t low_power_autoneg:1; + + uint32_t pause_rx:1; + uint32_t pause_tx:1; + uint32_t rsvd4:1; + uint32_t downshift:1; + uint32_t downshift_retry:4; +}; + +struct link_control_s { + uint32_t mode:4; + + uint32_t disable_crc_corruption : 1; + uint32_t discard_short_frames : 1; + uint32_t flow_control_mode : 1; + uint32_t disable_length_check : 1; + uint32_t discard_errored_frames : 1; + uint32_t control_frame_enable : 1; + uint32_t enable_tx_padding : 1; + uint32_t enable_crc_forwarding : 1; + uint32_t enable_frame_padding_removal_rx: 1; + uint32_t promiscuous_mode: 1; + uint32_t rsvd:18; +}; + +struct thermal_shutdown_s { + uint32_t shutdown_enable :1; + uint32_t warning_enable :1; + uint32_t rsvd:6; + + uint32_t shutdown_temp_threshold :8; + uint32_t warning_cold_temp_threshold :8; + uint32_t warning_hot_tempThreshold :8; +}; + +struct mac_address_s { + uint8_t mac_address[6]; +}; + +struct sleep_proxy_s { + struct wake_on_lan_s { + uint32_t wake_on_magic_packet:1; + uint32_t wake_on_pattern:1; + uint32_t wake_on_link_up:1; + uint32_t wake_on_link_down:1; + uint32_t wake_on_ping:1; + uint32_t wake_on_timer:1; + uint32_t wake_on_link_mac_method:1; + uint32_t rsrvd1:1; + uint32_t restore_link_before_wake:1; + uint32_t rsvd:23; + + uint32_t link_up_timeout; + uint32_t link_down_timeout; + uint32_t timer; + + struct { + uint32_t mask[4]; + uint32_t crc32; + } wake_up_patterns[8]; + } wake_on_lan; + + struct { + uint32_t arp_responder:1; + uint32_t echo_responder:1; + uint32_t igmp_client:1; + uint32_t echo_truncate:1; + uint32_t address_guard:1; + uint32_t ignore_fragmented:1; + uint32_t rsvd:2; + uint32_t echo_max_len:16; + uint32_t ipv4[8]; + uint32_t reserved[8]; + + } ipv4_offload; + + struct { + uint32_t ns_responder:1; + uint32_t echo_responder:1; + uint32_t mld_client:1; + uint32_t echo_truncate:1; + uint32_t address_guard:1; + uint32_t rsvd:3; + uint32_t echo_max_len:16; + uint32_t ipv6[16][4]; + } ipv6_offload; + + struct { + uint16_t ports[16]; + } tcp_port_offload; + + struct { + uint16_t ports[16]; + } udp_port_offload; + + struct ka4_offloads_s { + uint32_t retry_count; + uint32_t retry_interval; + + struct ka4_offload_s { + uint32_t timeout; + uint16_t local_port; + uint16_t remote_port; + uint8_t remote_mac_addr[6]; + uint32_t rsvd:16; + uint32_t rsvd2:32; + uint32_t rsvd3:32; + uint32_t rsvd4:16; + uint16_t win_size; + uint32_t seq_num; + uint32_t ack_num; + uint32_t local_ip; + uint32_t remote_ip; + } offloads[16]; + } ka4_offload; + + struct ka6_offloads_s { + uint32_t retry_count; + uint32_t retry_interval; + + struct ka6_offload_s { + uint32_t timeout; + uint16_t local_port; + uint16_t remote_port; + uint8_t remote_mac_addr[6]; + uint32_t rsvd:16; + uint32_t rsvd2:32; + uint32_t rsvd3:32; + uint32_t rsvd4:16; + uint16_t win_size; + uint32_t seq_num; + uint32_t ack_num; + uint32_t local_ip[4]; + uint32_t remote_ip[4]; + } offloads[16]; + } ka6_offload; + + struct { + uint32_t rr_count; + uint32_t rr_buf_len; + uint32_t idx_offset; + uint32_t rr__offset; + } mdns; + /* WARN: where this gap actually is not known */ + uint32_t reserve_fw_gap:16; +}; + +struct ptp_s { + uint32_t enable:1; +}; + +struct pause_quanta_s { + uint16_t quanta_10M; + uint16_t threshold_10M; + uint16_t quanta_100M; + uint16_t threshold_100M; + uint16_t quanta_1G; + uint16_t threshold_1G; + uint16_t quanta_2P5G; + uint16_t threshold_2P5G; + uint16_t quanta_5G; + uint16_t threshold_5G; + uint16_t quanta_10G; + uint16_t threshold_10G; +}; + +struct data_buffer_status_s { + uint32_t data_offset; + uint32_t data_length; +}; + +struct device_caps_s { + uint32_t finite_flashless: 1; + uint32_t cable_diag: 1; + uint32_t ncsi: 1; + uint32_t avb: 1; + uint32_t rsvd: 28; + uint32_t rsvd2: 32; +}; + +struct version_s { + struct bundle_version_t { + uint32_t major:8; + uint32_t minor:8; + uint32_t build:16; + } bundle; + struct mac_version_t { + uint32_t major:8; + uint32_t minor:8; + uint32_t build:16; + } mac; + struct phy_version_t { + uint32_t major:8; + uint32_t minor:8; + uint32_t build:16; + } phy; + uint32_t rsvd:32; +}; + +struct link_status_s { + uint32_t link_state:4; + uint32_t link_rate:4; + + uint32_t pause_tx:1; + uint32_t pause_rx:1; + uint32_t eee:1; + uint32_t duplex:1; + uint32_t rsvd:4; + + uint32_t rsvd2:16; +}; + +struct wol_status_s { + uint32_t wake_count:8; + uint32_t wake_reason:8; + uint32_t wake_up_packet_length :12; + uint32_t wake_up_pattern_number :3; + uint32_t rsvd:1; + uint32_t wake_up_packet[379]; +}; + +struct mac_health_monitor_s { + uint32_t mac_ready:1; + uint32_t mac_fault:1; + uint32_t mac_flashless_finished:1; + uint32_t rsvd:5; + uint32_t mac_temperature:8; + uint32_t mac_heart_beat:16; + uint32_t mac_fault_code:16; + uint32_t rsvd2:16; +}; + +struct phy_health_monitor_s { + uint32_t phy_ready:1; + uint32_t phy_fault:1; + uint32_t phy_hot_warning:1; + uint32_t rsvd:5; + uint32_t phy_temperature:8; + uint32_t phy_heart_beat:16; + uint32_t phy_fault_code:16; + uint32_t rsvd2:16; +}; + +struct device_link_caps_s { + uint32_t rsvd:3; + uint32_t internal_loopback:1; + uint32_t external_loopback:1; + uint32_t rate_10M_hd:1; + uint32_t rate_100M_hd:1; + uint32_t rate_1G_hd:1; + + uint32_t rate_10M:1; + uint32_t rate_100M:1; + uint32_t rate_1G:1; + uint32_t rate_2P5G:1; + uint32_t rate_N2P5G:1; + uint32_t rate_5G:1; + uint32_t rate_N5G:1; + uint32_t rate_10G:1; + + uint32_t rsvd3:1; + uint32_t eee_100M:1; + uint32_t eee_1G:1; + uint32_t eee_2P5G:1; + uint32_t rsvd4:1; + uint32_t eee_5G:1; + uint32_t rsvd5:1; + uint32_t eee_10G:1; + + uint32_t pause_rx:1; + uint32_t pause_tx:1; + uint32_t pfc:1; + uint32_t downshift:1; + uint32_t downshift_retry:4; +}; + +struct sleep_proxy_caps_s { + uint32_t ipv4_offload:1; + uint32_t ipv6_offload:1; + uint32_t tcp_port_offload:1; + uint32_t udp_port_offload:1; + uint32_t ka4_offload:1; + uint32_t ka6_offload:1; + uint32_t mdns_offload:1; + uint32_t wake_on_ping:1; + + uint32_t wake_on_magic_packet:1; + uint32_t wake_on_pattern:1; + uint32_t wake_on_timer:1; + uint32_t wake_on_link:1; + uint32_t wake_patterns_count:4; + + uint32_t ipv4_count:8; + uint32_t ipv6_count:8; + + uint32_t tcp_port_offload_count:8; + uint32_t udp_port_offload_count:8; + + uint32_t tcp4_ka_count:8; + uint32_t tcp6_ka_count:8; + + uint32_t igmp_offload:1; + uint32_t mld_offload:1; + uint32_t rsvd:30; +}; + +struct lkp_link_caps_s { + uint32_t rsvd:5; + uint32_t rate_10M_hd:1; + uint32_t rate_100M_hd:1; + uint32_t rate_1G_hd:1; + + uint32_t rate_10M:1; + uint32_t rate_100M:1; + uint32_t rate_1G:1; + uint32_t rate_2P5G:1; + uint32_t rate_N2P5G:1; + uint32_t rate_5G:1; + uint32_t rate_N5G:1; + uint32_t rate_10G:1; + + uint32_t rsvd2:1; + uint32_t eee_100M:1; + uint32_t eee_1G:1; + uint32_t eee_2P5G:1; + uint32_t rsvd3:1; + uint32_t eee_5G:1; + uint32_t rsvd4:1; + uint32_t eee_10G:1; + + uint32_t pause_rx:1; + uint32_t pause_tx:1; + uint32_t rsvd5:6; +}; + +struct core_dump_s { + uint32_t reg0; + uint32_t reg1; + uint32_t reg2; + + uint32_t hi; + uint32_t lo; + + uint32_t regs[32]; +}; + +struct trace_s { + uint32_t sync_counter; + uint32_t mem_buffer[0x1ff]; +}; + +struct cable_diag_control_s { + uint32_t toggle :1; + uint32_t rsvd:7; + uint32_t wait_timeout_sec:8; + uint32_t rsvd2:16; + +}; + +struct cable_diag_lane_data_s { + uint32_t result_code :8; + uint32_t dist :8; + uint32_t far_dist :8; + uint32_t rsvd:8; +}; + +struct cable_diag_status_s { + struct cable_diag_lane_data_s lane_data[4]; + uint32_t transact_id:8; + uint32_t status:4; + uint32_t rsvd:20; +}; + +struct phy_fw_load_status_s { + uint32_t phy_fw_load_from_host :1; + uint32_t phy_fw_load_from_flash :1; + uint32_t phy_fw_load_from_d_c :1; + uint32_t phy_load_from_flash_failed :1; + uint32_t phy_load_from_host_failed :1; + uint32_t phy_load_from_d_c_failed :1; + uint32_t phy_hash_validation_failed :1; + uint32_t phy_fw_started :1; + + uint32_t phy_stall_timeout :1; + uint32_t phy_unstall_timeout :1; + uint32_t phy_fw_start_timeout :1; + uint32_t phy_iram_load_error :1; + uint32_t phy_dram_load_error :1; + uint32_t phy_mcp_run_failed :1; + uint32_t phy_mcp_stall_failed :1; + uint32_t phy_mcp_unstall_failed :1; + + uint32_t phy_wait_for_semaphore :1; + uint32_t phy_semaphore_locked :1; + uint32_t rsvd :2; + uint32_t phy_worst_block_upload_retry_number:4; + + uint32_t phy_worst_upload_block_number :6; + uint32_t rsvd2:2; +}; + +struct statistics_s { + struct { + uint32_t link_up; + uint32_t link_down; + } link; + + struct { + uint64_t tx_unicast_octets; + uint64_t tx_multicast_octets; + uint64_t tx_broadcast_octets; + uint64_t rx_unicast_octets; + uint64_t rx_multicast_octets; + uint64_t rx_broadcast_octets; + + uint32_t tx_unicast_frames; + uint32_t tx_multicast_frames; + uint32_t tx_broadcast_frames; + uint32_t tx_errors; + + uint32_t rx_unicast_frames; + uint32_t rx_multicast_frames; + uint32_t rx_broadcast_frames; + uint32_t rx_dropped_frames; + uint32_t rx_error_frames; + + uint32_t tx_good_frames; + uint32_t rx_good_frames; + uint32_t reserve_fw_gap; + } msm; + uint32_t main_loop_cycles; + uint32_t reserve_fw_gap; +}; + +struct filter_caps_s { + uint8_t l2_filters_base_index:6; + uint8_t flexible_filter_mask:2; + uint8_t l2_filter_count; + uint8_t ethertype_filter_base_index; + uint8_t ethertype_filter_count; + + uint8_t vlan_filter_base_index; + uint8_t vlan_filter_count; + uint8_t l3_ip4_filter_base_index:4; + uint8_t l3_ip4_filter_count:4; + uint8_t l3_ip6_filter_base_index:4; + uint8_t l3_ip6_filter_count:4; + + uint8_t l4_filter_base_index:4; + uint8_t l4_filter_count:4; + uint8_t l4_flex_filter_base_index:4; + uint8_t l4_flex_filter_count:4; + uint8_t rslv_tbl_base_index; + uint8_t rslv_tbl_count; +}; + +struct request_policy_s { + struct { + uint8_t all:1; + uint8_t rsvd:1; + uint8_t rx_queue_tc_index:5; + uint8_t queue_or_tc:1; + } promisc; + + struct { + uint8_t accept:1; + uint8_t rsvd:1; + uint8_t rx_queue_tc_index:5; + uint8_t queue_or_tc:1; + } bcast; + + struct { + uint8_t accept:1; + uint8_t promisc:1; + uint8_t rx_queue_tc_index:5; + uint8_t queue_or_tc:1; + } mcast; + + uint8_t rsvd:8; +}; + +struct fw_interface_in { + uint32_t mtu; + uint32_t rsvd1:32; + struct mac_address_s mac_address; + uint16_t rsvd; + struct link_control_s link_control; + uint32_t rsvd2:32; + struct link_options_s link_options; + uint32_t rsvd3:32; + struct thermal_shutdown_s thermal_shutdown; + uint32_t rsvd4:32; + struct sleep_proxy_s sleep_proxy; + uint32_t rsvd5:32; + struct pause_quanta_s pause_quanta[8]; + struct cable_diag_control_s cable_diag_control; + uint32_t rsvd6:32; + struct data_buffer_status_s data_buffer_status; + uint32_t rsvd7:32; + struct request_policy_s request_policy; +}; + +struct transaction_counter_s { + uint32_t transaction_cnt_a:16; + uint32_t transaction_cnt_b:16; +}; + +struct management_status_s { + struct mac_address_s mac_address; + uint16_t vlan; + + struct{ + uint32_t enable : 1; + uint32_t rsvd:31; + } flags; + + uint32_t rsvd1:32; + uint32_t rsvd2:32; + uint32_t rsvd3:32; + uint32_t rsvd4:32; + uint32_t rsvd5:32; +}; + +struct fw_interface_out { + struct transaction_counter_s transaction_id; + struct version_s version; + struct link_status_s link_status; + struct wol_status_s wol_status; + uint32_t rsvd:32; + uint32_t rsvd2:32; + struct mac_health_monitor_s mac_health_monitor; + uint32_t rsvd3:32; + uint32_t rsvd4:32; + struct phy_health_monitor_s phy_health_monitor; + uint32_t rsvd5:32; + uint32_t rsvd6:32; + struct cable_diag_status_s cable_diag_status; + uint32_t rsvd7:32; + struct device_link_caps_s device_link_caps; + uint32_t rsvd8:32; + struct sleep_proxy_caps_s sleep_proxy_caps; + uint32_t rsvd9:32; + struct lkp_link_caps_s lkp_link_caps; + uint32_t rsvd10:32; + struct core_dump_s core_dump; + uint32_t rsvd11:32; + struct statistics_s stats; + uint32_t rsvd12:32; + struct filter_caps_s filter_caps; + struct device_caps_s device_caps; + uint32_t rsvd13:32; + struct management_status_s management_status; + uint32_t reserve[21]; + struct trace_s trace; +}; + +struct fw_iti_subblock_header { + uint32_t type :8; + uint32_t length :24; +}; + +struct fw_iti_hdr { + uint32_t instuction_bitmask; + uint32_t reserved; + struct fw_iti_subblock_header iti[6]; +}; +/* End of HW byte packed interface declaration */ +#pragma pack(pop) + + +#define ATL2_FW_LINK_RATE_INVALID 0 +#define ATL2_FW_LINK_RATE_10M 1 +#define ATL2_FW_LINK_RATE_100M 2 +#define ATL2_FW_LINK_RATE_1G 3 +#define ATL2_FW_LINK_RATE_2G5 4 +#define ATL2_FW_LINK_RATE_5G 5 +#define ATL2_FW_LINK_RATE_10G 6 + +#define ATL2_HOST_MODE_INVALID 0 +#define ATL2_HOST_MODE_ACTIVE 1 +#define ATL2_HOST_MODE_SLEEP_PROXY 2 +#define ATL2_HOST_MODE_LOW_POWER 3 +#define ATL2_HOST_MODE_SHUTDOWN 4 + +#define ATL2_FW_CABLE_STATUS_OPEN_CIRCUIT 7 +#define ATL2_FW_CABLE_STATUS_HIGH_MISMATCH 6 +#define ATL2_FW_CABLE_STATUS_LOW_MISMATCH 5 +#define ATL2_FW_CABLE_STATUS_SHORT_CIRCUIT 4 +#define ATL2_FW_CABLE_STATUS_PAIR_D 3 +#define ATL2_FW_CABLE_STATUS_PAIR_C 2 +#define ATL2_FW_CABLE_STATUS_PAIR_B 1 +#define ATL2_FW_CABLE_STATUS_OK 0 + +#define ATL2_FW_HOST_INTERRUPT_MAC_READY 0x0004 +#define ATL2_FW_HOST_INTERRUPT_DATA_HANDLED 0x0100 +#define ATL2_FW_HOST_INTERRUPT_LINK_UP 0x0200 +#define ATL2_FW_HOST_INTERRUPT_LINK_DOWN 0x0400 +#define ATL2_FW_HOST_INTERRUPT_PHY_FAULT 0x0800 +#define ATL2_FW_HOST_INTERRUPT_MAC_FAULT 0x1000 +#define ATL2_FW_HOST_INTERRUPT_TEMPERATURE_WARNING 0x2000 +#define ATL2_FW_HOST_INTERRUPT_HEARTBEAT 0x4000 + +#define ATL2_ITI_ADDRESS_START 0x100000 +#define ATL2_ITI_ADDRESS_BLOCK_1 (ATL2_ITI_ADDRESS_START +\ + sizeof(struct fw_iti_hdr) / sizeof(uint32_t)) +enum { + ATL2_MEMORY_MAILBOX_STATUS_FAIL = 0, + ATL2_MEMORY_MAILBOX_STATUS_SUCCESS = 1 +}; + +enum { + ATL2_MEMORY_MAILBOX_TARGET_MEMORY = 0, + ATL2_MEMORY_MAILBOX_TARGET_MDIO = 1 +}; + +enum { + ATL2_MEMORY_MAILBOX_OPERATION_READ = 0, + ATL2_MEMORY_MAILBOX_OPERATION_WRITE = 1 +}; + +enum ATL2_WAKE_REASON { + ATL2_WAKE_REASON_UNKNOWN, + ATL2_WAKE_REASON_PANIC, + ATL2_WAKE_REASON_LINK, + ATL2_WAKE_REASON_TIMER, + ATL2_WAKE_REASON_RESERVED, + ATL2_WAKE_REASON_NAME_GUARD, + ATL2_WAKE_REASON_ADDR_GUARD, + ATL2_WAKE_REASON_TCPKA, + + ATL2_WAKE_REASON_DATA_FLAG, + + ATL2_WAKE_REASON_PING, + ATL2_WAKE_REASON_SYN, + ATL2_WAKE_REASON_UDP, + ATL2_WAKE_REASON_PATTERN, + ATL2_WAKE_REASON_MAGIC_PACKET +}; + +int atl2_fw_init(struct atl_hw *hw); +int atl2_get_fw_version(struct atl_hw *hw, u32 *fw_version); +int atl2_fw_set_filter_policy(struct atl_hw *hw, bool promisc, bool allmulti); + + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_common.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_common.h new file mode 100644 index 000000000000..85643c61983a --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_common.h @@ -0,0 +1,446 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_COMMON_H_ +#define _ATL_COMMON_H_ + +#include +#include +#include +#include +#include +#include +#include + +#define ATL_VERSION "1.1.11" + +struct atl_nic; + +#include "atl_compat.h" +#include "atl_hw.h" +#include "atl_log.h" +#include "atl_ring_desc.h" +#include "atl_stats.h" + +#define ATL_MAX_QUEUES 8 + +#include "atl_fwd.h" + +struct atl_ptp; + +enum { + ATL_RXF_VLAN_BASE = 0, + ATL_RXF_VLAN_MAX = ATL_VLAN_FLT_NUM, + ATL_RXF_ETYPE_BASE = ATL_RXF_VLAN_BASE + ATL_RXF_VLAN_MAX, + ATL_RXF_ETYPE_MAX = ATL_ETYPE_FLT_NUM, + ATL2_RPF_ETYPE_TAGS = 7, + ATL_RXF_NTUPLE_BASE = ATL_RXF_ETYPE_BASE + ATL_RXF_ETYPE_MAX, + ATL_RXF_NTUPLE_MAX = ATL_NTUPLE_FLT_NUM, + ATL_RXF_FLEX_BASE = ATL_RXF_NTUPLE_BASE + ATL_RXF_NTUPLE_MAX, + ATL_RXF_FLEX_MAX = 1, +}; + +enum atl_rxf_common_cmd { + ATL_RXF_EN = BIT(31), + ATL_RXF_RXQ_MSK = BIT(5) - 1, + ATL_RXF_ACT_SHIFT = 16, + ATL_RXF_ACT_MASK = BIT(3) - 1, + ATL_RXF_ACT_TOHOST = BIT(0) << ATL_RXF_ACT_SHIFT, +}; + +enum atl_ntuple_cmd { + ATL_NTC_EN = ATL_RXF_EN, /* Filter enabled */ + ATL_NTC_V6 = BIT(30), /* IPv6 mode -- only valid in filters + * 0 and 4 */ + ATL_NTC_SA = BIT(29), /* Match source address */ + ATL_NTC_DA = BIT(28), /* Match destination address */ + ATL_NTC_SP = BIT(27), /* Match source port */ + ATL_NTC_DP = BIT(26), /* Match destination port */ + ATL_NTC_PROTO = BIT(25), /* Match L4 proto */ + ATL_NTC_ARP = BIT(24), + ATL_NTC_RXQ = BIT(23), /* Assign Rx queue */ + ATL_NTC_ACT_SHIFT = ATL_RXF_ACT_SHIFT, + ATL_NTC_RXQ_SHIFT = 8, + ATL_NTC_RXQ_MASK = ATL_RXF_RXQ_MSK << ATL_NTC_RXQ_SHIFT, + ATL_NTC_L4_MASK = BIT(3) - 1, + ATL_NTC_L4_TCP = 0, + ATL_NTC_L4_UDP = 1, + ATL_NTC_L4_SCTP = 2, + ATL_NTC_L4_ICMP = 3, +}; + +enum atl2_ntuple_cmd { + ATL2_NTC_L3_IPV4_EN = BIT(0), /* Filter enabled */ + ATL2_NTC_L3_IPV4_SA = BIT(1), + ATL2_NTC_L3_IPV4_DA = BIT(2), + ATL2_NTC_L3_IPV4_PROTO = BIT(3), + ATL2_NTC_L3_IPV4_TAG_SHIFT = 4, + ATL2_NTC_L3_IPV4_PROTO_SHIFT = 8, + + ATL2_NTC_L3_IPV6_EN = BIT(0x10), /* Filter enabled */ + ATL2_NTC_L3_IPV6_SA = BIT(0x11), + ATL2_NTC_L3_IPV6_DA = BIT(0x12), + ATL2_NTC_L3_IPV6_PROTO = BIT(0x13), + ATL2_NTC_L3_IPV6_TAG_SHIFT = 0x14, + ATL2_NTC_L3_IPV6_PROTO_SHIFT = 0x18, + + ATL2_NTC_L4_EN = BIT(0), /* Filter enabled */ + ATL2_NTC_L4_DP = BIT(1), + ATL2_NTC_L4_SP = BIT(2), +}; + +struct atl2_rxf_l3 { + union { + struct { + __be32 dst_ip4; + __be32 src_ip4; + }; + struct { + __be32 dst_ip6[4]; + __be32 src_ip6[4]; + }; + }; + u16 proto; + u32 cmd; + u16 usage; +}; + +struct atl2_rxf_l4 { + __be16 dst_port; + __be16 src_port; + u32 cmd; + u16 usage; +}; + +struct atl_rxf_ntuple { + union { + struct { + __be32 dst_ip4[ATL_RXF_NTUPLE_MAX]; + __be32 src_ip4[ATL_RXF_NTUPLE_MAX]; + }; + struct { + __be32 dst_ip6[ATL_RXF_NTUPLE_MAX][4]; + __be32 src_ip6[ATL_RXF_NTUPLE_MAX][4]; + }; + }; + __be16 dst_port[ATL_RXF_NTUPLE_MAX]; + __be16 src_port[ATL_RXF_NTUPLE_MAX]; + + struct atl2_rxf_l3 l3v4[ATL_RXF_NTUPLE_MAX]; + struct atl2_rxf_l3 l3v6[ATL_RXF_NTUPLE_MAX]; + struct atl2_rxf_l4 l4[ATL_RXF_NTUPLE_MAX]; + s8 l3_idx[ATL_RXF_NTUPLE_MAX]; + bool is_ipv6[ATL_RXF_NTUPLE_MAX]; + s8 l4_idx[ATL_RXF_NTUPLE_MAX]; + uint32_t cmd[ATL_RXF_NTUPLE_MAX]; + int count; + int l3_v4_base_index; + int l3_v4_available; + int l3_v6_base_index; + int l3_v6_available; + int l4_base_index; + int l4_available; +}; + +enum atl_vlan_cmd { + ATL_VLAN_EN = ATL_RXF_EN, + ATL_VLAN_RXQ = BIT(28), + ATL_VLAN_RXQ_SHIFT = 20, + ATL_VLAN_RXQ_MASK = ATL_RXF_RXQ_MSK << ATL_VLAN_RXQ_SHIFT, + ATL_VLAN_ACT_SHIFT = ATL_RXF_ACT_SHIFT, + ATL_VLAN_VID_MASK = BIT(12) - 1, +}; + +#define ATL_VID_MAP_LEN BITS_TO_LONGS(BIT(12)) + +struct atl_rxf_vlan { + uint32_t cmd[ATL_RXF_VLAN_MAX]; + int count; + unsigned long map[ATL_VID_MAP_LEN]; + int vlans_active; + int promisc_count; + int base_index; + int available; +}; + +enum atl_etype_cmd { + ATL_ETYPE_EN = ATL_RXF_EN, + ATL_ETYPE_RXQ = BIT(29), + ATL_ETYPE_RXQ_SHIFT = 20, + ATL_ETYPE_RXQ_MASK = ATL_RXF_RXQ_MSK << ATL_ETYPE_RXQ_SHIFT, + ATL_ETYPE_ACT_SHIFT = ATL_RXF_ACT_SHIFT, + ATL_ETYPE_VAL_MASK = BIT(16) - 1, +}; + +struct atl2_tag_policy { + u16 action; + u16 usage; +}; + +struct atl_rxf_etype { + uint32_t cmd[ATL_RXF_ETYPE_MAX]; + int count; + struct atl2_tag_policy tags_policy[ATL_RXF_ETYPE_MAX]; + int tag[ATL_RXF_ETYPE_MAX]; + int base_index; + int available; + int tag_top; +}; + +struct atl_rxf_mac { + int base_index; + int available; +}; + +enum atl_flex_cmd { + ATL_FLEX_EN = ATL_RXF_EN, + ATL_FLEX_RXQ = BIT(30), + ATL_FLEX_RXQ_SHIFT = 20, + ATL_FLEX_RXQ_MASK = ATL_RXF_RXQ_MSK << ATL_FLEX_RXQ_SHIFT, + ATL_FLEX_ACT_SHIFT = ATL_RXF_ACT_SHIFT, +}; + +struct atl_rxf_flex { + uint32_t cmd[ATL_RXF_FLEX_MAX]; + int count; + int base_index; + int available; +}; + +struct atl_queue_vec; + +#define ATL_NUM_FWD_RINGS ATL_MAX_QUEUES +#define ATL_FWD_RING_BASE ATL_MAX_QUEUES /* Use TC 1 for offload + * engine rings */ +#define ATL_NUM_MSI_VECS 32 +enum { + ATL_IRQ_LINK = 0, + ATL_IRQ_PTP, + ATL_NUM_NON_RING_IRQS, +}; + +#define ATL_RXF_RING_ANY 32 + +#define ATL_FWD_MSI_BASE (ATL_MAX_QUEUES + ATL_NUM_NON_RING_IRQS) + +enum atl_fwd_dir { + ATL_FWDIR_RX = 0, + ATL_FWDIR_TX = 1, + ATL_FWDIR_NUM, +}; + +struct atl_fwd { + unsigned long ring_map[ATL_FWDIR_NUM]; + struct atl_fwd_ring *rings[ATL_FWDIR_NUM][ATL_NUM_FWD_RINGS]; + unsigned long msi_map; + struct blocking_notifier_head nh_clients; +}; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) +struct atl_fwdnl { + struct atl_desc_ring ring_desc[ATL_NUM_FWD_RINGS * 2]; + /* State of forced redirections */ + int force_icmp_via; + int force_tx_via; + /* Deferred TX head cleanup */ + struct delayed_work *tx_cleanup_wq; + u32 tx_bunch; +}; +#endif /* CONFIG_ATLFWD_FWD_NETLINK */ + +struct atl_nic { + struct net_device *ndev; + + struct atl_queue_vec *qvecs; + int nvecs; + struct atl_hw hw; + unsigned flags; + uint32_t priv_flags; + struct timer_list work_timer; + int max_mtu; + unsigned int requested_nvecs; + int requested_rx_size; + int requested_tx_size; + int rx_intr_delay; + int tx_intr_delay; + struct atl_global_stats stats; + spinlock_t stats_lock; + struct work_struct work; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + struct atl_fwd fwd; +#endif +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + struct atl_fwdnl fwdnl; +#endif + + struct atl_rxf_ntuple rxf_ntuple; + struct atl_rxf_vlan rxf_vlan; + struct atl_rxf_etype rxf_etype; + struct atl_rxf_mac rxf_mac; + struct atl_rxf_flex rxf_flex; + + /* PTP support */ + struct atl_ptp *ptp; +}; + +/* Flags only modified with RTNL lock held */ +enum atl_nic_flags { + ATL_FL_MULTIPLE_VECTORS = BIT(0), + ATL_FL_WOL = BIT(1), +}; + +#define ATL_PF(_name) ATL_PF_ ## _name +#define ATL_PF_BIT(_name) ATL_PF_ ## _name ## _BIT +#define ATL_DEF_PF_BIT(_name) ATL_PF_BIT(_name) = BIT(ATL_PF(_name)) + +enum atl_priv_flags { + ATL_PF_LPB_SYS_PB, + ATL_PF_LPB_SYS_DMA, + ATL_PF_LPB_NET_DMA, + ATL_PF_LPB_INT_PHY, + ATL_PF_LPB_EXT_PHY, + ATL_PF_LPI_RX_MAC, + ATL_PF_LPI_TX_MAC, + ATL_PF_LPI_RX_PHY, + ATL_PF_LPI_TX_PHY, + ATL_PF_STATS_RESET, + ATL_PF_STRIP_PAD, + ATL_PF_MEDIA_DETECT, +}; + +enum atl_priv_flag_bits { + ATL_DEF_PF_BIT(LPB_SYS_PB), + ATL_DEF_PF_BIT(LPB_SYS_DMA), + ATL_DEF_PF_BIT(LPB_NET_DMA), + ATL_DEF_PF_BIT(LPB_INT_PHY), + ATL_DEF_PF_BIT(LPB_EXT_PHY), + + ATL_PF_LPB_MASK = ATL_PF_BIT(LPB_SYS_DMA) | ATL_PF_BIT(LPB_SYS_PB) | + ATL_PF_BIT(LPB_NET_DMA) | ATL_PF_BIT(LPB_INT_PHY) | + ATL_PF_BIT(LPB_EXT_PHY), + + ATL_DEF_PF_BIT(LPI_RX_MAC), + ATL_DEF_PF_BIT(LPI_TX_MAC), + ATL_DEF_PF_BIT(LPI_RX_PHY), + ATL_DEF_PF_BIT(LPI_TX_PHY), + ATL_PF_LPI_MASK = ATL_PF_BIT(LPI_RX_MAC) | ATL_PF_BIT(LPI_TX_MAC) | + ATL_PF_BIT(LPI_RX_PHY) | ATL_PF_BIT(LPI_TX_PHY), + + ATL_DEF_PF_BIT(STATS_RESET), + + ATL_DEF_PF_BIT(STRIP_PAD), + ATL_DEF_PF_BIT(MEDIA_DETECT), + + ATL_PF_RW_MASK = ATL_PF_LPB_MASK | ATL_PF_BIT(STATS_RESET) | + ATL_PF_BIT(STRIP_PAD) | ATL_PF_BIT(MEDIA_DETECT), + ATL_PF_RO_MASK = ATL_PF_LPI_MASK, +}; + +#define ATL_MAX_MTU (16352 - ETH_FCS_LEN - ETH_HLEN) + +#define ATL_MAX_RING_SIZE (8192 - 8) +#define ATL_RING_SIZE 4096 + +extern const char atl_driver_name[]; + +extern const struct ethtool_ops atl_ethtool_ops; + +extern unsigned int atl_max_queues; +extern unsigned int atl_max_queues_non_msi; +extern unsigned atl_rx_linear; +extern unsigned atl_min_intr_delay; +extern bool atl_enable_msi; +extern bool atl_wq_non_msi; +extern unsigned int atl_tx_clean_budget; +extern unsigned int atl_tx_free_low; +extern unsigned int atl_tx_free_high; + +#define atl_module_param(_name, _type, _mode) \ + module_param_named(_name, atl_ ## _name, _type, _mode) + +static inline void atl_intr_enable_non_ring(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + atl_intr_enable(hw, hw->non_ring_intr_mask); +} + +static inline void atl_intr_disable_non_ring(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + atl_intr_disable(hw, hw->non_ring_intr_mask); +} + +netdev_tx_t atl_start_xmit(struct sk_buff *skb, struct net_device *ndev); +int atl_vlan_rx_add_vid(struct net_device *ndev, __be16 proto, u16 vid); +int atl_vlan_rx_kill_vid(struct net_device *ndev, __be16 proto, u16 vid); +void atl_set_rx_mode(struct net_device *ndev); +int atl_set_features(struct net_device *ndev, netdev_features_t features); +void atl_get_stats64(struct net_device *ndev, + struct rtnl_link_stats64 *stats); +int atl_setup_datapath(struct atl_nic *nic); +void atl_clear_datapath(struct atl_nic *nic); +int atl_start_rings(struct atl_nic *nic); +void atl_stop_rings(struct atl_nic *nic); +void atl_clear_rdm_cache(struct atl_nic *nic); +void atl_clear_tdm_cache(struct atl_nic *nic); +int atl_alloc_rings(struct atl_nic *nic); +void atl_free_rings(struct atl_nic *nic); +irqreturn_t atl_ring_irq(int irq, void *priv); +void atl_ring_work(struct work_struct *work); +void atl_start_hw_global(struct atl_nic *nic); +int atl_intr_init(struct atl_nic *nic); +void atl_intr_release(struct atl_nic *nic); +int atl_hw_reset(struct atl_hw *hw); +int atl_fw_init(struct atl_hw *hw); +int atl_fw_configure(struct atl_hw *hw); +int atl_reconfigure(struct atl_nic *nic); +void atl_reset_stats(struct atl_nic *nic); +void atl_update_global_stats(struct atl_nic *nic); +void atl_set_loopback(struct atl_nic *nic, int idx, bool on); +void atl_set_intr_mod(struct atl_nic *nic); +void atl_update_ntuple_flt(struct atl_nic *nic, int idx); +void atl_set_vlan_promisc(struct atl_hw *hw, int promisc); +int atl_hwsem_get(struct atl_hw *hw, int idx); +void atl_hwsem_put(struct atl_hw *hw, int idx); +int __atl_msm_read(struct atl_hw *hw, uint32_t addr, uint32_t *val); +int atl_msm_read(struct atl_hw *hw, uint32_t addr, uint32_t *val); +int __atl_msm_write(struct atl_hw *hw, uint32_t addr, uint32_t val); +int atl_msm_write(struct atl_hw *hw, uint32_t addr, uint32_t val); +int atl_update_eth_stats(struct atl_nic *nic); +void atl_adjust_eth_stats(struct atl_ether_stats *stats, + struct atl_ether_stats *base, bool add); +void atl_fwd_release_rings(struct atl_nic *nic); +#if IS_ENABLED(CONFIG_ATLFWD_FWD) +enum atl_fwd_notify; +int atl_fwd_suspend_rings(struct atl_nic *nic); +int atl_fwd_resume_rings(struct atl_nic *nic); +void atl_fwd_notify(struct atl_nic *nic, enum atl_fwd_notify notif, void *data); +#else +static inline int atl_fwd_suspend_rings(struct atl_nic *nic) { return 0; } +static inline int atl_fwd_resume_rings(struct atl_nic *nic) { return 0; } +static inline void atl_fwd_notify(struct atl_nic *nic, + enum atl_fwd_notify notif, void *data) {} +#endif +int atl_get_lpi_timer(struct atl_nic *nic, uint32_t *lpi_delay); +void atl_refresh_rxfs(struct atl_nic *nic); +void atl_schedule_work(struct atl_nic *nic); +int atl_hwmon_init(struct atl_nic *nic); +void atl_thermal_check(struct atl_hw *hw, bool alarm); +int atl_update_thermal(struct atl_hw *hw); +int atl_update_thermal_flag(struct atl_hw *hw, int bit, bool val); +int atl_verify_thermal_limits(struct atl_hw *hw, struct atl_thermal *thermal); +int atl_do_reset(struct atl_nic *nic); +int atl_set_media_detect(struct atl_nic *nic, bool on); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.c new file mode 100644 index 000000000000..a4c8531a8296 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.c @@ -0,0 +1,139 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * Portions Copyright (C) various contributors (see specific commit references) + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_common.h" +#include "atl_ring.h" +#include +#include +#include + +#ifdef ATL_COMPAT_PCI_ALLOC_IRQ_VECTORS + +/* From commit aff171641d181ea573380efc3f559c9de4741fc5 */ +int atl_compat_pci_irq_vector(struct pci_dev *dev, unsigned int nr) +{ +#ifdef CONFIG_PCI_MSI + if (dev->msix_enabled) { + struct msi_desc *entry; + int i = 0; + + for_each_pci_msi_entry(entry, dev) { + if (i == nr) + return entry->irq; + i++; + } + WARN_ON_ONCE(1); + return -EINVAL; + } + + if (dev->msi_enabled) { + struct msi_desc *entry = first_pci_msi_entry(dev); + + if (WARN_ON_ONCE(nr >= entry->nvec_used)) + return -EINVAL; + } else { + if (WARN_ON_ONCE(nr > 0)) + return -EINVAL; + } +#endif + + return dev->irq + nr; +} + +int atl_compat_pci_alloc_irq_vectors(struct pci_dev *dev, + unsigned int min_vecs, unsigned int max_vecs, unsigned int flags) +{ + int vecs = -ENOSPC; + + if (flags & PCI_IRQ_MSIX) { + struct msix_entry *entries; + int i; + + entries = kcalloc(max_vecs, sizeof(*entries), GFP_KERNEL); + if (!entries) + return -ENOMEM; + + for (i = 0; i < max_vecs; i++) + entries[i].entry = i; + + vecs = pci_enable_msix_range(dev, entries, min_vecs, max_vecs); + kfree(entries); + if (vecs > 0) + return vecs; + } + + if (flags & PCI_IRQ_MSI) { + vecs = pci_enable_msi_range(dev, min_vecs, max_vecs); + if (vecs > 0) + return vecs; + } + + /* use legacy irq if allowed */ + if ((flags & PCI_IRQ_LEGACY) && min_vecs == 1) { + pci_intx(dev, 1); + return 1; + } + + return vecs; +} + +#endif + +#ifdef ATL_COMPAT_PCI_ENABLE_MSIX_RANGE +/* from commit 302a2523c277bea0bbe8340312b09507905849ed */ +int atl_compat_pci_enable_msi_range(struct pci_dev *dev, int minvec, int maxvec) +{ + int nvec = maxvec; + int rc; + + if (maxvec < minvec) + return -ERANGE; + + do { + rc = pci_enable_msi_block(dev, nvec); + if (rc < 0) { + return rc; + } else if (rc > 0) { + if (rc < minvec) + return -ENOSPC; + nvec = rc; + } + } while (rc); + + return nvec; +} + +int atl_compat_pci_enable_msix_range(struct pci_dev *dev, + struct msix_entry *entries, + int minvec, int maxvec) +{ + int nvec = maxvec; + int rc; + + if (maxvec < minvec) + return -ERANGE; + + do { + rc = pci_enable_msix(dev, entries, nvec); + if (rc < 0) { + return rc; + } else if (rc > 0) { + if (rc < minvec) + return -ENOSPC; + nvec = rc; + } + } while (rc); + + return nvec; +} + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.h new file mode 100644 index 000000000000..b17e7f42e7c3 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_compat.h @@ -0,0 +1,495 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * Portions Copyright (C) various contributors (see specific commit references) + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_COMPAT_H_ +#define _ATL_COMPAT_H_ + +#include + +#include +#include +#include +#include + +#ifndef IS_REACHABLE +#define IS_REACHABLE defined +#endif + +/* If the kernel is not RHEL / CentOS, then the 2 identifiers below will be + * undefined. Define them this way to simplify the checks below. + */ +#ifndef RHEL_RELEASE_CODE +#define RHEL_RELEASE_CODE 0 +#endif +#ifndef RHEL_RELEASE_VERSION +#define RHEL_RELEASE_VERSION(a,b) 1 +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) +#define ATL_HAVE_MINMAX_MTU +#elif (RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(7,5)) && \ + (RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(8,0)) +#define ndo_change_mtu ndo_change_mtu_rh74 +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,6,0) || RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(7,3) +/* introduced in commit 3f1ac7a700d039c61d8d8b99f28d605d489a60cf */ +#define ATL_HAVE_ETHTOOL_KSETTINGS + +/* introduced in commit 72bb68721f80a1441e871b6afc9ab0b3793d5031 */ +#define ATL_HAVE_IPV6_NTUPLE +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,19,0) || RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(7,3) +/* introduced in commit 892311f66f2411b813ca631009356891a0c2b0a1 */ +#define ATL_HAVE_RXHASH_TYPE +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,16,0) || RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(7,3) +/* introduced in commit 3de0b592394d17b2c41a261a6a493a521213f299 */ +#define ATL_HAVE_ETHTOOL_RXHASH +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,11,0) + +/* ->ndo_get_stats64 return type was changed to void in commit + * bc1f44709cf27fb2a5766cadafe7e2ad5e9cb221. It's safe to just cast + * the pointer to avoid the warning because the only place + * ->ndo_get_stats64 was invoked before the change ignored the return + * value. */ +#define ATL_COMPAT_CAST_NDO_GET_STATS64 + +#endif /* 4.11.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,10,0) + +/* introduced in commit 94842b4fc4d6b1691cfc86c6f5251f299d27f4ba */ +#define ETHTOOL_LINK_MODE_2500baseT_Full_BIT 47 +#define ETHTOOL_LINK_MODE_5000baseT_Full_BIT 48 + +#endif /* 4.10.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,7,0) + +/* from commit 1dff8083a024650c75a9c961c38082473ceae8cf */ +#define page_to_virt(x) __va(PFN_PHYS(page_to_pfn(x))) +#endif /* 4.7.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,19,0) + +/* from commit d31eb342409b24e3d2e1989c775f3361e93acc08 */ +/* Helpers to hide struct msi_desc implementation details */ +#define msi_desc_to_dev(desc) (&(desc)->dev.dev) +#define dev_to_msi_list(dev) (&to_pci_dev((dev))->msi_list) +#define first_msi_entry(dev) \ + list_first_entry(dev_to_msi_list((dev)), struct msi_desc, list) +#define for_each_msi_entry(desc, dev) \ + list_for_each_entry((desc), dev_to_msi_list((dev)), list) + +#define first_pci_msi_entry(pdev) first_msi_entry(&(pdev)->dev) +#define for_each_pci_msi_entry(desc, pdev) \ + for_each_msi_entry((desc), &(pdev)->dev) + +static inline struct pci_dev *msi_desc_to_pci_dev(struct msi_desc *desc) +{ + return desc->dev; +} + +#endif /* 3.19.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,18,0) && RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7,3) +/* ->xmit_more introduced in commit + * 0b725a2ca61bedc33a2a63d0451d528b268cf975 for 3.18-rc1 */ +static inline int skb_xmit_more(struct sk_buff *skb) +{ + return 0; +} +#elif LINUX_VERSION_CODE >= KERNEL_VERSION(5, 2, 0) || RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(8, 2) +static inline int skb_xmit_more(struct sk_buff *skb) +{ + return netdev_xmit_more(); +} +#else /* 3.18.0-5.2.0 for vanilla, anything less than 5.2.0 for RHEL */ +static inline int skb_xmit_more(struct sk_buff *skb) +{ + return skb->xmit_more; +} +#endif /* 3.18.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 14, 0) && \ + RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7, 3) +typedef u16 (*select_queue_fallback_t)(struct net_device *dev, + struct sk_buff *skb); +static inline u16 atlfwd_nl_pick0(struct net_device *dev, struct sk_buff *skb) +{ + return 0; +} +#endif + +/* NB! select_queue_fallback_t MUST be defined before #include on RHEL < 7.3 */ +#include "atl_fwdnl.h" + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 13, 0) && \ + RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7, 3) +static inline u16 atlfwd_nl_select_queue(struct net_device *dev, + struct sk_buff *skb) +{ + return atlfwd_nl_select_queue_fallback(dev, skb, NULL, atlfwd_nl_pick0); +} +#elif LINUX_VERSION_CODE < KERNEL_VERSION(3, 14, 0) && \ + RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7, 3) +static inline u16 atlfwd_nl_select_queue(struct net_device *dev, + struct sk_buff *skb, void *accel_priv) +{ + return atlfwd_nl_select_queue_fallback( + dev, skb, (struct net_device *)accel_priv, atlfwd_nl_pick0); +} +#elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) && \ + RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(8, 0) +static inline u16 atlfwd_nl_select_queue(struct net_device *dev, + struct sk_buff *skb, void *accel_priv, + select_queue_fallback_t fallback) +{ + return atlfwd_nl_select_queue_fallback( + dev, skb, (struct net_device *)accel_priv, fallback); +} +#elif LINUX_VERSION_CODE < KERNEL_VERSION(5, 2, 0) +static inline u16 atlfwd_nl_select_queue(struct net_device *dev, + struct sk_buff *skb, + struct net_device *sb_dev, + select_queue_fallback_t fallback) +{ + return atlfwd_nl_select_queue_fallback(dev, skb, sb_dev, fallback); +} +#else +static inline u16 atlfwd_nl_select_queue(struct net_device *dev, + struct sk_buff *skb, + struct net_device *sb_dev) +{ + return atlfwd_nl_select_queue_fallback(dev, skb, sb_dev, + netdev_pick_tx); +} +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) && RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7,6) +/* introduced in commit 686fef928bba6be13cabe639f154af7d72b63120 */ +static inline void timer_setup(struct timer_list *timer, + void (*callback)(struct timer_list *), unsigned int flags) +{ + setup_timer(timer, (void (*)(unsigned long))callback, + (unsigned long)timer); +} + +#define from_timer(var, callback_timer, timer_fieldname) \ + container_of(callback_timer, typeof(*var), timer_fieldname) +#endif /* 4.14.0 && RHEL < 7.6 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,8,0) && RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7,5) + +#define ATL_COMPAT_PCI_ALLOC_IRQ_VECTORS +int atl_compat_pci_alloc_irq_vectors(struct pci_dev *dev, + unsigned int min_vecs, unsigned int max_vecs, unsigned int flags); +static inline int pci_alloc_irq_vectors(struct pci_dev *dev, + unsigned int min_vecs, unsigned int max_vecs, unsigned int flags) +{ + return atl_compat_pci_alloc_irq_vectors(dev, min_vecs, + max_vecs, flags); +} + +int atl_compat_pci_irq_vector(struct pci_dev *dev, unsigned int nr); +static inline int pci_irq_vector(struct pci_dev *dev, unsigned int nr) +{ + return atl_compat_pci_irq_vector(dev, nr); +} + +static inline void pci_free_irq_vectors(struct pci_dev *dev) +{ + pci_disable_msix(dev); + pci_disable_msi(dev); +} + +/* from commit 4fe0d154880bb6eb833cbe84fa6f385f400f0b9c */ +#define PCI_IRQ_LEGACY (1 << 0) /* allow legacy interrupts */ +#define PCI_IRQ_MSI (1 << 1) /* allow MSI interrupts */ +#define PCI_IRQ_MSIX (1 << 2) /* allow MSI-X interrupts */ + +#endif /* 4.8.0 && RHEL < 7.5 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,8,0) && RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7,4) +/* from commit be9d2e8927cef02076bb7b5b2637cd9f4be2e8df */ +static inline int +pci_request_mem_regions(struct pci_dev *pdev, const char *name) +{ + return pci_request_selected_regions(pdev, + pci_select_bars(pdev, IORESOURCE_MEM), name); +} +#endif /* 4.8.0 && RHEL < 7.4 */ + +#if RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7,3) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + +/* from commit fe896d1878949ea92ba547587bc3075cc688fb8f */ +static inline void page_ref_inc(struct page *page) +{ + atomic_inc(&page->_count); +} + +/* introduced in commit 795bb1c00dd338aa0d12f9a7f1f4776fb3160416 */ +#define napi_consume_skb(__skb, __budget) dev_consume_skb_any(__skb) + +/* from commit 3f1ac7a700d039c61d8d8b99f28d605d489a60cf */ +#define ETHTOOL_LINK_MODE_10baseT_Half_BIT 0 +#define ETHTOOL_LINK_MODE_10baseT_Full_BIT 1 +#define ETHTOOL_LINK_MODE_100baseT_Half_BIT 2 +#define ETHTOOL_LINK_MODE_100baseT_Full_BIT 3 +#define ETHTOOL_LINK_MODE_1000baseT_Half_BIT 4 +#define ETHTOOL_LINK_MODE_1000baseT_Full_BIT 5 +#define ETHTOOL_LINK_MODE_10000baseT_Full_BIT 12 + +/* IPv6 NFC API introduced in commit + * 72bb68721f80a1441e871b6afc9ab0b3793d5031 */ + +/* Define the IPv6 constants for kernels not supporting IPv6 in the + * NFC API to reduce the number of #ifdefs in the code. The constants + * themselves may already be defined for RSS hash management API, so + * #undef them first */ +#undef TCP_V6_FLOW +#define TCP_V6_FLOW 0x05 + +#undef UDP_V6_FLOW +#define UDP_V6_FLOW 0x06 + +#undef SCTP_V6_FLOW +#define SCTP_V6_FLOW 0x07 + +#undef IPV6_USER_FLOW +#define IPV6_USER_FLOW 0x0e +#define IPV4_USER_FLOW IP_USER_FLOW + +#endif /* 4.6.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,5,0) + +/* introduced in commit c7f5d105495a38ed09e70d825f75d9d7d5407264 + * stub it */ +static inline int eth_platform_get_mac_address(struct device *dev, u8 *mac_addr) +{ + return -ENODEV; +} + +#endif /* 4.5.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,1,0) + +/* introduced in 2f064f3485cd29633ad1b3cfb00cc519509a3d72 + * The new implementation can't be used directly as it requires + * changes to linux/mm code. Use the old check described in the + * commit with older kernels insted. It can lead to false positives, + * but as we only use it to determine whether the page is re-usable, + * the false positives can only decrease performance. */ +static inline bool page_is_pfmemalloc(struct page *page) +{ + return page->pfmemalloc && !page->mapping; +} +#endif /* 4.1.0 */ + +#endif /* RHEL_RELEASE_CODE < 7.3 */ + +#if RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(7,2) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,2,0) +#define ETHTOOL_RX_FLOW_SPEC_RING 0x00000000FFFFFFFFLL +#define ETHTOOL_RX_FLOW_SPEC_RING_VF 0x000000FF00000000LL +#define ETHTOOL_RX_FLOW_SPEC_RING_VF_OFF 32 +static inline __u64 ethtool_get_flow_spec_ring(__u64 ring_cookie) +{ + return ETHTOOL_RX_FLOW_SPEC_RING & ring_cookie; +}; +static inline __u64 ethtool_get_flow_spec_ring_vf(__u64 ring_cookie) +{ + return (ETHTOOL_RX_FLOW_SPEC_RING_VF & ring_cookie) >> + ETHTOOL_RX_FLOW_SPEC_RING_VF_OFF; +}; +#endif /* 4.2.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0) + +/* renamed in commit df8a39defad46b83694ea6dd868d332976d62cc0 */ +#define skb_vlan_tag_present(__skb) vlan_tx_tag_present(__skb) +#define skb_vlan_tag_get(__skb) vlan_tx_tag_get(__skb) +#endif /* 4.0.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,19,0) + +/* introduced in commit 1077fa36f23e259858caf6f269a47393a5aff523 + * use plain rmb() for now*/ +#define dma_rmb() rmb() + +/* from commit 9c0c112422a2a6b06fcddcaf21957676490cebba */ +static inline int eth_skb_pad(struct sk_buff *skb) +{ + unsigned int len = ETH_ZLEN; + unsigned int size = skb->len; + + if (unlikely(size < len)) { + len -= size; + if (skb_pad(skb, len)) + return -ENOMEM; + __skb_put(skb, len); + } + return 0; +} + +/* introduced in commit 71dfda58aaaf4bf6b1bc59f9d8afa635fa1337d4 */ +#define __dev_alloc_pages(__flags, __order) __skb_alloc_pages(__flags | __GFP_COMP, NULL, __order) + +/* introduced in commit fd11a83dd3630ec6a60f8a702446532c5c7e1991 */ +#define napi_alloc_skb(__napi, __len) netdev_alloc_skb_ip_align((__napi)->dev, __len) + +/* introduced in commit 3b47d30396bae4f0bd1ff0dbcd7c4f5077e7df4e */ +#define napi_complete_done(__napi, __work_done) napi_complete(__napi) + +/* introduced in commit bc9ad166e38ae1cdcb5323a8aa45dff834d68bfa */ +#define napi_schedule_irqoff(__napi) napi_schedule(__napi) + +/* READ_ONCE() / WRITE_ONCE + * from commit 230fa253df6352af12ad0a16128760b5cb3f92df with changes + * from 43239cbe79fc369f5d2160bd7f69e28b5c50a58c and + * 7bd3e239d6c6d1cad276e8f130b386df4234dcd7 */ + +/* READ_ONCE / WRTIE_ONCE were also cherry-picked into 3.12.58 as + * b5be8baf9e0d5ea035588a0430b2af4989e07572 and + * dda458f0183649e8613a62bb59bec4e5acb883aa */ +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,12,58) || LINUX_VERSION_CODE >= KERNEL_VERSION(3,13,0) + +static __always_inline void __read_once_size(volatile void *p, void *res, int size) +{ + switch (size) { + case 1: *(__u8 *)res = *(volatile __u8 *)p; break; + case 2: *(__u16 *)res = *(volatile __u16 *)p; break; + case 4: *(__u32 *)res = *(volatile __u32 *)p; break; + case 8: *(__u64 *)res = *(volatile __u64 *)p; break; + default: + barrier(); + __builtin_memcpy((void *)res, (const void *)p, size); + barrier(); + } +} + +static __always_inline void __write_once_size(volatile void *p, void *res, int size) +{ + switch (size) { + case 1: *(volatile __u8 *)p = *(__u8 *)res; break; + case 2: *(volatile __u16 *)p = *(__u16 *)res; break; + case 4: *(volatile __u32 *)p = *(__u32 *)res; break; + case 8: *(volatile __u64 *)p = *(__u64 *)res; break; + default: + barrier(); + __builtin_memcpy((void *)p, (const void *)res, size); + barrier(); + } +} + +#define READ_ONCE(x) \ + ({ typeof(x) __val; __read_once_size(&x, &__val, sizeof(__val)); __val; }) + +#define WRITE_ONCE(x, val) \ + ({ typeof(x) __val; __val = val; __write_once_size(&x, &__val, sizeof(__val)); __val; }) + +#endif /* 3.12.58 */ + +#endif /* 3.19.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,18,0) + +/* introduced in commit 56193d1bce2b2759cb4bdcc00cd05544894a0c90 + * pull the whole head buffer len for now*/ +#define eth_get_headlen(__data, __max_len) (__max_len) + +#endif /* 3.18.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) + +/* from commit 286ab723d4b83d37deb4017008ef1444a95cfb0d */ +static inline void ether_addr_copy(u8 *dst, const u8 *src) +{ +#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) + *(u32 *)dst = *(const u32 *)src; + *(u16 *)(dst + 4) = *(const u16 *)(src + 4); +#else + u16 *a = (u16 *)dst; + const u16 *b = (const u16 *)src; + + a[0] = b[0]; + a[1] = b[1]; + a[2] = b[2]; +#endif +} + +/* introduced in commit e6247027e5173c00efb2084d688d06ff835bc3b0 */ +#define dev_consume_skb_any(__skb) dev_kfree_skb_any(__skb) + +/* from commit 09323cc479316e046931a2c679932204b36fea6c */ +enum pkt_hash_types { + PKT_HASH_TYPE_NONE, /* Undefined type */ + PKT_HASH_TYPE_L2, /* Input: src_MAC, dest_MAC */ + PKT_HASH_TYPE_L3, /* Input: src_IP, dst_IP */ + PKT_HASH_TYPE_L4, /* Input: src_IP, dst_IP, src_port, dst_port */ +}; + +static inline void +skb_set_hash(struct sk_buff *skb, __u32 hash, enum pkt_hash_types type) +{ + skb->l4_rxhash = (type == PKT_HASH_TYPE_L4); + skb->rxhash = hash; +} + +/* from commit 302a2523c277bea0bbe8340312b09507905849ed */ +#define ATL_COMPAT_PCI_ENABLE_MSIX_RANGE +int atl_compat_pci_enable_msi_range(struct pci_dev *dev, int minvec, + int maxvec); +int atl_compat_pci_enable_msix_range(struct pci_dev *dev, + struct msix_entry *entries, int minvec, int maxvec); + +static inline int pci_enable_msi_range(struct pci_dev *dev, int minvec, + int maxvec) +{ + return atl_compat_pci_enable_msi_range(dev, minvec, maxvec); +} + +static inline int pci_enable_msix_range(struct pci_dev *dev, + struct msix_entry *entries, int minvec, int maxvec) +{ + return atl_compat_pci_enable_msix_range(dev, entries, minvec, maxvec); +} + +#endif /* 3.14.0 */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,12,27) + +/* from commit 4aa806b771d16b810771d86ce23c4c3160888db3 + * also cherry-picked into 3.12-stable as + * 8bac7a35e60ca70c8d12ddbfdf28a8df5a976b2b */ +static inline int dma_set_mask_and_coherent(struct device *dev, u64 mask) +{ + int rc = dma_set_mask(dev, mask); + if (rc == 0) + dma_set_coherent_mask(dev, mask); + return rc; +} + +#endif /* 3.12.27 */ + +#endif /* RHEL_RELEASE_CODE < 7.2 */ + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_desc.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_desc.h new file mode 100644 index 000000000000..4dd382d9c222 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_desc.h @@ -0,0 +1,159 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_DESC_H_ +#define _ATL_DESC_H_ + +#include + +#if defined(__LITTLE_ENDIAN_BITFIELD) +struct __packed atl_tx_ctx { + unsigned long long :40; //0 + unsigned tun_len:8; //40 + unsigned out_len:16; //48 + unsigned type:3; //64 + unsigned idx:1; //67 + unsigned vlan_tag:16; //68 + unsigned cmd:4; //84 + unsigned l2_len:7; //88 + unsigned l3_len:9; //95 + unsigned l4_len:8; //104 + unsigned mss_len:16; //112 +}; + +struct __packed atl_tx_desc { + unsigned long long daddr:64; //0 + unsigned type:3; //64 + unsigned :1; //67 + unsigned len:16; //68 + unsigned dd:1; //84 + unsigned eop:1; //85 + unsigned cmd:8; //86 + unsigned :14; //94 + unsigned ct_idx:1; //108 + unsigned ct_en:1; //109 + unsigned pay_len:18; //110 +}; + +#define ATL_DATA_PER_TXD 16384 // despite ->len being 16 bits + +enum atl_tx_desc_type { + tx_desc_type_desc = 1, + tx_desc_type_context = 2, +}; + +enum atl_tx_desc_cmd { + tx_desc_cmd_vlan = 1, + tx_desc_cmd_fcs = 2, + tx_desc_cmd_ipv4cs = 4, + tx_desc_cmd_l4cs = 8, + tx_desc_cmd_lso = 0x10, + tx_desc_cmd_wb = 0x20, +}; + +enum atl_tx_ctx_cmd { + ctx_cmd_snap = 1, // SNAP / ~802.3 + ctx_cmd_ipv6 = 2, // IPv6 / ~IPv4 + ctx_cmd_tcp = 4, // TCP / ~UDP +}; + +struct __packed atl_rx_desc { + uint64_t daddr; //0 + union { + struct { + unsigned dd:1; //64 + uint64_t haddr63:63; //65 + }; + uint64_t haddr; + }; +}; + +struct __packed atl_rx_desc_wb { + unsigned rss_type:4; //0 + unsigned pkt_type:8; //4 + unsigned rdm_err:1; //12 + unsigned :6; //13 + unsigned rx_cntl:2; //19 + unsigned sph:1; //21 + unsigned hdr_len:10; //22 + unsigned rss_hash:32; //32 + unsigned dd:1; //64 + unsigned eop:1; //65 + unsigned rx_stat:4; //66 + unsigned rx_estat:6; //70 + unsigned rsc_cnt:4; //76 + unsigned pkt_len:16; //80 + unsigned next_desp:16; //96 + unsigned vlan_tag:16; //112 +}; + +struct __packed atl_rx_desc_hwts_wb { + u32 sec_hw; + u32 ns; + u32 dd:1; + u32 rsvd:1; + u32 sec_lw0:30; + u32 sec_lw1; +}; + +enum atl_rx_stat { + atl_rx_stat_mac_err = 1, + atl_rx_stat_ipv4_err = 2, + atl_rx_stat_l4_err = 4, + atl_rx_stat_l4_valid = 8, + atl_rx_stat_err_msk = atl_rx_stat_mac_err | atl_rx_stat_ipv4_err | + atl_rx_stat_l4_err, +}; + +enum atl_rx_estat { + atl_rx_estat_vlan_stripped = 1, + atl_rx_estat_l2_ucast_match = 2, + atl_rx_estat_vxlan = 1 << 2, + atl_rx_estat_nvgre = 2 << 2, + atl_rx_estat_geneve = 3 << 2, + atl_rx_estat_tun_msk = 3 << 2, + atl_rx_estat_outer_ipv4_err = 16, + atl_rx_estat_outer_ipv4_valid = 32, +}; + +enum atl_rx_pkt_type { + atl_rx_pkt_type_ipv4 = 0, + atl_rx_pkt_type_ipv6 = 1, + atl_rx_pkt_type_l3_other = 2, + atl_rx_pkt_type_l3_arp = 3, + atl_rx_pkt_type_l3_msk = 3, + atl_rx_pkt_type_tcp = 0 << 2, + atl_rx_pkt_type_udp = 1 << 2 , + atl_rx_pkt_type_sctp = 2 << 2, + atl_rx_pkt_type_icmp = 3 << 2, + atl_rx_pkt_type_l4_msk = ((1 << 3) - 1) << 2, + atl_rx_pkt_type_vlan = 1 << 5, + atl_rx_pkt_type_dbl_vlan = 2 << 5, + atl_rx_pkt_type_vlan_msk = ((1 << 2) - 1) << 5, +}; + +#else // defined(__LITTLE_ENDIAN_BITFIELD) +#error XXX Fix bigendian bitfields +#endif // defined(__LITTLE_ENDIAN_BITFIELD) + +union __packed atl_desc { + struct atl_rx_desc rx; + union { + struct atl_rx_desc_wb wb; + struct atl_rx_desc_hwts_wb hwts_wb; + }; + struct atl_tx_ctx ctx; + struct atl_tx_desc tx; + uint8_t raw[16]; +}; + + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_drviface.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_drviface.h new file mode 100644 index 000000000000..d79640e3ec2d --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_drviface.h @@ -0,0 +1,421 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_DRVIFACE_H_ +#define _ATL_DRVIFACE_H_ + +typedef uint16_t in_port_t; +typedef uint32_t in_addr_t; + +struct __attribute__((packed)) offloadKAv4 { + uint32_t timeout; + in_port_t local_port; + in_port_t remote_port; + uint8_t remote_mac_addr[6]; + uint16_t win_size; + uint32_t seq_num; + uint32_t ack_num; + in_addr_t local_ip; + in_addr_t remote_ip; +}; + +struct __attribute__((packed)) offloadKAv6 { + uint32_t timeout; + in_port_t local_port; + in_port_t remote_port; + uint8_t remote_mac_addr[6]; + uint16_t win_size; + uint32_t seq_num; + uint32_t ack_num; + struct in6_addr local_ip; + struct in6_addr remote_ip; +}; + +struct __attribute__((packed)) offloadIPInfo { + uint8_t v4LocalAddrCount; + uint8_t v4AddrCount; + uint8_t v6LocalAddrCount; + uint8_t v6AddrCount; + // FW will add the base to the following offset fields and will treat them as pointers. + // The offsets are relative to the start of this struct so that the struct is pretty much self-contained + // in_addr_t * + uint32_t v4AddrOfft; + // uint8_t * + uint32_t v4PrefixOfft; + // in6_addr * + uint32_t v6AddrOfft; + // uint8_t * + uint32_t v6PrefixOfft; +}; + +struct __attribute__((packed)) offloadPortInfo { + uint16_t UDPPortCount; + uint16_t TCPPortCount; + // in_port_t * + uint32_t UDPPortOfft; // See the comment in the offloadIPInfo struct + // in_port_t * + uint32_t TCPPortOfft; +}; + +struct __attribute__((packed)) offloadKAInfo { + uint16_t v4KACount; + uint16_t v6KACount; + uint32_t retryCount; + uint32_t retryInterval; + // struct offloadKAv4 * + uint32_t v4KAOfft; // See the comment in the offloadIPInfo struct + // struct offloadKAv6 * + uint32_t v6KAOfft; +}; + +struct __attribute__((packed)) offloadRRInfo { + uint32_t RRCount; + uint32_t RRBufLen; + // Offset to RR index table relative to the start of offloadRRInfo struct. The indices + // themselves are relative to the start of RR buffer. FW will add the buffer address + // and will treat them as pointers. + // uint8_t ** + uint32_t RRIdxOfft; + // Offset to the RR buffer relative to the start of offloadRRInfo struct. + // uint8_t * + uint32_t RRBufOfft; +}; + +struct __attribute__((packed)) offloadInfo { + uint32_t version; // = 0 till it stabilizes some + uint32_t len; // The whole structure length including the variable-size buf + uint8_t macAddr[8]; + struct offloadIPInfo ips; + struct offloadPortInfo ports; + struct offloadKAInfo kas; + struct offloadRRInfo rrs; + uint8_t buf[0]; +}; + +#define FW_PACK_STRUCT __attribute__((packed)) + +#define DRV_REQUEST_SIZE 3072 +#define DRV_MSG_PING 0x01 +#define DRV_MSG_ARP 0x02 +#define DRV_MSG_INJECT 0x03 +#define DRV_MSG_WOL_ADD 0x04 +#define DRV_MSG_WOL_REMOVE 0x05 +#define DRV_MSG_ENABLE_WAKEUP 0x06 +#define DRV_MSG_MSM 0x07 +#define DRV_MSG_PROVISIONING 0x08 +#define DRV_MSG_OFFLOAD_ADD 0x09 +#define DRV_MSG_OFFLOAD_REMOVE 0x0A +#define DRV_MSG_MSM_EX 0x0B +#define DRV_MSG_SMBUS_PROXY 0x0C + +#define DRV_PROV_APPLY 1 +#define DRV_PROV_REPLACE 2 +#define DRV_PROV_ADD 3 + +#define FW_RPC_INJECT_PACKET_LEN 1514U + +typedef enum { + EVENT_DRIVER_ENABLE_WOL +} eDriverEvent; + +//typedef enum { +// HOST_UNINIT = 0, +// HOST_RESET, +// HOST_INIT, +// HOST_RESERVED, +// HOST_SLEEP, +// HOST_INVALID +//} hostState_t; + +struct drvMsgPing { + uint32_t ping; +} FW_PACK_STRUCT; + +union IPAddr { + struct + { + uint8_t addr[16]; + } FW_PACK_STRUCT v6; + struct + { + uint8_t padding[12]; + uint8_t addr[4]; + } FW_PACK_STRUCT v4; +} FW_PACK_STRUCT; + +struct drvMsgArp { + uint8_t macAddr[6]; + uint32_t uIpAddrCnt; + struct + { + union IPAddr addr; + union IPAddr mask; + } FW_PACK_STRUCT ip[1]; +} FW_PACK_STRUCT; + +struct drvMsgInject { + uint32_t len; + uint8_t packet[FW_RPC_INJECT_PACKET_LEN]; +} FW_PACK_STRUCT; + +enum ndisPmWoLPacket { + ndisPMWoLPacketUnspecified = 0, + ndisPMWoLPacketBitmapPattern, + ndisPMWoLPacketMagicPacket, + ndisPMWoLPacketIPv4TcpSyn, + ndisPMWoLPacketIPv6TcpSyn, + ndisPMWoLPacketEapolRequestIdMessage, + ndisPMWoLPacketMaximum +}; + +enum aqPmWoLPacket { + aqPMWoLPacketUnspecified = 0x10000, + aqPMWoLPacketArp, + aqPMWoLPacketIPv4Ping, + aqPMWoLPacketIpv6NsPacket, + aqPMWoLPacketIpv6Ping, + aqPMWoLReasonLinkUp, + aqPMWoLReasonLinkDown, + aqPMWoLPacketMaximum +}; + +enum ndisPmProtocolOffloadType { + ndisPMProtocolOffloadIdUnspecified, + ndisPMProtocolOffloadIdIPv4ARP, + ndisPMProtocolOffloadIdIPv6NS, + ndisPMProtocolOffload80211RSNRekey, + ndisPMProtocolOffloadIdMaximum +}; + +struct drvMsgEnableWakeup { + uint32_t patternMaskWindows; + uint32_t patternMaskAquantia; + uint32_t patternMaskOther; + uint32_t offloadsMaskWindows; + uint32_t offloadsMaskAquantia; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddIpv4TcpSynWoLPacketParameters { + uint32_t flags; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4SourceAddress; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4DestAddress; + union { + uint8_t v8[2]; + uint16_t v16; + } TCPSourcePortNumber; + union { + uint8_t v8[2]; + uint16_t v16; + } TCPDestPortNumber; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddIpv6TcpSynWoLPacketParameters { + uint32_t flags; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } IPv6SourceAddress; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } IPv6DestAddress; + union { + uint8_t v8[2]; + uint16_t v16; + } TCPSourcePortNumber; + union { + uint8_t v8[2]; + uint16_t v16; + } TCPDestPortNumber; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddIpv4PingWoLPacketParameters { + uint32_t flags; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4SourceAddress; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4DestAddress; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddIpv6PingWoLPacketParameters { + uint32_t flags; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } IPv6SourceAddress; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } IPv6DestAddress; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddEapolRequestIdMessageWoLPacketParameters { + uint32_t flags; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4SourceAddress; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4DestAddress; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddBitmapPattern { + uint32_t Flags; + uint32_t MaskOffset; + uint32_t MaskSize; + uint32_t PatternOffset; + uint32_t PatternSize; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddMagicPacketPattern { + uint8_t macAddr[6]; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddArpWoLPacketParameters { + uint32_t flags; + union { + uint8_t v8[4]; + uint32_t v32; + } IPv4Address; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddLinkUpWoLParameters { + uint32_t timeout; +} FW_PACK_STRUCT; + +struct drvMsgWoLAddLinkDownWoLParameters { + uint32_t timeout; +} FW_PACK_STRUCT; + +struct drvMsgWoLAdd { + uint32_t priority; // Currently not used + uint32_t packetType; // One of ndisPmWoLPacket or aqPmWoLPacket + uint32_t patternId; // Id to save - will be used in remove message + uint32_t nextWoLPatternOffset; // For chaining multiple additions in one request + + // Depends on `parrernId` + union _WOL_PATTERN { + struct drvMsgWoLAddIpv4TcpSynWoLPacketParameters wolIpv4TcpSyn; + struct drvMsgWoLAddIpv6TcpSynWoLPacketParameters wolIpv6TcpSyn; + struct drvMsgWoLAddEapolRequestIdMessageWoLPacketParameters wolEapolRequestIdMessage; + struct drvMsgWoLAddBitmapPattern wolBitmap; + struct drvMsgWoLAddMagicPacketPattern wolMagicPacket; + struct drvMsgWoLAddIpv4PingWoLPacketParameters wolIpv4Ping; + struct drvMsgWoLAddIpv6PingWoLPacketParameters wolIpv6Ping; + struct drvMsgWoLAddArpWoLPacketParameters wolArp; + struct drvMsgWoLAddLinkUpWoLParameters wolLinkUpReason; + struct drvMsgWoLAddLinkDownWoLParameters wolLinkDownReason; + } wolPattern; +} FW_PACK_STRUCT; + +struct drvMsgWoLRemove { + uint32_t id; +} FW_PACK_STRUCT; + +struct ipv4ArpParameters { + uint32_t flags; + uint8_t remoteIPv4Address[4]; + uint8_t hostIPv4Address[4]; + uint8_t macAddress[6]; +} FW_PACK_STRUCT; + +struct ipv6NsParameters { + uint32_t flags; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } remoteIPv6Address; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } solicitedNodeIPv6Address; + union { + uint8_t v8[16]; + uint32_t v32[4]; + } targetIPv6Addresses[2]; + uint8_t macAddress[6]; +} FW_PACK_STRUCT; + +struct drvMsgOffloadAdd { + uint32_t priority; + uint32_t protocolOffloadType; + uint32_t protocolOffloadId; + uint32_t nextProtocolOffloadOffset; + union { + struct ipv4ArpParameters ipv4Arp; + struct ipv6NsParameters ipv6Ns; + } wolOffload; +} FW_PACK_STRUCT; + +struct drvMsgOffloadRemove { + uint32_t id; +} FW_PACK_STRUCT; + +struct drvMsmSettings { + uint32_t msmReg054; + uint32_t msmReg058; + uint32_t msmReg05c; + uint32_t msmReg060; + uint32_t msmReg064; + uint32_t msmReg068; + uint32_t msmReg06c; + uint32_t msmReg070; + uint32_t flags; // Valid for message DRV_MSG_MSM_EX only +} FW_PACK_STRUCT; + +//struct drvMsgProvisioning { +// uint32_t command; +// uint32_t len; +// provList_t list; +//} FW_PACK_STRUCT; + +//struct drvMsgSmbusProxy { +// uint32_t typeMsg; +// union { +// struct smbusProxyWrite smbWrite; +// struct smbusProxyRead smbRead; +// struct smbusProxyGetStatus smbStatus; +// struct smbusProxyReadResp smbReadResp; +// } FW_PACK_STRUCT; +//} FW_PACK_STRUCT; + +struct drvIface { + uint32_t msgId; + + union { + struct drvMsgPing msgPing; + struct drvMsgArp msgArp; + struct drvMsgInject msgInject; + struct drvMsgWoLAdd msgWoLAdd; + struct drvMsgWoLRemove msgWoLRemove; + struct drvMsgEnableWakeup msgEnableWakeup; + struct drvMsmSettings msgMsm; +// struct drvMsgProvisioning msgProvisioning; + struct drvMsgOffloadAdd msgOffloadAdd; + struct drvMsgOffloadRemove msgOffloadRemove; +// struct drvMsgSmbusProxy msgSmbusProxy; + struct offloadInfo fw2xOffloads; + } FW_PACK_STRUCT; +} FW_PACK_STRUCT; + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.c new file mode 100644 index 000000000000..d845bd8f9dc7 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.c @@ -0,0 +1,3108 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include + +#include "atl_ethtool.h" +#include "atl_common.h" +#include "atl_mdio.h" +#include "atl_ring.h" +#include "atl_fwdnl.h" +#include "atl_macsec.h" +#include "atl_ptp.h" + +static uint32_t atl_ethtool_get_link(struct net_device *ndev) +{ + return ethtool_op_get_link(ndev); +} + +static void atl_link_to_kernel(unsigned int bits, unsigned long *kernel, + bool legacy) +{ + struct atl_link_type *type; + int i; + + atl_for_each_rate(i, type) { + if (legacy && type->ethtool_idx > 31) + continue; + + if (bits & BIT(i)) + __set_bit(type->ethtool_idx, kernel); + } +} + +#define atl_ethtool_get_common(base, modes, lstate, legacy) \ +do { \ + struct atl_fc_state *fc = &(lstate)->fc; \ + (base)->port = PORT_TP; \ + (base)->autoneg = AUTONEG_DISABLE; \ + (base)->eth_tp_mdix = ETH_TP_MDI_INVALID; \ + (base)->eth_tp_mdix_ctrl = ETH_TP_MDI_INVALID; \ + \ + atl_add_link_supported(modes, Autoneg); \ + atl_add_link_supported(modes, TP); \ + atl_add_link_supported(modes, Pause); \ + atl_add_link_supported(modes, Asym_Pause); \ + atl_add_link_advertised(modes, TP); \ + atl_add_link_lpadvertised(modes, Autoneg); \ + \ + if (lstate->autoneg) { \ + (base)->autoneg = AUTONEG_ENABLE; \ + atl_add_link_advertised(modes, Autoneg); \ + } \ + \ + if (fc->req & atl_fc_rx) \ + atl_add_link_advertised(modes, Pause); \ + \ + if (!!(fc->req & atl_fc_rx) ^ !!(fc->req & atl_fc_tx)) \ + atl_add_link_advertised(modes, Asym_Pause); \ + \ + if (fc->cur & atl_fc_rx) \ + atl_add_link_lpadvertised(modes, Pause); \ + \ + if (!!(fc->cur & atl_fc_rx) ^ !!(fc->cur & atl_fc_tx)) \ + atl_add_link_lpadvertised(modes, Asym_Pause); \ + \ + atl_link_to_kernel((lstate)->supported, \ + (unsigned long *)&(modes)->link_modes.supported, \ + legacy); \ + atl_link_to_kernel((lstate)->advertized, \ + (unsigned long *)&(modes)->link_modes.advertising, \ + legacy); \ + atl_link_to_kernel((lstate)->lp_advertized, \ + (unsigned long *)&(modes)->link_modes.lp_advertising, \ + legacy); \ +} while (0) + +#define atl_add_link_supported(ptr, mode) \ + atl_add_link_mode(ptr, SUPPORTED, supported, mode) + +#define atl_add_link_advertised(ptr, mode) \ + atl_add_link_mode(ptr, ADVERTISED, advertising, mode) + +#define atl_add_link_lpadvertised(ptr, mode) \ + atl_add_link_mode(ptr, ADVERTISED, lp_advertising, mode) + +#ifndef ATL_HAVE_ETHTOOL_KSETTINGS + +struct atl_ethtool_compat { + struct { + unsigned long supported; + unsigned long advertising; + unsigned long lp_advertising; + } link_modes; +}; + +#define atl_add_link_mode(ptr, nameuc, namelc, mode) \ + do { \ + (ptr)->link_modes.namelc |= nameuc ## _ ## mode; \ + } while (0) + +static int atl_ethtool_get_settings(struct net_device *ndev, + struct ethtool_cmd *cmd) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_link_state *lstate = &nic->hw.link_state; + struct atl_ethtool_compat cmd_compat; + + memset(&cmd_compat, 0, sizeof(cmd_compat)); + + atl_ethtool_get_common(cmd, &cmd_compat, lstate, true); + cmd->supported = cmd_compat.link_modes.supported; + cmd->advertising = cmd_compat.link_modes.advertising; + cmd->lp_advertising = cmd_compat.link_modes.lp_advertising; + + ethtool_cmd_speed_set(cmd, lstate->link ? lstate->link->speed : 0); + cmd->duplex = (lstate->link) ? lstate->link->duplex : DUPLEX_UNKNOWN; + + return 0; +} + +#else + +#define atl_add_link_mode(ptr, nameuc, namelc, mode) \ + ethtool_link_ksettings_add_link_mode(ptr, namelc, mode) + +static int atl_ethtool_get_ksettings(struct net_device *ndev, + struct ethtool_link_ksettings *cmd) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_link_state *lstate = &nic->hw.link_state; + + ethtool_link_ksettings_zero_link_mode(cmd, supported); + ethtool_link_ksettings_zero_link_mode(cmd, advertising); + ethtool_link_ksettings_zero_link_mode(cmd, lp_advertising); + + atl_ethtool_get_common(&cmd->base, cmd, lstate, false); + + cmd->base.speed = lstate->link ? lstate->link->speed : 0; + cmd->base.duplex = lstate->link ? lstate->link->duplex : DUPLEX_UNKNOWN; + + return 0; +} + +#endif + +#undef atl_add_link_supported +#undef atl_add_link_advertised +#undef atl_add_link_lpadvertised +#undef atl_add_link_mode + +static unsigned int atl_kernel_to_link(const unsigned long int *bits, + bool legacy) +{ + unsigned int ret = 0; + int i; + struct atl_link_type *type; + + atl_for_each_rate(i, type) { + if (legacy && type->ethtool_idx > 31) + continue; + + if (test_bit(type->ethtool_idx, bits)) + ret |= BIT(i); + } + + return ret; +} + +static int atl_set_fixed_speed(struct atl_hw *hw, unsigned int speed, + unsigned int duplex) +{ + unsigned int dplx = (duplex == DUPLEX_HALF) ? DUPLEX_HALF : DUPLEX_FULL; + struct atl_link_state *lstate = &hw->link_state; + struct atl_link_type *type; + unsigned long tmp; + int i; + + lstate->advertized &= ~ATL_EEE_MASK; + atl_for_each_rate(i, type) + if (type->speed == speed && type->duplex == dplx) { + if (!(lstate->supported & BIT(i))) + return -EINVAL; + + lstate->advertized = BIT(i); + break; + } + + if (lstate->eee_enabled) { + atl_link_to_kernel(lstate->supported >> ATL_EEE_BIT_OFFT, + &tmp, false); + /* advertize the supported links */ + tmp = atl_kernel_to_link(&tmp, false); + lstate->advertized |= tmp << ATL_EEE_BIT_OFFT; + } + + lstate->autoneg = false; + hw->mcp.ops->set_link(hw, false); + return 0; +} + +#define atl_ethtool_set_common(base, lstate, advertise, tmp, legacy, speed) \ +do { \ + struct atl_fc_state *fc = &lstate->fc; \ + \ + if ((base)->port != PORT_TP) \ + return -EINVAL; \ + \ + if ((base)->autoneg != AUTONEG_ENABLE) \ + return atl_set_fixed_speed(hw, speed, (base)->duplex); \ + \ + atl_add_link_bit(tmp, Autoneg); \ + atl_add_link_bit(tmp, TP); \ + atl_add_link_bit(tmp, Pause); \ + atl_add_link_bit(tmp, Asym_Pause); \ + atl_link_to_kernel((lstate)->supported, tmp, legacy); \ + \ + if (atl_complement_intersect(advertise, tmp)) { \ + atl_nic_dbg("Unsupported advertising bits from ethtool\n"); \ + return -EINVAL; \ + } \ + \ + lstate->autoneg = true; \ + (lstate)->advertized |= atl_kernel_to_link(advertise, legacy); \ + \ + fc->req = 0; \ + if (atl_test_link_bit(advertise, Pause)) \ + fc->req |= atl_fc_full; \ + \ + if (atl_test_link_bit(advertise, Asym_Pause)) \ + fc->req ^= atl_fc_tx; \ + \ +} while (0) + +#ifndef ATL_HAVE_ETHTOOL_KSETTINGS + +#define atl_add_link_bit(ptr, name) \ + (*(ptr) |= SUPPORTED_ ## name) + +#define atl_test_link_bit(ptr, name) \ + (*(ptr) & SUPPORTED_ ## name) + +static inline bool atl_complement_intersect(const unsigned long *advertised, + unsigned long *supported) +{ + return !!(*(uint32_t *)advertised & ~*(uint32_t *)supported); +} + +static int atl_ethtool_set_settings(struct net_device *ndev, + struct ethtool_cmd *cmd) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + struct atl_link_state *lstate = &hw->link_state; + unsigned long tmp = 0; + uint32_t speed = ethtool_cmd_speed(cmd); + + atl_ethtool_set_common(cmd, lstate, + (unsigned long *)&cmd->advertising, &tmp, true, speed); + hw->mcp.ops->set_link(hw, false); + return 0; +} + +#else + +#define atl_add_link_bit(ptr, name) \ + __set_bit(ETHTOOL_LINK_MODE_ ## name ## _BIT, ptr) + +#define atl_test_link_bit(ptr, name) \ + test_bit(ETHTOOL_LINK_MODE_ ## name ## _BIT, ptr) + +static inline bool atl_complement_intersect(const unsigned long *advertised, + unsigned long *supported) +{ + bitmap_complement(supported, supported, + __ETHTOOL_LINK_MODE_MASK_NBITS); + return bitmap_intersects(advertised, supported, + __ETHTOOL_LINK_MODE_MASK_NBITS); +} + +static int atl_ethtool_set_ksettings(struct net_device *ndev, + const struct ethtool_link_ksettings *cmd) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + struct atl_link_state *lstate = &hw->link_state; + const struct ethtool_link_settings *base = &cmd->base; + __ETHTOOL_DECLARE_LINK_MODE_MASK(tmp); + + bitmap_zero(tmp, __ETHTOOL_LINK_MODE_MASK_NBITS); + + atl_ethtool_set_common(base, lstate, cmd->link_modes.advertising, tmp, + false, cmd->base.speed); + hw->mcp.ops->set_link(hw, false); + return 0; +} + +#endif + +#undef atl_add_link_bit +#undef atl_test_link_bit + +static uint32_t atl_rss_tbl_size(struct net_device *ndev) +{ + return ATL_RSS_TBL_SIZE; +} + +static uint32_t atl_rss_key_size(struct net_device *ndev) +{ + return ATL_RSS_KEY_SIZE; +} + +static int atl_rss_get_rxfh(struct net_device *ndev, uint32_t *tbl, + uint8_t *key, uint8_t *htype) +{ + struct atl_hw *hw = &((struct atl_nic *)netdev_priv(ndev))->hw; + int i; + + if (htype) + *htype = ETH_RSS_HASH_TOP; + + if (key) + memcpy(key, hw->rss_key, atl_rss_key_size(ndev)); + + if (tbl) + for (i = 0; i < atl_rss_tbl_size(ndev); i++) + tbl[i] = hw->rss_tbl[i]; + + return 0; +} + +static int atl_rss_set_rxfh(struct net_device *ndev, const uint32_t *tbl, + const uint8_t *key, const uint8_t htype) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + int i; + uint32_t tbl_size = atl_rss_tbl_size(ndev); + + if (htype && htype != ETH_RSS_HASH_TOP) + return -EINVAL; + + if (tbl) { + for (i = 0; i < tbl_size; i++) + if (tbl[i] >= nic->nvecs) + return -EINVAL; + + for (i = 0; i < tbl_size; i++) + hw->rss_tbl[i] = tbl[i]; + } + + if (key) { + memcpy(hw->rss_key, key, atl_rss_key_size(ndev)); + atl_set_rss_key(hw); + } + + if (tbl) + atl_set_rss_tbl(hw); + + return 0; +} + +static void atl_get_channels(struct net_device *ndev, + struct ethtool_channels *chan) +{ + struct atl_nic *nic = netdev_priv(ndev); + int max_rings; + + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) + max_rings = atl_max_queues; + else + max_rings = atl_max_queues_non_msi; + if (max_rings > num_present_cpus()) + max_rings = num_present_cpus(); + + chan->max_combined = max_rings; + chan->combined_count = nic->nvecs; + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) + chan->max_other = chan->other_count = ATL_NUM_NON_RING_IRQS; +} + +static int atl_set_channels(struct net_device *ndev, + struct ethtool_channels *chan) +{ + struct atl_nic *nic = netdev_priv(ndev); + unsigned int nvecs = chan->combined_count; + + if (!nvecs || chan->rx_count || chan->tx_count) + return -EINVAL; + + if (nic->flags & ATL_FL_MULTIPLE_VECTORS && + chan->other_count != ATL_NUM_NON_RING_IRQS) + return -EINVAL; + + if (!(nic->flags & ATL_FL_MULTIPLE_VECTORS) && + chan->other_count) + return -EINVAL; + + if (nvecs > atl_max_queues) + return -EINVAL; + + nic->requested_nvecs = nvecs; + + return atl_reconfigure(nic); +} + +static void atl_get_pauseparam(struct net_device *ndev, + struct ethtool_pauseparam *pause) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_fc_state *fc = &nic->hw.link_state.fc; + + pause->autoneg = 0; + pause->rx_pause = !!(fc->req & atl_fc_rx); + pause->tx_pause = !!(fc->req & atl_fc_tx); +} + +static int atl_set_pauseparam(struct net_device *ndev, + struct ethtool_pauseparam *pause) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + struct atl_link_state *lstate = &hw->link_state; + struct atl_fc_state *fc = &lstate->fc; + + if ((hw->chip_id == ATL_ATLANTIC) && (atl_fw_major(hw) < 2)) + return -EOPNOTSUPP; + + if (pause->autoneg) + return -EINVAL; + + fc->req = (!!pause->rx_pause << atl_fc_rx_shift) | + (!!pause->tx_pause << atl_fc_tx_shift); + + hw->mcp.ops->set_link(hw, false); + return 0; +} + +static int atl_get_eee(struct net_device *ndev, struct ethtool_eee *eee) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_link_state *lstate = &nic->hw.link_state; + int ret = 0; + + eee->supported = eee->advertised = eee->lp_advertised = 0; + + /* Casting to unsigned long is safe, as atl_link_to_kernel() + * will only access low 32 bits when called with legacy == true + */ + atl_link_to_kernel(lstate->supported >> ATL_EEE_BIT_OFFT, + (unsigned long *)&eee->supported, true); + atl_link_to_kernel(lstate->advertized >> ATL_EEE_BIT_OFFT, + (unsigned long *)&eee->advertised, true); + atl_link_to_kernel(lstate->lp_advertized >> ATL_EEE_BIT_OFFT, + (unsigned long *)&eee->lp_advertised, true); + + eee->eee_enabled = eee->tx_lpi_enabled = lstate->eee_enabled; + eee->eee_active = lstate->eee; + + ret = atl_get_lpi_timer(nic, &nic->hw.lpi_timer); + if (ret == -ENODATA) + ret = 0; + + eee->tx_lpi_timer = nic->hw.lpi_timer; + return ret; +} + +static int atl_set_eee(struct net_device *ndev, struct ethtool_eee *eee) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + struct atl_link_state *lstate = &hw->link_state; + unsigned long tmp = 0; + + if ((hw->chip_id == ATL_ATLANTIC) && (atl_fw_major(hw) < 2)) + return -EOPNOTSUPP; + + if (eee->tx_lpi_timer != nic->hw.lpi_timer) + return -EOPNOTSUPP; + + lstate->eee_enabled = eee->eee_enabled; + + if (lstate->eee_enabled) { + atl_link_to_kernel(lstate->supported >> ATL_EEE_BIT_OFFT, + &tmp, false); + if (eee->advertised & ~tmp) + return -EINVAL; + + /* advertize the requested link or all supported */ + if (eee->advertised) + tmp = eee->advertised; + tmp = atl_kernel_to_link(&tmp, false); + } + + lstate->advertized &= ~ATL_EEE_MASK; + if (lstate->eee_enabled) + lstate->advertized |= tmp << ATL_EEE_BIT_OFFT; + + hw->mcp.ops->set_link(hw, false); + return 0; +} + +static void atl_get_drvinfo(struct net_device *ndev, + struct ethtool_drvinfo *drvinfo) +{ + struct atl_nic *nic = netdev_priv(ndev); + uint32_t fw_rev = nic->hw.mcp.fw_rev; + + strlcpy(drvinfo->driver, atl_driver_name, sizeof(drvinfo->driver)); + strlcpy(drvinfo->version, ATL_VERSION, sizeof(drvinfo->version)); + snprintf(drvinfo->fw_version, sizeof(drvinfo->fw_version), + "%d.%d.%d", fw_rev >> 24, fw_rev >> 16 & 0xff, + fw_rev & 0xffff); + strlcpy(drvinfo->bus_info, pci_name(nic->hw.pdev), + sizeof(drvinfo->bus_info)); +} + +static int atl_nway_reset(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + + return hw->mcp.ops->restart_aneg(hw); +} + +static void atl_get_ringparam(struct net_device *ndev, + struct ethtool_ringparam *rp) +{ + struct atl_nic *nic = netdev_priv(ndev); + + rp->rx_mini_max_pending = rp->rx_mini_pending = 0; + rp->rx_jumbo_max_pending = rp->rx_jumbo_pending = 0; + + rp->rx_max_pending = rp->tx_max_pending = ATL_MAX_RING_SIZE; + + rp->rx_pending = nic->requested_rx_size; + rp->tx_pending = nic->requested_tx_size; +} + +static int atl_set_ringparam(struct net_device *ndev, + struct ethtool_ringparam *rp) +{ + struct atl_nic *nic = netdev_priv(ndev); + + if (rp->rx_mini_pending || rp->rx_jumbo_pending) + return -EINVAL; + + if (rp->rx_pending < 8 || rp->tx_pending < 8) + return -EINVAL; + + nic->requested_rx_size = rp->rx_pending & ~7; + nic->requested_tx_size = rp->tx_pending & ~7; + + return atl_reconfigure(nic); +} + +struct atl_stat_desc { + char stat_name[ETH_GSTRING_LEN]; + int idx; +}; + +#define ATL_TX_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_tx_ring_stats, _field) / \ + sizeof(uint64_t), \ +} + +#define ATL_RX_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_rx_ring_stats, _field) / \ + sizeof(uint64_t), \ +} + +#define ATL_RX_FWD_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_rx_fwd_ring_stats, _field) / \ + sizeof(uint64_t), \ +} + +#define ATL_ETH_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_ether_stats, _field) / \ + sizeof(uint64_t), \ +} + +static const struct atl_stat_desc tx_stat_descs[] = { + ATL_TX_STAT(tx_packets, packets), + ATL_TX_STAT(tx_bytes, bytes), + ATL_TX_STAT(tx_busy, tx_busy), + ATL_TX_STAT(tx_queue_restart, tx_restart), + ATL_TX_STAT(tx_dma_map_failed, dma_map_failed), +}; + +static const struct atl_stat_desc rx_stat_descs[] = { + ATL_RX_STAT(rx_packets, packets), + ATL_RX_STAT(rx_bytes, bytes), + ATL_RX_STAT(rx_multicast_packets, multicast), + ATL_RX_STAT(rx_lin_skb_overrun, linear_dropped), + ATL_RX_STAT(rx_skb_alloc_failed, alloc_skb_failed), + ATL_RX_STAT(rx_head_page_reused, reused_head_page), + ATL_RX_STAT(rx_data_page_reused, reused_data_page), + ATL_RX_STAT(rx_head_page_allocated, alloc_head_page), + ATL_RX_STAT(rx_data_page_allocated, alloc_data_page), + ATL_RX_STAT(rx_head_page_alloc_failed, alloc_head_page_failed), + ATL_RX_STAT(rx_data_page_alloc_failed, alloc_data_page_failed), + ATL_RX_STAT(rx_non_eop_descs, non_eop_descs), + ATL_RX_STAT(rx_mac_err, mac_err), + ATL_RX_STAT(rx_checksum_err, csum_err), +}; + +static const struct atl_stat_desc rx_fwd_stat_descs[] = { + ATL_RX_FWD_STAT(rx_fwd_packets, packets), + ATL_RX_FWD_STAT(rx_fwd_bytes, bytes), +}; + +static const struct atl_stat_desc eth_stat_descs[] = { + ATL_ETH_STAT(tx_pause, tx_pause), + ATL_ETH_STAT(rx_pause, rx_pause), + ATL_ETH_STAT(rx_ether_drops, rx_ether_drops), + ATL_ETH_STAT(rx_ether_octets, rx_ether_octets), + ATL_ETH_STAT(rx_ether_pkts, rx_ether_pkts), + ATL_ETH_STAT(rx_ether_broacasts, rx_ether_broacasts), + ATL_ETH_STAT(rx_ether_multicasts, rx_ether_multicasts), + ATL_ETH_STAT(rx_ether_crc_align_errs, rx_ether_crc_align_errs), + ATL_ETH_STAT(rx_filter_host, rx_filter_host), + ATL_ETH_STAT(rx_filter_lost, rx_filter_lost), +}; + +#define ATL_PRIV_FLAG(_name, _bit) \ + [ATL_PF(_bit)] = #_name + +static const char atl_priv_flags[][ETH_GSTRING_LEN] = { + ATL_PRIV_FLAG(PKTSystemLoopback, LPB_SYS_PB), + ATL_PRIV_FLAG(DMASystemLoopback, LPB_SYS_DMA), + ATL_PRIV_FLAG(DMANetworkLoopback, LPB_NET_DMA), + ATL_PRIV_FLAG(PHYInternalLoopback, LPB_INT_PHY), + ATL_PRIV_FLAG(PHYExternalLoopback, LPB_EXT_PHY), + ATL_PRIV_FLAG(RX_LPI_MAC, LPI_RX_MAC), + ATL_PRIV_FLAG(TX_LPI_MAC, LPI_TX_MAC), + ATL_PRIV_FLAG(RX_LPI_PHY, LPI_RX_PHY), + ATL_PRIV_FLAG(TX_LPI_PHY, LPI_TX_PHY), + ATL_PRIV_FLAG(ResetStatistics, STATS_RESET), + ATL_PRIV_FLAG(StripEtherPadding, STRIP_PAD), + ATL_PRIV_FLAG(MediaDetect, MEDIA_DETECT), +}; + +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + +#define ATL_MACSEC_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_macsec_common_stats, _field) / \ + sizeof(uint64_t), \ +} + +#define ATL_MACSEC_RX_SA_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_macsec_rx_sa_stats, _field) / \ + sizeof(uint64_t), \ +} + +#define ATL_MACSEC_TX_SA_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_macsec_tx_sa_stats, _field) / \ + sizeof(uint64_t), \ +} + +#define ATL_MACSEC_TX_SC_STAT(_name, _field) \ +{ \ + .stat_name = #_name, \ + .idx = offsetof(struct atl_macsec_tx_sc_stats, _field) / \ + sizeof(uint64_t), \ +} + +static const struct atl_stat_desc macsec_stat_descs[] = { + ATL_MACSEC_STAT(in_ctl_pkts, in.ctl_pkts), + ATL_MACSEC_STAT(in_tagged_miss_pkts, in.tagged_miss_pkts), + ATL_MACSEC_STAT(in_untagged_miss_pkts, in.untagged_miss_pkts), + ATL_MACSEC_STAT(in_notag_pkts, in.notag_pkts), + ATL_MACSEC_STAT(in_untagged_pkts, in.untagged_pkts), + ATL_MACSEC_STAT(in_bad_tag_pkts, in.bad_tag_pkts), + ATL_MACSEC_STAT(in_no_sci_pkts, in.no_sci_pkts), + ATL_MACSEC_STAT(in_unknown_sci_pkts, in.unknown_sci_pkts), + ATL_MACSEC_STAT(in_ctrl_prt_pass_pkts, in.ctrl_prt_pass_pkts), + ATL_MACSEC_STAT(in_unctrl_prt_pass_pkts, in.unctrl_prt_pass_pkts), + ATL_MACSEC_STAT(in_ctrl_prt_fail_pkts, in.ctrl_prt_fail_pkts), + ATL_MACSEC_STAT(in_unctrl_prt_fail_pkts, in.unctrl_prt_fail_pkts), + ATL_MACSEC_STAT(in_too_long_pkts, in.too_long_pkts), + ATL_MACSEC_STAT(in_igpoc_ctl_pkts, in.igpoc_ctl_pkts), + ATL_MACSEC_STAT(in_ecc_error_pkts, in.ecc_error_pkts), + ATL_MACSEC_STAT(in_unctrl_hit_drop_redir, in.unctrl_hit_drop_redir), + ATL_MACSEC_STAT(out_ctl_pkts, out.ctl_pkts), + ATL_MACSEC_STAT(out_unknown_sa_pkts, out.unknown_sa_pkts), + ATL_MACSEC_STAT(out_untagged_pkts, out.untagged_pkts), + ATL_MACSEC_STAT(out_too_long, out.too_long), + ATL_MACSEC_STAT(out_ecc_error_pkts, out.ecc_error_pkts), + ATL_MACSEC_STAT(out_unctrl_hit_drop_redir, out.unctrl_hit_drop_redir), +}; + +static const struct atl_stat_desc macsec_rx_sa_stat_descs[] = { + ATL_MACSEC_RX_SA_STAT(untagged_hit_pkts, untagged_hit_pkts), + ATL_MACSEC_RX_SA_STAT(ctrl_hit_drop_redir_pkts, ctrl_hit_drop_redir_pkts), + ATL_MACSEC_RX_SA_STAT(not_using_sa, not_using_sa), + ATL_MACSEC_RX_SA_STAT(unused_sa, unused_sa), + ATL_MACSEC_RX_SA_STAT(not_valid_pkts, not_valid_pkts), + ATL_MACSEC_RX_SA_STAT(invalid_pkts, invalid_pkts), + ATL_MACSEC_RX_SA_STAT(ok_pkts, ok_pkts), + ATL_MACSEC_RX_SA_STAT(late_pkts, late_pkts), + ATL_MACSEC_RX_SA_STAT(delayed_pkts, delayed_pkts), + ATL_MACSEC_RX_SA_STAT(unchecked_pkts, unchecked_pkts), + ATL_MACSEC_RX_SA_STAT(validated_octets, validated_octets), + ATL_MACSEC_RX_SA_STAT(decrypted_octets, decrypted_octets), +}; + +static const struct atl_stat_desc macsec_tx_sa_stat_descs[] = { + ATL_MACSEC_TX_SA_STAT(hit_drop_redirect, sa_hit_drop_redirect), + ATL_MACSEC_TX_SA_STAT(protected2_pkts, sa_protected2_pkts), + ATL_MACSEC_TX_SA_STAT(protected_pkts, sa_protected_pkts), + ATL_MACSEC_TX_SA_STAT(encrypted_pkts, sa_encrypted_pkts), +}; + + +static const struct atl_stat_desc macsec_tx_sc_stat_descs[] = { + ATL_MACSEC_TX_SC_STAT(protected_pkts, sc_protected_pkts), + ATL_MACSEC_TX_SC_STAT(encrypted_pkts, sc_encrypted_pkts), + ATL_MACSEC_TX_SC_STAT(protected_octets, sc_protected_octets), + ATL_MACSEC_TX_SC_STAT(encrypted_octets, sc_encrypted_octets), +}; +#endif + +static int atl_get_sset_count(struct net_device *ndev, int sset) +{ + struct atl_nic *nic = netdev_priv(ndev); + + switch (sset) { + case ETH_SS_STATS: + return ARRAY_SIZE(tx_stat_descs) * (nic->nvecs + 1) + + ARRAY_SIZE(rx_stat_descs) * (nic->nvecs + 1) + + ARRAY_SIZE(eth_stat_descs) +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + + ARRAY_SIZE(rx_fwd_stat_descs) +#endif +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + + ARRAY_SIZE(tx_stat_descs) * + hweight_long(nic->fwd.ring_map[ATL_FWDIR_TX]) + + ARRAY_SIZE(rx_stat_descs) * + hweight_long(nic->fwd.ring_map[ATL_FWDIR_RX]) +#endif +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + + ARRAY_SIZE(macsec_stat_descs) + + ARRAY_SIZE(macsec_tx_sc_stat_descs) * + atl_macsec_tx_sc_cnt(&nic->hw) + + ARRAY_SIZE(macsec_tx_sa_stat_descs) * + atl_macsec_tx_sa_cnt(&nic->hw) + + ARRAY_SIZE(macsec_rx_sa_stat_descs) * + atl_macsec_rx_sa_cnt(&nic->hw) +#endif + ; + + case ETH_SS_PRIV_FLAGS: + return ARRAY_SIZE(atl_priv_flags); + + default: + return -EOPNOTSUPP; + } +} + +static void atl_copy_stats_strings(char **data, char *prefix, + const struct atl_stat_desc *descs, int len) +{ + int i; + + for (i = 0; i < len; i++) { + snprintf(*data, ETH_GSTRING_LEN, "%s%s", prefix, + descs[i].stat_name); + *data += ETH_GSTRING_LEN; + } +} + +static void atl_copy_stats_string_set(char **data, char *prefix) +{ + atl_copy_stats_strings(data, prefix, tx_stat_descs, + ARRAY_SIZE(tx_stat_descs)); + atl_copy_stats_strings(data, prefix, rx_stat_descs, + ARRAY_SIZE(rx_stat_descs)); +} + +static void atl_get_strings(struct net_device *ndev, uint32_t sset, + uint8_t *data) +{ + struct atl_nic *nic = netdev_priv(ndev); + int i; + char prefix[16]; + char *p = data; + + switch (sset) { + case ETH_SS_STATS: + atl_copy_stats_string_set(&p, ""); + +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + atl_copy_stats_strings(&p, "", rx_fwd_stat_descs, + ARRAY_SIZE(rx_fwd_stat_descs)); +#endif + atl_copy_stats_strings(&p, "", eth_stat_descs, + ARRAY_SIZE(eth_stat_descs)); + + for (i = 0; i < nic->nvecs; i++) { + snprintf(prefix, sizeof(prefix), "ring_%d_", i); + atl_copy_stats_string_set(&p, prefix); + } + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + for (i = 0; i < ATL_NUM_FWD_RINGS; i++) { + snprintf(prefix, sizeof(prefix), "fwd_ring_%d_", i); + + if (atlfwd_nl_is_tx_fwd_ring_created(ndev, i)) + atl_copy_stats_strings( + &p, prefix, tx_stat_descs, + ARRAY_SIZE(tx_stat_descs)); + if (atlfwd_nl_is_rx_fwd_ring_created(ndev, i)) + atl_copy_stats_strings( + &p, prefix, rx_stat_descs, + ARRAY_SIZE(rx_stat_descs)); + } +#endif +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + atl_copy_stats_strings(&p, "macsec_", macsec_stat_descs, + ARRAY_SIZE(macsec_stat_descs)); + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + struct atl_macsec_txsc *atl_txsc = + &nic->hw.macsec_cfg.atl_txsc[i]; + int assoc_num; + + if (!(test_bit(i, &nic->hw.macsec_cfg.txsc_idx_busy))) + continue; + + snprintf(prefix, sizeof(prefix), "txsc%d_", + atl_txsc->hw_sc_idx); + atl_copy_stats_strings( + &p, prefix, macsec_tx_sc_stat_descs, + ARRAY_SIZE(macsec_tx_sc_stat_descs)); + for (assoc_num = 0; assoc_num < MACSEC_NUM_AN; + assoc_num++) { + if (!test_bit(assoc_num, + &atl_txsc->tx_sa_idx_busy)) + continue; + snprintf(prefix, sizeof(prefix), "txsc%d_sa%d_", + atl_txsc->hw_sc_idx, assoc_num); + atl_copy_stats_strings( + &p, prefix, macsec_tx_sa_stat_descs, + ARRAY_SIZE(macsec_tx_sa_stat_descs)); + } + } + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + struct atl_macsec_rxsc *atl_rxsc = + &nic->hw.macsec_cfg.atl_rxsc[i]; + int assoc_num; + + if (!(test_bit(i, &nic->hw.macsec_cfg.rxsc_idx_busy))) + continue; + + for (assoc_num = 0; assoc_num < MACSEC_NUM_AN; + assoc_num++) { + if (!test_bit(assoc_num, + &atl_rxsc->rx_sa_idx_busy)) + continue; + snprintf(prefix, sizeof(prefix), "rxsc%d_sa%d_", + atl_rxsc->hw_sc_idx, assoc_num); + atl_copy_stats_strings( + &p, prefix, macsec_rx_sa_stat_descs, + ARRAY_SIZE(macsec_rx_sa_stat_descs)); + } + } +#endif + return; + + case ETH_SS_PRIV_FLAGS: + memcpy(p, atl_priv_flags, sizeof(atl_priv_flags)); + return; + } +} + +#define atl_write_stats(stats, descs, data, type) \ +do { \ + type *_stats = (type *)(stats); \ + int i; \ + \ + for (i = 0; i < ARRAY_SIZE(descs); i++) \ + *(data)++ = _stats[descs[i].idx]; \ +} while (0) + + +static void atl_get_ethtool_stats(struct net_device *ndev, + struct ethtool_stats *stats, u64 *data) +{ + struct atl_nic *nic = netdev_priv(ndev); + int i; + + atl_update_eth_stats(nic); + atl_update_global_stats(nic); +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + atl_macsec_update_stats(&nic->hw); +#endif + atl_write_stats(&nic->stats.tx, tx_stat_descs, data, uint64_t); + atl_write_stats(&nic->stats.rx, rx_stat_descs, data, uint64_t); +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + atl_write_stats(&nic->stats.rx_fwd, rx_fwd_stat_descs, data, uint64_t); +#endif + + atl_write_stats(&nic->stats.eth, eth_stat_descs, data, uint64_t); + + for (i = 0; i < nic->nvecs; i++) { + struct atl_queue_vec *qvec = &nic->qvecs[i]; + struct atl_ring_stats tmp; + + atl_get_ring_stats(&qvec->tx, &tmp); + atl_write_stats(&tmp.tx, tx_stat_descs, data, uint64_t); + atl_get_ring_stats(&qvec->rx, &tmp); + atl_write_stats(&tmp.rx, rx_stat_descs, data, uint64_t); + } + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + for (i = 0; i < ATL_NUM_FWD_RINGS; i++) { + struct atl_ring_stats tmp; + + if (atlfwd_nl_is_tx_fwd_ring_created(ndev, i)) { + atl_fwd_get_ring_stats(nic->fwd.rings[ATL_FWDIR_TX][i], + &tmp); + atl_write_stats(&tmp.tx, tx_stat_descs, data, uint64_t); + } + if (atlfwd_nl_is_rx_fwd_ring_created(ndev, i)) { + atl_fwd_get_ring_stats(nic->fwd.rings[ATL_FWDIR_RX][i], + &tmp); + atl_write_stats(&tmp.rx, rx_stat_descs, data, uint64_t); + } + } +#endif +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + atl_write_stats(&nic->hw.macsec_cfg.stats, macsec_stat_descs, data, + uint64_t); + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + struct atl_macsec_txsc *atl_txsc = + &nic->hw.macsec_cfg.atl_txsc[i]; + int assoc_num; + + if (!(test_bit(i, &nic->hw.macsec_cfg.txsc_idx_busy))) + continue; + + atl_write_stats(&atl_txsc->stats, macsec_tx_sc_stat_descs, data, + uint64_t); + + for (assoc_num = 0; assoc_num < MACSEC_NUM_AN; assoc_num++) { + if (!test_bit(assoc_num, &atl_txsc->tx_sa_idx_busy)) + continue; + atl_write_stats(&atl_txsc->tx_sa_stats[assoc_num], + macsec_tx_sa_stat_descs, data, + uint64_t); + } + } + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + struct atl_macsec_rxsc *atl_rxsc = + &nic->hw.macsec_cfg.atl_rxsc[i]; + int assoc_num; + + if (!(test_bit(i, &nic->hw.macsec_cfg.rxsc_idx_busy))) + continue; + + for (assoc_num = 0; assoc_num < MACSEC_NUM_AN; assoc_num++) { + if (!test_bit(assoc_num, &atl_rxsc->rx_sa_idx_busy)) + continue; + atl_write_stats(&atl_rxsc->rx_sa_stats[assoc_num], + macsec_rx_sa_stat_descs, data, uint64_t); + } + } +#endif +} + +static int atl_update_eee_pflags(struct atl_nic *nic) +{ + int ret = 0; + uint8_t prtad = 0; + uint32_t val; + uint16_t phy_val; + uint32_t flags = nic->priv_flags; + struct atl_link_type *link = nic->hw.link_state.link; + struct atl_hw *hw = &nic->hw; + + flags &= ~ATL_PF_LPI_MASK; + + if (!link || link->speed == 100) + goto done; + + if (link->speed == 1000) { + ret = atl_mdio_read(hw, prtad, 3, 1, &phy_val); + if (ret) + goto done; + + if (phy_val & BIT(9)) + flags |= ATL_PF_BIT(LPI_TX_PHY); + + if (phy_val & BIT(8)) + flags |= ATL_PF_BIT(LPI_RX_PHY); + } else { + ret = atl_mdio_read(hw, prtad, 3, 0xc830, &phy_val); + if (ret) + goto done; + + if (phy_val & BIT(0)) + flags |= ATL_PF_BIT(LPI_TX_PHY); + + ret = atl_mdio_read(hw, prtad, 3, 0xe834, &phy_val); + if (ret) + goto done; + + if (phy_val & BIT(0)) + flags |= ATL_PF_BIT(LPI_RX_PHY); + + } + + ret = atl_msm_read(&nic->hw, ATL_MSM_GEN_STS, &val); + if (ret) + goto done; + + if (val & BIT(8)) + flags |= ATL_PF_BIT(LPI_TX_MAC); + if (val & BIT(4)) + flags |= ATL_PF_BIT(LPI_RX_MAC); + +done: + nic->priv_flags = flags; + return ret; +} + +void atl_reset_stats(struct atl_nic *nic) +{ + struct atl_queue_vec *qvec; + + /* Fetch current MSM stats */ + atl_update_eth_stats(nic); + + spin_lock(&nic->stats_lock); + /* Adding current relative values to base makes it equal to + * current absolute values, thus zeroing the relative values. */ + atl_adjust_eth_stats(&nic->stats.eth_base, &nic->stats.eth, true); + + atl_for_each_qvec(nic, qvec) { + memset(&qvec->rx.stats, 0, sizeof(qvec->rx.stats)); + memset(&qvec->tx.stats, 0, sizeof(qvec->tx.stats)); + } + memset(&nic->stats.rx_fwd, 0, sizeof(nic->stats.rx_fwd)); + + spin_unlock(&nic->stats_lock); +} + +static int atl_set_pad_stripping(struct atl_nic *nic, bool on) +{ + struct atl_hw *hw = &nic->hw; + int ret; + + ret = hw->mcp.ops->set_pad_stripping(hw, on); + + return ret; +} + +int atl_set_media_detect(struct atl_nic *nic, bool on) +{ + struct atl_hw *hw = &nic->hw; + int ret; + + ret = hw->mcp.ops->set_mediadetect(hw, on); + + return ret; +} + +static uint32_t atl_get_priv_flags(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + + atl_update_eee_pflags(nic); + return nic->priv_flags; +} + +static int atl_set_priv_flags(struct net_device *ndev, uint32_t flags) +{ + struct atl_nic *nic = netdev_priv(ndev); + uint32_t diff = flags ^ nic->priv_flags; + uint32_t curr = nic->priv_flags & ATL_PF_LPB_MASK; + uint32_t lpb = flags & ATL_PF_LPB_MASK; + int ret; + + if (diff & ATL_PF_RO_MASK) + return -EINVAL; + + if (diff & ~ATL_PF_RW_MASK) + return -EOPNOTSUPP; + + if (flags & ATL_PF_BIT(STATS_RESET)) + atl_reset_stats(nic); + flags &= ~ATL_PF_BIT(STATS_RESET); + + if (diff & ATL_PF_BIT(STRIP_PAD)) { + ret = atl_set_pad_stripping(nic, + !!(flags & ATL_PF_BIT(STRIP_PAD))); + if (ret) + return ret; + } + + if (diff & ATL_PF_BIT(MEDIA_DETECT)) { + ret = atl_set_media_detect(nic, + !!(flags & ATL_PF_BIT(MEDIA_DETECT))); + if (ret) + return ret; + } + + if (hweight32(lpb) > 1) { + atl_nic_err("Can't enable more than one loopback simultaneously\n"); + return -EINVAL; + } + + if (lpb & ATL_PF_BIT(LPB_SYS_DMA) && !atl_rx_linear) { + atl_nic_err("System DMA loopback suported only in rx_linear mode\n"); + return -EINVAL; + } + + nic->priv_flags = flags; + + if (curr) + atl_set_loopback(nic, ffs(curr) - 1, false); + + if (lpb) + atl_set_loopback(nic, ffs(lpb) - 1, true); + + return 0; +} + +static int atl_get_coalesce(struct net_device *ndev, + struct ethtool_coalesce *ec) +{ + struct atl_nic *nic = netdev_priv(ndev); + + memset(ec, 0, sizeof(*ec)); + ec->rx_coalesce_usecs = nic->rx_intr_delay; + ec->tx_coalesce_usecs = nic->tx_intr_delay; + + return 0; +} + +static int atl_set_coalesce(struct net_device *ndev, + struct ethtool_coalesce *ec) +{ + struct atl_nic *nic = netdev_priv(ndev); + + if (ec->rx_max_coalesced_frames || +#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 7, 0) + ec->use_adaptive_rx_coalesce || ec->use_adaptive_tx_coalesce || + ec->rx_max_coalesced_frames_irq || ec->rx_coalesce_usecs_irq || + ec->tx_max_coalesced_frames_irq || ec->tx_coalesce_usecs_irq || +#endif + ec->tx_max_coalesced_frames) + return -EOPNOTSUPP; + + if (ec->rx_coalesce_usecs < atl_min_intr_delay || + ec->tx_coalesce_usecs < atl_min_intr_delay) { + atl_nic_err("Interrupt coalescing delays less than min_intr_delay (%d uS) not supported\n", + atl_min_intr_delay); + return -EINVAL; + } + + nic->rx_intr_delay = ec->rx_coalesce_usecs; + nic->tx_intr_delay = ec->tx_coalesce_usecs; + + atl_set_intr_mod(nic); + + return 0; +} + +static int atl_get_ts_info(struct net_device *ndev, + struct ethtool_ts_info *info) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct ptp_clock *ptp_clock; + + ethtool_op_get_ts_info(ndev, info); + + if (!nic->ptp) + return 0; + + info->so_timestamping |= + SOF_TIMESTAMPING_TX_HARDWARE | + SOF_TIMESTAMPING_RX_HARDWARE | + SOF_TIMESTAMPING_RAW_HARDWARE; + + info->tx_types = BIT(HWTSTAMP_TX_OFF) | + BIT(HWTSTAMP_TX_ON); + + info->rx_filters = BIT(HWTSTAMP_FILTER_NONE); + + info->rx_filters |= BIT(HWTSTAMP_FILTER_PTP_V2_L4_EVENT) | + BIT(HWTSTAMP_FILTER_PTP_V2_L2_EVENT) | + BIT(HWTSTAMP_FILTER_PTP_V2_EVENT); + + ptp_clock = atl_ptp_get_ptp_clock(nic); + if (ptp_clock) + info->phc_index = ptp_clock_index(ptp_clock); + + return 0; +} + +struct atl_rxf_flt_desc { + int base; + int max; + uint32_t rxq_bit; + int rxq_shift; + size_t cmd_offt; + size_t count_offt; + int (*get_rxf)(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp); + int (*set_rxf)(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp); + void (*update_rxf)(struct atl_nic *nic, int idx); + int (*check_rxf)(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp); +}; + +#define atl_for_each_rxf_desc(_desc) \ +for (_desc = atl_rxf_descs; \ + _desc < atl_rxf_descs + ARRAY_SIZE(atl_rxf_descs); \ + _desc++) + +#define atl_for_each_rxf_idx(_desc, _idx) \ + for (_idx = 0; _idx < _desc->max; _idx++) + +static inline int atl_rxf_idx(const struct atl_rxf_flt_desc *desc, + struct ethtool_rx_flow_spec *fsp) +{ + return fsp->location - desc->base; +} + +static inline uint64_t atl_ring_cookie(const struct atl_rxf_flt_desc *desc, + uint32_t cmd) +{ + if (cmd & desc->rxq_bit) + return (cmd >> desc->rxq_shift) & ATL_RXF_RXQ_MSK; + else if (cmd & ATL_RXF_ACT_TOHOST) + return ATL_RXF_RING_ANY; + else + return RX_CLS_FLOW_DISC; +} + +static int atl_rxf_get_vlan(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + int idx = atl_rxf_idx(desc, fsp); + uint32_t cmd = vlan->cmd[idx]; + + if (!(cmd & ATL_RXF_EN)) + return -EINVAL; + + fsp->flow_type = ETHER_FLOW | FLOW_EXT; + fsp->h_ext.vlan_tci = htons(cmd & ATL_VLAN_VID_MASK); + fsp->m_ext.vlan_tci = htons(BIT(12) - 1); + fsp->ring_cookie = atl_ring_cookie(desc, cmd); + + return 0; +} + +static int atl_rxf_get_etype(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_etype *etype = &nic->rxf_etype; + int idx = atl_rxf_idx(desc, fsp); + uint32_t cmd = etype->cmd[idx]; + + if (!(cmd & ATL_RXF_EN)) + return -EINVAL; + + fsp->flow_type = ETHER_FLOW; + fsp->m_u.ether_spec.h_proto = 0xffff; + fsp->h_u.ether_spec.h_proto = htons(cmd & ATL_ETYPE_VAL_MASK); + fsp->ring_cookie = atl_ring_cookie(desc, cmd); + + return 0; +} + +static inline void atl_ntuple_swap_v6(__be32 dst[4], __be32 src[4]) +{ + int i; + + for (i = 0; i < 4; i++) + dst[i] = src[3 - i]; +} + +static int atl_rxf_get_ntuple(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_ntuple *ntuples = &nic->rxf_ntuple; + uint32_t idx = atl_rxf_idx(desc, fsp); + uint32_t cmd = ntuples->cmd[idx]; + + if (!(cmd & ATL_RXF_EN)) + return -EINVAL; + +#ifdef ATL_HAVE_IPV6_NTUPLE + if (cmd & ATL_NTC_V6) { + fsp->flow_type = IPV6_USER_FLOW; + } else +#endif + { + fsp->flow_type = IPV4_USER_FLOW; + fsp->h_u.usr_ip4_spec.ip_ver = ETH_RX_NFC_IP4; + } + + if (cmd & ATL_NTC_PROTO) { + switch (cmd & ATL_NTC_L4_MASK) { + case ATL_NTC_L4_TCP: + fsp->flow_type = cmd & ATL_NTC_V6 ? + TCP_V6_FLOW : TCP_V4_FLOW; + break; + + case ATL_NTC_L4_UDP: + fsp->flow_type = cmd & ATL_NTC_V6 ? + UDP_V6_FLOW : UDP_V4_FLOW; + break; + + case ATL_NTC_L4_SCTP: + fsp->flow_type = cmd & ATL_NTC_V6 ? + SCTP_V6_FLOW : SCTP_V4_FLOW; + break; + + case ATL_NTC_L4_ICMP: +#ifdef ATL_HAVE_IPV6_NTUPLE + if (cmd & ATL_NTC_V6) { + fsp->h_u.usr_ip6_spec.l4_proto = IPPROTO_ICMPV6; + fsp->m_u.usr_ip6_spec.l4_proto = 0xff; + } else +#endif + { + fsp->h_u.usr_ip4_spec.proto = IPPROTO_ICMP; + fsp->m_u.usr_ip4_spec.proto = 0xff; + } + break; + + default: + return -EINVAL; + } + } + +#ifdef ATL_HAVE_IPV6_NTUPLE + if (cmd & ATL_NTC_V6) { + struct ethtool_tcpip6_spec *rule = &fsp->h_u.tcp_ip6_spec; + struct ethtool_tcpip6_spec *mask = &fsp->m_u.tcp_ip6_spec; + + if (cmd & ATL_NTC_SA) { + atl_ntuple_swap_v6(rule->ip6src, + ntuples->src_ip6[idx]); + memset(mask->ip6src, 0xff, sizeof(mask->ip6src)); + } + + if (cmd & ATL_NTC_DA) { + atl_ntuple_swap_v6(rule->ip6dst, + ntuples->dst_ip6[idx]); + memset(mask->ip6dst, 0xff, sizeof(mask->ip6dst)); + } + + if (cmd & ATL_NTC_SP) { + rule->psrc = ntuples->src_port[idx]; + mask->psrc = -1; + } + + if (cmd & ATL_NTC_DP) { + rule->pdst = ntuples->dst_port[idx]; + mask->pdst = -1; + } + } else +#endif + { + struct ethtool_tcpip4_spec *rule = &fsp->h_u.tcp_ip4_spec; + struct ethtool_tcpip4_spec *mask = &fsp->m_u.tcp_ip4_spec; + + if (cmd & ATL_NTC_SA) { + rule->ip4src = ntuples->src_ip4[idx]; + mask->ip4src = -1; + } + + if (cmd & ATL_NTC_DA) { + rule->ip4dst = ntuples->dst_ip4[idx]; + mask->ip4dst = -1; + } + + if (cmd & ATL_NTC_SP) { + rule->psrc = ntuples->src_port[idx]; + mask->psrc = -1; + } + + if (cmd & ATL_NTC_DP) { + rule->pdst = ntuples->dst_port[idx]; + mask->pdst = -1; + } + } + + fsp->ring_cookie = atl_ring_cookie(desc, cmd); + + return 0; +} + +static int atl_rxf_get_flex(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_flex *flex = &nic->rxf_flex; + int idx = atl_rxf_idx(desc, fsp); + uint32_t cmd = flex->cmd[idx]; + + if (!(cmd & ATL_RXF_EN)) + return -EINVAL; + + fsp->flow_type = ETHER_FLOW; + fsp->ring_cookie = atl_ring_cookie(desc, cmd); + + return 0; +} + +static int atl_check_mask(uint8_t *mask, int len, uint32_t *cmd, uint32_t flag) +{ + uint8_t first = mask[0]; + uint8_t *p; + + if (first != 0 && first != 0xff) + return -EINVAL; + + for (p = mask; p < &mask[len]; p++) + if (*p != first) + return -EINVAL; + + if (first == 0xff) { + if (cmd) + *cmd |= flag; + else + return -EINVAL; + } + + return 0; +} + +static int atl_rxf_check_ring(struct atl_nic *nic, uint32_t ring) +{ + if (ring > ATL_RXF_RING_ANY) + return -EINVAL; + + if (ring < nic->nvecs || ring == ATL_RXF_RING_ANY) + return 0; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + if (test_bit(ring, &nic->fwd.ring_map[ATL_FWDIR_RX])) + return 0; +#endif + + return -EINVAL; +} + +static int atl_rxf_set_ring(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp, uint32_t *cmd) +{ + uint64_t ring_cookie = fsp->ring_cookie; + uint32_t ring; + + if (ring_cookie == RX_CLS_FLOW_DISC) + return 0; + + ring = ethtool_get_flow_spec_ring(ring_cookie); + if (atl_rxf_check_ring(nic, ring)) { + atl_nic_err("Invalid Rx filter queue %d\n", ring); + return -EINVAL; + } + + if (ethtool_get_flow_spec_ring_vf(ring_cookie)) { + atl_nic_err("Rx filter queue VF must be zero"); + return -EINVAL; + } + + *cmd |= ATL_RXF_ACT_TOHOST; + + if (ring != ATL_RXF_RING_ANY) + *cmd |= ring << desc->rxq_shift | desc->rxq_bit; + + return 0; +} + +static int atl_rxf_check_vlan_etype_common(struct ethtool_rx_flow_spec *fsp) +{ + int ret; + + ret = atl_check_mask((uint8_t *)&fsp->m_u.ether_spec.h_source, + sizeof(fsp->m_u.ether_spec.h_source), NULL, 0); + if (ret) + return ret; + + ret = atl_check_mask((uint8_t *)&fsp->m_ext.data, + sizeof(fsp->m_ext.data), NULL, 0); + if (ret) + return ret; + + ret = atl_check_mask((uint8_t *)&fsp->m_ext.vlan_etype, + sizeof(fsp->m_ext.vlan_etype), NULL, 0); + + return ret; +} + +static int atl_rxf_check_vlan(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + uint16_t vid, mask; + int ret; + + if (fsp->flow_type != (ETHER_FLOW | FLOW_EXT)) { + if (!(fsp->location & RX_CLS_LOC_SPECIAL)) + atl_nic_err("Only ether flow-type supported for VLAN filters\n"); + return -EINVAL; + } + + ret = atl_rxf_check_vlan_etype_common(fsp); + if (ret) + return ret; + + if (fsp->m_u.ether_spec.h_proto) + return -EINVAL; + + vid = ntohs(fsp->h_ext.vlan_tci); + mask = ntohs(fsp->m_ext.vlan_tci); + + if (mask & 0xf000 && vid & 0xf000 & mask) + return -EINVAL; + + if ((mask & 0xfff) != 0xfff) + return -EINVAL; + + return 0; +} + +enum atl_rxf_vlan_idx { + ATL_VIDX_FOUND = BIT(31), + ATL_VIDX_FREE = BIT(30), + ATL_VIDX_REPL = BIT(29), + ATL_VIDX_NONE = BIT(28), + ATL_VIDX_MASK = BIT(28) - 1, +}; + +/* If a filter is enabled for VID, return its index ored with + * ATL_VIDX_FOUND. Otherwise find an unused filter index and return + * it ored with ATL_VIDX_FREE. If no unused filter exists and + * try_repl is set, try finding a candidate for replacement and return + * its index ored with ATL_VIDX_REPL. If all of the above fail, + * return ATL_VIDX_NONE. + * + * A replacement candidate filter must be configured to accept + * packets, not set to direct to a specific ring and must match a VID + * from a VLAN subinterface. + */ +static uint32_t atl_rxf_find_vid(struct atl_nic *nic, uint16_t vid, + bool try_repl) +{ + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + int idx, free = vlan->available, repl = vlan->available; + + for (idx = 0; idx < vlan->available; idx++) { + uint32_t cmd = vlan->cmd[idx]; + + if (!(cmd & ATL_RXF_EN)) { + if (free == vlan->available) { + free = idx; + if (vid == 0xffff) + break; + } + continue; + } + + if ((cmd & ATL_VLAN_VID_MASK) == vid) + return idx | ATL_VIDX_FOUND; + + if (try_repl && repl == vlan->available && + (cmd & ATL_RXF_ACT_TOHOST) && + !(cmd & ATL_VLAN_RXQ)) { + + if (!test_bit(cmd & ATL_VLAN_VID_MASK, vlan->map)) + continue; + + repl = idx; + } + } + + if (free != vlan->available) + return free | ATL_VIDX_FREE; + + if (try_repl && repl != vlan->available) + return repl | ATL_VIDX_REPL; + + return ATL_VIDX_NONE; +} + +static uint16_t atl_rxf_vid(struct atl_rxf_vlan *vlan, int idx) +{ + uint32_t cmd = vlan->cmd[idx]; + + return cmd & ATL_RXF_EN ? cmd & ATL_VLAN_VID_MASK : 0xffff; +} + +static int atl_rxf_dup_vid(struct atl_rxf_vlan *vlan, int idx, uint16_t vid) +{ + int i; + + for (i = 0; i < vlan->available; i++) { + if (i == idx) + continue; + + if (atl_rxf_vid(vlan, i) == vid) + return i; + } + + return -1; +} + +static int atl_rxf_set_vlan(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + int idx; + int ret, promisc_delta = 0; + uint32_t cmd = ATL_RXF_EN; + int present; + uint16_t old_vid, vid = ntohs(fsp->h_ext.vlan_tci) & 0xfff; + + if (!(fsp->location & RX_CLS_LOC_SPECIAL)) { + int dup; + + idx = atl_rxf_idx(desc, fsp); + if (idx >= vlan->available) + return -ENOSPC; + + dup = atl_rxf_dup_vid(vlan, idx, vid); + if (dup >= 0) { + atl_nic_err("Can't add duplicate VLAN filter @%d (existing @%d)\n", + idx, dup); + return -EINVAL; + } + + old_vid = atl_rxf_vid(vlan, idx); + if (old_vid != 0xffff && vid != old_vid && + test_bit(old_vid, vlan->map)) { + atl_nic_err("Can't overwrite Linux VLAN filter @%d VID %hd with a different VID %hd\n", + idx, old_vid, vid); + return -EINVAL; + } + + ret = atl_rxf_check_vlan(desc, nic, fsp); + if (ret) + return ret; + + } else { + /* atl_rxf_check_vlan() already succeeded */ + idx = atl_rxf_find_vid(nic, vid, true); + + if (idx == ATL_VIDX_NONE) + return -EINVAL; + + /* If a filter is being added for a VID without a + * corresponding VLAN subdevice, and we're reusing a + * filter previously used for a VLAN subdevice-covered + * VID, the promisc count needs to be bumped (but + * only if filter change succeeds). */ + if ((idx & ATL_VIDX_REPL) && !test_bit(vid, vlan->map)) + promisc_delta++; + + idx &= ATL_VIDX_MASK; + fsp->location = idx + desc->base; + } + + cmd |= vid; + + ret = atl_rxf_set_ring(desc, nic, fsp, &cmd); + if (ret) + return ret; + + /* If a VLAN subdevice exists, override filter to accept + * packets */ + if (test_bit(vid, vlan->map)) + cmd |= ATL_RXF_ACT_TOHOST; + + present = !!(vlan->cmd[idx] & ATL_RXF_EN); + vlan->cmd[idx] = cmd; + vlan->promisc_count += promisc_delta; + + return !present; +} + +/** Find tag with the same action or new free tag + * top - top inclusive tag value + * action - action for ActionResolverTable + */ +static inline int atl2_filter_tag_get(struct atl2_tag_policy *tags, + int top, u16 action) +{ + int i; + + for (i = 1; i <= top; i++) + if ((tags[i].usage > 0) && (tags[i].action == action)) { + tags[i].usage++; + return i; + } + + for (i = 1; i <= top; i++) + if (tags[i].usage == 0) { + tags[i].usage = 1; + tags[i].action = action; + return i; + } + + return -1; +} + +static inline void atl2_filter_tag_put(struct atl2_tag_policy *tags, int tag) +{ + if (tags[tag].usage > 0) + tags[tag].usage--; +} + +static int atl_rxf_set_etype(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_etype *etype = &nic->rxf_etype; + int idx = atl_rxf_idx(desc, fsp); + int ret; + uint32_t cmd = ATL_RXF_EN; + int present = !!(etype->cmd[idx] & ATL_RXF_EN); + + if (fsp->flow_type != (ETHER_FLOW)) { + atl_nic_err("Only ether flow-type supported for ethertype filters\n"); + return -EINVAL; + } + + ret = atl_rxf_check_vlan_etype_common(fsp); + if (ret) + return ret; + + if (fsp->m_ext.vlan_tci) + return -EINVAL; + + if (fsp->m_u.ether_spec.h_proto != 0xffff) + return -EINVAL; + + if (idx >= etype->available) + return -ENOSPC; + + cmd |= ntohs(fsp->h_u.ether_spec.h_proto); + + ret = atl_rxf_set_ring(desc, nic, fsp, &cmd); + if (ret) + return ret; + + if (nic->hw.new_rpf) { + uint16_t action; + + if (!(cmd & ATL_RXF_ACT_TOHOST)) { + action = ATL2_ACTION_DROP; + } else if (!(cmd & ATL_ETYPE_RXQ)) { + action = ATL2_ACTION_ASSIGN_TC(0); + } else { + int queue = (cmd >> ATL_ETYPE_RXQ_SHIFT) & ATL_RXF_RXQ_MSK; + + action = ATL2_ACTION_ASSIGN_QUEUE(queue); + } + + etype->tag[idx] = atl2_filter_tag_get(etype->tags_policy, + etype->tag_top, + action); + if (etype->tag[idx] < 0) + return -ENOSPC; + } + + etype->cmd[idx] = cmd; + + return !present; +} + +static int atl2_rxf_l3_is_equal(struct atl2_rxf_l3 *f1, struct atl2_rxf_l3 *f2) +{ + if (f1->cmd != f2->cmd) + return false; + + if (f1->cmd & ATL2_NTC_L3_IPV4_SA) + if (f1->src_ip4 != f2->src_ip4) + return false; + + if (f1->cmd & ATL2_NTC_L3_IPV4_DA) + if (f1->dst_ip4 != f2->dst_ip4) + return false; + + if (f1->cmd & (ATL2_NTC_L3_IPV4_PROTO | ATL2_NTC_L3_IPV6_PROTO)) + if (f1->proto != f2->proto) + return false; + + if (f1->cmd & ATL2_NTC_L3_IPV6_SA) + if (memcmp(f1->src_ip6, f2->src_ip6, 16)) + return false; + + if (f1->cmd & ATL2_NTC_L3_IPV6_DA) + if (memcmp(f1->dst_ip6, f2->dst_ip6, 16)) + return false; + + return true; +} + +static int atl2_rxf_l4_is_equal(struct atl2_rxf_l4 *f1, struct atl2_rxf_l4 *f2) +{ + if (f1->cmd != f2->cmd) + return false; + + if (f1->cmd & ATL2_NTC_L4_SP) + if (f1->src_port != f2->src_port) + return false; + + if (f1->cmd & ATL2_NTC_L4_DP) + if (f1->dst_port != f2->dst_port) + return false; + + return true; +} + +static void atl2_rxf_write_l3_cmd(struct atl_hw *hw, int l3_idx, bool is_ipv6, + uint32_t cmd) +{ + uint32_t mask = is_ipv6 ? 0xFF7F0000 : 0x0000FFFF; + uint32_t value = (atl_read(hw, ATL2_RPF_L3_FLT(l3_idx)) & ~mask) | cmd; + + atl_write(hw, ATL2_RPF_L3_FLT(l3_idx), value); +} + +static void atl2_rxf_l3_put(struct atl_hw *hw, struct atl2_rxf_l3 *l3, int idx) +{ + if (l3->usage) + l3->usage--; + + if (!l3->usage) { + atl2_rxf_write_l3_cmd(hw, idx, l3->cmd & ATL2_NTC_L3_IPV6_EN, 0); + l3->cmd = 0; + } +} + +static void atl2_rxf_l3_get(struct atl2_rxf_l3 *l3, int idx, + const struct atl2_rxf_l3 *_l3) +{ + int i; + + l3->usage++; + l3->cmd = _l3->cmd; + for (i = 0; i < 4; i++) { + l3->src_ip6[i] = _l3->src_ip6[i]; + l3->dst_ip6[i] = _l3->dst_ip6[i]; + } + l3->proto = _l3->proto; +} + +static void atl2_rxf_l4_put(struct atl_hw *hw, struct atl2_rxf_l4 *l4, int idx) +{ + if (l4->usage) + l4->usage--; + + if (!l4->usage) { + l4->cmd = 0; + atl_write(hw, ATL2_RPF_L4_FLT(idx), l4->cmd); + } +} + +static void atl2_rxf_l4_get(struct atl2_rxf_l4 *l4, int idx, + const struct atl2_rxf_l4 *_l4) +{ + l4->usage++; + l4->cmd = _l4->cmd; + l4->src_port = _l4->src_port; + l4->dst_port = _l4->dst_port; +} + +static void atl2_rxf_configure_l3l4(struct atl_rxf_ntuple *ntuple, int idx, + struct atl2_rxf_l3 *l3, + struct atl2_rxf_l4 *l4) +{ + if (ntuple->cmd[idx] & ATL_NTC_PROTO) + l3->cmd |= ntuple->cmd[idx] & ATL_NTC_V6 ? + ATL2_NTC_L3_IPV6_PROTO | ATL2_NTC_L3_IPV6_EN : + ATL2_NTC_L3_IPV4_PROTO | ATL2_NTC_L3_IPV4_EN; + + switch (ntuple->cmd[idx] & ATL_NTC_L4_MASK) { + case ATL_NTC_L4_TCP: + l3->cmd |= ntuple->cmd[idx] & ATL_NTC_V6 ? + IPPROTO_TCP << ATL2_NTC_L3_IPV6_PROTO_SHIFT : + IPPROTO_TCP << ATL2_NTC_L3_IPV4_PROTO_SHIFT; + break; + + case ATL_NTC_L4_UDP: + l3->cmd |= ntuple->cmd[idx] & ATL_NTC_V6 ? + IPPROTO_UDP << ATL2_NTC_L3_IPV6_PROTO_SHIFT : + IPPROTO_UDP << ATL2_NTC_L3_IPV4_PROTO_SHIFT; + break; + + case ATL_NTC_L4_SCTP: + l3->cmd |= ntuple->cmd[idx] & ATL_NTC_V6 ? + IPPROTO_SCTP << ATL2_NTC_L3_IPV6_PROTO_SHIFT : + IPPROTO_SCTP << ATL2_NTC_L3_IPV4_PROTO_SHIFT; + break; + + case ATL_NTC_L4_ICMP: +#ifdef ATL_HAVE_IPV6_NTUPLE + l3->cmd |= ntuple->cmd[idx] & ATL_NTC_V6 ? + IPPROTO_ICMPV6 << ATL2_NTC_L3_IPV6_PROTO_SHIFT : + IPPROTO_ICMP << ATL2_NTC_L3_IPV4_PROTO_SHIFT; +#else + l3->cmd |= IPPROTO_ICMP << ATL2_NTC_L3_IPV4_PROTO_SHIFT; +#endif + break; + + } + + if (ntuple->cmd[idx] & ATL_NTC_SA) { + if (ntuple->cmd[idx] & ATL_NTC_V6) { + l3->cmd |= ATL2_NTC_L3_IPV6_SA | ATL2_NTC_L3_IPV6_EN; + memcpy(l3->src_ip6, ntuple->src_ip6[idx], 16); + } else { + l3->cmd |= ATL2_NTC_L3_IPV4_SA | ATL2_NTC_L3_IPV4_EN; + l3->src_ip4 = ntuple->src_ip4[idx]; + } + } + if (ntuple->cmd[idx] & ATL_NTC_DA) { + if (ntuple->cmd[idx] & ATL_NTC_V6) { + l3->cmd |= ATL2_NTC_L3_IPV6_DA | ATL2_NTC_L3_IPV6_EN; + memcpy(l3->dst_ip6, ntuple->dst_ip6[idx], 16); + } else { + l3->cmd |= ATL2_NTC_L3_IPV4_DA | ATL2_NTC_L3_IPV4_EN; + l3->dst_ip4 = ntuple->dst_ip4[idx]; + } + } + if (ntuple->cmd[idx] & ATL_NTC_SP) { + l4->cmd |= ATL2_NTC_L4_SP | ATL2_NTC_L4_EN; + l4->src_port = ntuple->src_port[idx]; + } + if (ntuple->cmd[idx] & ATL_NTC_DP) { + l4->cmd |= ATL2_NTC_L4_DP | ATL2_NTC_L4_EN; + l4->dst_port = ntuple->dst_port[idx]; + } +} + +static int atl2_rxf_fl3l4_find_l3(struct atl_rxf_ntuple *ntuple, + struct atl2_rxf_l3 *l3) +{ + struct atl2_rxf_l3 *nl3 = (l3->cmd & ATL2_NTC_L3_IPV4_EN) ? + ntuple->l3v4 : ntuple->l3v6; + int first = (l3->cmd & ATL2_NTC_L3_IPV4_EN) ? + ntuple->l3_v4_base_index : + ntuple->l3_v6_base_index; + int last = first + (l3->cmd & ATL2_NTC_L3_IPV4_EN) ? + ntuple->l3_v4_available : + ntuple->l3_v6_available; + int l3_idx = -1; + int i; + + for (i = first; i < last; i++) { + if (atl2_rxf_l3_is_equal(&nl3[i], l3)) { + l3_idx = i; + break; + } + } + if (l3_idx < 0) + for (i = first; i < last; i++) + if ((nl3[i].cmd & (ATL2_NTC_L3_IPV4_EN | + ATL2_NTC_L3_IPV6_EN)) == 0) { + l3_idx = i; + break; + } + if (l3_idx < 0) + return -ENOSPC; + + return l3_idx; +} + +static int atl2_rxf_fl3l4_find_l4(struct atl_rxf_ntuple *ntuple, + struct atl2_rxf_l4 *l4) +{ + int l4_idx = -1; + int i; + + for (i = ntuple->l4_base_index; i < ntuple->l4_available; i++) { + if (atl2_rxf_l4_is_equal(&ntuple->l4[i], l4)) + l4_idx = i; + } + if (l4_idx >= 0) + return l4_idx; + + for (i = ntuple->l4_base_index; i < ntuple->l4_available; i++) { + if ((ntuple->l4[i].cmd & ATL2_NTC_L4_EN) == 0) { + l4_idx = i; + break; + } + } + if (l4_idx < 0) + return -ENOSPC; + return l4_idx; +} + +static int atl2_rxf_set_ntuple(struct atl_nic *nic, + struct atl_rxf_ntuple *ntuple, + int idx) +{ + struct atl2_rxf_l3 l3; + struct atl2_rxf_l4 l4; + struct atl2_rxf_l3 *l3_filters; + s8 l3_idx = -1; + s8 l4_idx = -1; + + memset(&l3, 0, sizeof(l3)); + memset(&l4, 0, sizeof(l4)); + atl2_rxf_configure_l3l4(ntuple, idx, &l3, &l4); + + /* find L3 and L4 filters */ + if (l3.cmd & (ATL2_NTC_L3_IPV4_EN | ATL2_NTC_L3_IPV6_EN)) { + l3_idx = atl2_rxf_fl3l4_find_l3(ntuple, &l3); + if (l3_idx < 0) + return l3_idx; + } + + if (l4.cmd & ATL2_NTC_L4_EN) { + l4_idx = atl2_rxf_fl3l4_find_l4(ntuple, &l4); + if (l4_idx < 0) + return l4_idx; + + if (ntuple->l4_idx[idx] != l4_idx) + atl2_rxf_l4_get(&ntuple->l4[l4_idx], l4_idx, &l4); + } + + if (l3.cmd & (ATL2_NTC_L3_IPV4_EN | ATL2_NTC_L3_IPV6_EN)) { + if (l3.cmd & ATL2_NTC_L3_IPV4_EN) + l3_filters = ntuple->l3v4; + else + l3_filters = ntuple->l3v6; + + if (ntuple->l3_idx[idx] != l3_idx) + atl2_rxf_l3_get(&l3_filters[l3_idx], l3_idx, &l3); + } + + /* release old filter */ + if (ntuple->l3_idx[idx] != -1) { + if (ntuple->is_ipv6[idx]) + l3_filters = ntuple->l3v6; + else + l3_filters = ntuple->l3v4; + + if (!(atl2_rxf_l3_is_equal(&l3, + &l3_filters[ntuple->l3_idx[idx]]))) { + atl2_rxf_l3_put(&nic->hw, + &l3_filters[ntuple->l3_idx[idx]], + ntuple->l3_idx[idx]); + } + } + ntuple->l3_idx[idx] = l3_idx; + + if (ntuple->l4_idx[idx] != -1) + if (!(atl2_rxf_l4_is_equal(&l4, + &ntuple->l4[ntuple->l4_idx[idx]]))) { + atl2_rxf_l4_put(&nic->hw, + &ntuple->l4[ntuple->l4_idx[idx]], + ntuple->l4_idx[idx]); + } + ntuple->l4_idx[idx] = l4_idx; + + ntuple->is_ipv6[idx] = (l3.cmd & ATL2_NTC_L3_IPV4_EN) ? false : true; + + return 0; +} + +static int atl_rxf_set_ntuple(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_ntuple *ntuple = &nic->rxf_ntuple; + int idx = atl_rxf_idx(desc, fsp); + uint32_t cmd = ATL_NTC_EN; + int ret; + __be16 sport, dport; + int present = !!(ntuple->cmd[idx] & ATL_RXF_EN); + + ret = atl_rxf_set_ring(desc, nic, fsp, &cmd); + if (ret) + return ret; + + switch (fsp->flow_type) { +#ifdef ATL_HAVE_IPV6_NTUPLE + case TCP_V6_FLOW: + case UDP_V6_FLOW: + case SCTP_V6_FLOW: + if (fsp->m_u.tcp_ip6_spec.tclass != 0) { + atl_nic_err("Unsupported match field\n"); + return -EINVAL; + } + cmd |= ATL_NTC_PROTO | ATL_NTC_V6; + break; + + case IPV6_USER_FLOW: + if (fsp->m_u.usr_ip6_spec.l4_4_bytes != 0 || + fsp->m_u.usr_ip6_spec.tclass != 0) { + atl_nic_err("Unsupported match field\n"); + return -EINVAL; + } + + if (fsp->h_u.usr_ip6_spec.l4_proto == IPPROTO_ICMPV6) { + cmd |= ATL_NTC_L4_ICMP | ATL_NTC_PROTO; + } else if (fsp->m_u.usr_ip6_spec.l4_proto != 0) { + atl_nic_err("Unsupported match field\n"); + return -EINVAL; + } + + cmd |= ATL_NTC_V6; + break; +#endif + + case TCP_V4_FLOW: + case UDP_V4_FLOW: + case SCTP_V4_FLOW: + if (fsp->m_u.tcp_ip4_spec.tos != 0) { + atl_nic_err("Unsupported match field\n"); + return -EINVAL; + } + cmd |= ATL_NTC_PROTO; + break; + + case IPV4_USER_FLOW: + if (fsp->m_u.usr_ip4_spec.l4_4_bytes != 0 || + fsp->m_u.usr_ip4_spec.tos != 0 || + fsp->h_u.usr_ip4_spec.ip_ver != ETH_RX_NFC_IP4) { + atl_nic_err("Unsupported match field\n"); + return -EINVAL; + } + + if (fsp->h_u.usr_ip4_spec.proto == IPPROTO_ICMP) { + cmd |= ATL_NTC_L4_ICMP | ATL_NTC_PROTO; + } else if (fsp->m_u.usr_ip4_spec.proto != 0) { + atl_nic_err("Unsupported match field\n"); + return -EINVAL; + } + break; + + default: + return -EINVAL; + } + + switch (fsp->flow_type) { + case TCP_V6_FLOW: + case TCP_V4_FLOW: + cmd |= ATL_NTC_L4_TCP; + break; + + case UDP_V6_FLOW: + case UDP_V4_FLOW: + cmd |= ATL_NTC_L4_UDP; + break; + + case SCTP_V6_FLOW: + case SCTP_V4_FLOW: + cmd |= ATL_NTC_L4_SCTP; + break; + } + +#ifdef ATL_HAVE_IPV6_NTUPLE + if (cmd & ATL_NTC_V6) { + int i; + + if (!nic->hw.new_rpf) { + if (idx & 3) { + atl_nic_err("IPv6 filters only supported in locations 8 and 12\n"); + return -EINVAL; + } + + for (i = idx + 1; i < idx + 4; i++) + if (ntuple->cmd[i] & ATL_NTC_EN) { + atl_nic_err("IPv6 filter %d overlaps an IPv4 filter %d\n", + idx, i); + return -EINVAL; + } + } + + ret = atl_check_mask((uint8_t *)fsp->m_u.tcp_ip6_spec.ip6src, + sizeof(fsp->m_u.tcp_ip6_spec.ip6src), &cmd, ATL_NTC_SA); + if (ret) + return ret; + + ret = atl_check_mask((uint8_t *)fsp->m_u.tcp_ip6_spec.ip6dst, + sizeof(fsp->m_u.tcp_ip6_spec.ip6dst), &cmd, ATL_NTC_DA); + if (ret) + return ret; + + sport = fsp->h_u.tcp_ip6_spec.psrc; + ret = atl_check_mask((uint8_t *)&fsp->m_u.tcp_ip6_spec.psrc, + sizeof(fsp->m_u.tcp_ip6_spec.psrc), &cmd, ATL_NTC_SP); + if (ret) + return ret; + + dport = fsp->h_u.tcp_ip6_spec.pdst; + ret = atl_check_mask((uint8_t *)&fsp->m_u.tcp_ip6_spec.pdst, + sizeof(fsp->m_u.tcp_ip6_spec.pdst), &cmd, ATL_NTC_DP); + if (ret) + return ret; + } else +#endif + { + + ret = atl_check_mask((uint8_t *)&fsp->m_u.tcp_ip4_spec.ip4src, + sizeof(fsp->m_u.tcp_ip4_spec.ip4src), &cmd, ATL_NTC_SA); + if (ret) + return ret; + + ret = atl_check_mask((uint8_t *)&fsp->m_u.tcp_ip4_spec.ip4dst, + sizeof(fsp->m_u.tcp_ip4_spec.ip4dst), &cmd, ATL_NTC_DA); + if (ret) + return ret; + + sport = fsp->h_u.tcp_ip4_spec.psrc; + ret = atl_check_mask((uint8_t *)&fsp->m_u.tcp_ip4_spec.psrc, + sizeof(fsp->m_u.tcp_ip4_spec.psrc), &cmd, ATL_NTC_SP); + if (ret) + return ret; + + dport = fsp->h_u.tcp_ip4_spec.pdst; + ret = atl_check_mask((uint8_t *)&fsp->m_u.tcp_ip4_spec.pdst, + sizeof(fsp->m_u.tcp_ip4_spec.psrc), &cmd, ATL_NTC_DP); + if (ret) + return ret; + } + +#ifdef ATL_HAVE_IPV6_NTUPLE + if (cmd & ATL_NTC_V6) { + if (cmd & ATL_NTC_SA) + atl_ntuple_swap_v6(ntuple->src_ip6[idx], + fsp->h_u.tcp_ip6_spec.ip6src); + + if (cmd & ATL_NTC_DA) + atl_ntuple_swap_v6(ntuple->dst_ip6[idx], + fsp->h_u.tcp_ip6_spec.ip6dst); + } else +#endif + { + if (cmd & ATL_NTC_SA) + ntuple->src_ip4[idx] = fsp->h_u.tcp_ip4_spec.ip4src; + + if (cmd & ATL_NTC_DA) + ntuple->dst_ip4[idx] = fsp->h_u.tcp_ip4_spec.ip4dst; + } + + + if (cmd & ATL_NTC_SP) + ntuple->src_port[idx] = sport; + + if (cmd & ATL_NTC_DP) + ntuple->dst_port[idx] = dport; + + ntuple->cmd[idx] = cmd; + + if (nic->hw.new_rpf) { + ret = atl2_rxf_set_ntuple(nic, ntuple, idx); + if (ret < 0) + return ret; + } + + return !present; +} + +static int atl_rxf_set_flex(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_flex *flex = &nic->rxf_flex; + int idx = atl_rxf_idx(desc, fsp); + int ret; + uint32_t cmd = ATL_RXF_EN; + int present = !!(flex->cmd[idx] & ATL_RXF_EN);; + + ret = atl_rxf_set_ring(desc, nic, fsp, &cmd); + if (ret) + return ret; + + flex->cmd[idx] = cmd; + + return !present; +} + +static void atl_rxf_update_vlan(struct atl_nic *nic, int idx) +{ + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + uint32_t cmd = vlan->cmd[idx]; + struct atl_hw *hw = &nic->hw; + u16 action; + + atl_write(&nic->hw, ATL_RX_VLAN_FLT(vlan->base_index + idx), cmd); + + if (!nic->hw.new_rpf) + return; + + if (!(cmd & ATL_RXF_EN)) { + atl2_act_rslvr_table_set(hw, + hw->art_base_index + ATL2_RPF_VLAN_USER_INDEX + idx, + 0, + 0, + ATL2_ACTION_DISABLE); + return; + } + + if (!(cmd & ATL_RXF_ACT_TOHOST)) { + action = ATL2_ACTION_DROP; + } else if (!(cmd & ATL_VLAN_RXQ)) { + atl2_rpf_vlan_flr_tag_set(hw, 1, vlan->base_index + idx); + return; + } else { + int queue = (cmd >> ATL_VLAN_RXQ_SHIFT) & ATL_RXF_RXQ_MSK; + + action = ATL2_ACTION_ASSIGN_QUEUE(queue); + } + + atl2_rpf_vlan_flr_tag_set(hw, idx + 2, vlan->base_index + idx); + atl2_act_rslvr_table_set(hw, + hw->art_base_index + ATL2_RPF_VLAN_USER_INDEX + idx, + (idx + 2) << ATL2_RPF_TAG_VLAN_OFFSET, + ATL2_RPF_TAG_VLAN_MASK, + action); +} + +static void atl_rxf_update_etype(struct atl_nic *nic, int idx) +{ + struct atl_rxf_etype *etype = &nic->rxf_etype; + uint32_t cmd = etype->cmd[idx]; + struct atl_hw *hw = &nic->hw; + u16 action, index; + + atl_write(&nic->hw, ATL_RX_ETYPE_FLT(etype->base_index + idx), cmd); + + if (!nic->hw.new_rpf) + return; + + if (!(cmd & ATL_RXF_EN)) { + atl2_filter_tag_put(etype->tags_policy, + etype->tag[idx]); + index = hw->art_base_index + + ATL2_RPF_ET_PCP_USER_INDEX + idx; + atl2_act_rslvr_table_set(hw, + index, + 0, + 0, + ATL2_ACTION_DISABLE); + return; + } + + atl2_rpf_etht_flr_tag_set(hw, etype->tag[idx], etype->base_index + idx); + action = etype->tags_policy[etype->tag[idx]].action; + index = hw->art_base_index + ATL2_RPF_ET_PCP_USER_INDEX + idx; + atl2_act_rslvr_table_set(hw, + index, + etype->tag[idx] << ATL2_RPF_TAG_ET_OFFSET, + ATL2_RPF_TAG_ET_MASK, + action); +} + +static void atl2_update_ntuple_flt(struct atl_nic *nic, int idx) +{ + struct atl_hw *hw = &nic->hw; + struct atl_rxf_ntuple *ntuple = &nic->rxf_ntuple; + uint32_t tag = 0, mask = 0, action, cmd; + struct atl2_rxf_l3 *l3_filters; + struct atl2_rxf_l3 *l3 = NULL; + struct atl2_rxf_l4 *l4 = NULL; + s8 l3_idx = ntuple->l3_idx[idx]; + s8 l4_idx = ntuple->l4_idx[idx]; + bool is_ipv6 = ntuple->is_ipv6[idx]; + + l3_filters = ntuple->is_ipv6[idx] ? ntuple->l3v6 : ntuple->l3v4; + if (!(ntuple->cmd[idx] & ATL_NTC_EN)) { + if (l3_idx > -1) + atl2_rxf_l3_put(hw, &l3_filters[l3_idx], l3_idx); + + if (l4_idx > -1) + atl2_rxf_l4_put(hw, &ntuple->l4[l4_idx], l4_idx); + + ntuple->l4_idx[idx] = -1; + ntuple->l3_idx[idx] = -1; + atl2_act_rslvr_table_set(hw, + hw->art_base_index + ATL2_RPF_L3L4_USER_INDEX + idx, + 0, + 0, + ATL2_ACTION_DISABLE); + + return; + } + if (l3_idx > -1) { + l3 = &l3_filters[l3_idx]; + cmd = l3->cmd; + if (l3->cmd & ATL2_NTC_L3_IPV4_EN) { + tag |= (l3_idx + 1) << ATL2_RPF_TAG_L3_V4_OFFSET; + mask |= ATL2_RPF_TAG_L3_V4_MASK; + cmd |= (l3_idx + 1) << 0x4; + + if (l3->cmd & ATL2_NTC_L3_IPV4_SA) + atl2_rpf_l3_v4_sa_set(hw, l3_idx, l3->src_ip4); + if (l3->cmd & ATL2_NTC_L3_IPV4_DA) + atl2_rpf_l3_v4_da_set(hw, l3_idx, l3->dst_ip4); + } else if (l3->cmd & ATL2_NTC_L3_IPV6_EN) { + tag |= (l3_idx + 1) << ATL2_RPF_TAG_L3_V6_OFFSET; + mask |= ATL2_RPF_TAG_L3_V6_MASK; + cmd |= (l3_idx + 1) << 0x14; + + if (l3->cmd & ATL2_NTC_L3_IPV6_SA) + atl2_rpf_l3_v6_sa_set(hw, l3_idx, l3->src_ip6); + if (l3->cmd & ATL2_NTC_L3_IPV6_DA) + atl2_rpf_l3_v6_da_set(hw, l3_idx, l3->dst_ip6); + } else { + WARN(1, "L3 filter invalid"); + return; + } + + atl2_rxf_write_l3_cmd(hw, l3_idx, is_ipv6, cmd); + } + + if (l4_idx > -1) { + l4 = &ntuple->l4[l4_idx]; + if (l4->cmd & ATL2_NTC_L4_EN) { + tag |= (l4_idx + 1) << ATL2_RPF_TAG_L4_OFFSET; + mask |= ATL2_RPF_TAG_L4_MASK; + } else { + WARN(1, "L4 filter invalid"); + return; + } + cmd = l4->cmd | (l4_idx + 1) << 0x4; + atl_write(hw, ATL_NTUPLE_SPORT(l4_idx), + swab16(l4->src_port)); + atl_write(hw, ATL_NTUPLE_DPORT(l4_idx), + swab16(l4->dst_port)); + atl_write(hw, ATL2_RPF_L4_FLT(l4_idx), cmd); + } + + if (!(ntuple->cmd[idx] & ATL_RXF_ACT_TOHOST)) { + action = ATL2_ACTION_DROP; + } else if (!(ntuple->cmd[idx] & ATL_NTC_RXQ)) { + action = ATL2_ACTION_ASSIGN_TC(0); + } else { + int queue = (ntuple->cmd[idx] >> ATL_NTC_RXQ_SHIFT) & ATL_RXF_RXQ_MSK; + + action = ATL2_ACTION_ASSIGN_QUEUE(queue); + } + + atl2_act_rslvr_table_set(hw, + hw->art_base_index + ATL2_RPF_L3L4_USER_INDEX + idx, + tag, + mask, + action); +} + +void atl_update_ntuple_flt(struct atl_nic *nic, int idx) +{ + struct atl_hw *hw = &nic->hw; + struct atl_rxf_ntuple *ntuple = &nic->rxf_ntuple; + uint32_t cmd = ntuple->cmd[idx]; + int i; + + if (nic->hw.new_rpf) + return atl2_update_ntuple_flt(nic, idx); + + if (!(cmd & ATL_NTC_EN)) { + atl_write(hw, ATL_NTUPLE_CTRL(idx), cmd); + + return; + } + + if (cmd & ATL_NTC_V6) { + for (i = 0; i < 4; i++) { + if (cmd & ATL_NTC_SA) + atl_write(hw, ATL_NTUPLE_SADDR(idx + i), + swab32(ntuple->src_ip6[idx][i])); + + if (cmd & ATL_NTC_DA) + atl_write(hw, ATL_NTUPLE_DADDR(idx + i), + swab32(ntuple->dst_ip6[idx][i])); + } + } else { + if (cmd & ATL_NTC_SA) + atl_write(hw, ATL_NTUPLE_SADDR(idx), + swab32(ntuple->src_ip4[idx])); + + if (cmd & ATL_NTC_DA) + atl_write(hw, ATL_NTUPLE_DADDR(idx), + swab32(ntuple->dst_ip4[idx])); + } + + /* ports are used by both new RPF and legacy RPF, but with different + * locations + */ + if (!nic->hw.new_rpf) { + if (cmd & ATL_NTC_SP) + atl_write(hw, ATL_NTUPLE_SPORT(idx), + swab16(ntuple->src_port[idx])); + + if (cmd & ATL_NTC_DP) + atl_write(hw, ATL_NTUPLE_DPORT(idx), + swab16(ntuple->dst_port[idx])); + } + + if (cmd & ATL_NTC_RXQ) + cmd |= 1 << ATL_NTC_ACT_SHIFT; + + atl_write(hw, ATL_NTUPLE_CTRL(idx), cmd); +} + +static void atl_rxf_update_flex(struct atl_nic *nic, int idx) +{ + atl_write(&nic->hw, + ATL_RX_FLEX_FLT_CTRL(nic->rxf_flex.base_index + idx), + nic->rxf_flex.cmd[idx]); + + if (nic->hw.new_rpf) { + uint32_t action; + + atl2_rpf_flex_flr_tag_set(&nic->hw, idx + 1, + nic->rxf_flex.base_index + idx); + + if (!(nic->rxf_flex.cmd[idx] & ATL_FLEX_EN)) { + action = ATL2_ACTION_DISABLE; + } else if (!(nic->rxf_flex.cmd[idx] & ATL_RXF_ACT_TOHOST)) { + action = ATL2_ACTION_DROP; + } else if (!(nic->rxf_flex.cmd[idx] & ATL_FLEX_RXQ)) { + action = ATL2_ACTION_ASSIGN_TC(0); + } else { + int queue = (nic->rxf_flex.cmd[idx] >> ATL_FLEX_RXQ_SHIFT) & ATL_RXF_RXQ_MSK; + + action = ATL2_ACTION_ASSIGN_QUEUE(queue); + } + atl2_act_rslvr_table_set(&nic->hw, + nic->hw.art_base_index + ATL2_RPF_FLEX_USER_INDEX + idx, + (idx + 1) << ATL2_RPF_TAG_FLEX_OFFSET, + ATL2_RPF_TAG_FLEX_MASK, + action); + } +} + +static struct atl_rxf_flt_desc atl_rxf_descs[] = { + { + .base = ATL_RXF_VLAN_BASE, + .max = ATL_RXF_VLAN_MAX, + .rxq_bit = ATL_VLAN_RXQ, + .rxq_shift = ATL_VLAN_RXQ_SHIFT, + .cmd_offt = offsetof(struct atl_nic, rxf_vlan.cmd), + .count_offt = offsetof(struct atl_nic, rxf_vlan.count), + .get_rxf = atl_rxf_get_vlan, + .set_rxf = atl_rxf_set_vlan, + .update_rxf = atl_rxf_update_vlan, + .check_rxf = atl_rxf_check_vlan, + }, + { + .base = ATL_RXF_ETYPE_BASE, + .max = ATL_RXF_ETYPE_MAX, + .rxq_bit = ATL_ETYPE_RXQ, + .rxq_shift = ATL_ETYPE_RXQ_SHIFT, + .cmd_offt = offsetof(struct atl_nic, rxf_etype.cmd), + .count_offt = offsetof(struct atl_nic, rxf_etype.count), + .get_rxf = atl_rxf_get_etype, + .set_rxf = atl_rxf_set_etype, + .update_rxf = atl_rxf_update_etype, + }, + { + .base = ATL_RXF_NTUPLE_BASE, + .max = ATL_RXF_NTUPLE_MAX, + .rxq_bit = ATL_NTC_RXQ, + .rxq_shift = ATL_NTC_RXQ_SHIFT, + .cmd_offt = offsetof(struct atl_nic, rxf_ntuple.cmd), + .count_offt = offsetof(struct atl_nic, rxf_ntuple.count), + .get_rxf = atl_rxf_get_ntuple, + .set_rxf = atl_rxf_set_ntuple, + .update_rxf = atl_update_ntuple_flt, + }, + { + .base = ATL_RXF_FLEX_BASE, + .max = ATL_RXF_FLEX_MAX, + .rxq_bit = ATL_FLEX_RXQ, + .rxq_shift = ATL_FLEX_RXQ_SHIFT, + .cmd_offt = offsetof(struct atl_nic, rxf_flex.cmd), + .count_offt = offsetof(struct atl_nic, rxf_flex.count), + .get_rxf = atl_rxf_get_flex, + .set_rxf = atl_rxf_set_flex, + .update_rxf = atl_rxf_update_flex, + }, +}; + +s8 atl_reserve_filter(enum atl_rxf_type type) +{ + switch (type) { + case ATL_RXF_ETYPE: + WARN_ONCE(atl_rxf_descs[type].max != ATL_RXF_ETYPE_MAX, + "already reserved"); + atl_rxf_descs[type].max--; + return atl_rxf_descs[type].max; + case ATL_RXF_NTUPLE: + WARN_ONCE(atl_rxf_descs[type].max != ATL_RXF_NTUPLE_MAX, + "already reserved"); + atl_rxf_descs[type].max--; + return atl_rxf_descs[type].max; + default: + WARN_ONCE(true, "unexpected type"); + break; + } + + return -1; +} + +void atl_release_filter(enum atl_rxf_type type) +{ + switch (type) { + case ATL_RXF_ETYPE: + WARN_ONCE(atl_rxf_descs[type].max == ATL_RXF_ETYPE_MAX, + "already released"); + atl_rxf_descs[type].max++; + break; + case ATL_RXF_NTUPLE: + WARN_ONCE(atl_rxf_descs[type].max == ATL_RXF_NTUPLE_MAX, + "already released"); + atl_rxf_descs[type].max++; + break; + default: + WARN_ONCE(true, "unexpected type"); + break; + } +} + +static uint32_t *atl_rxf_cmd(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic) +{ + return (uint32_t *)((char *)nic + desc->cmd_offt); +} + +static int *atl_rxf_count(const struct atl_rxf_flt_desc *desc, struct atl_nic *nic) +{ + return (int *)((char *)nic + desc->count_offt); +} + +static const struct atl_rxf_flt_desc *atl_rxf_desc(struct atl_nic *nic, + struct ethtool_rx_flow_spec *fsp) +{ + uint32_t loc = fsp->location; + const struct atl_rxf_flt_desc *desc; + + atl_for_each_rxf_desc(desc) { + if (loc & RX_CLS_LOC_SPECIAL) { + if (desc->check_rxf && !desc->check_rxf(desc, nic, fsp)) + return desc; + + continue; + } + + if (loc < desc->base) + return NULL; + + if (loc < desc->base + desc->max) + return desc; + } + + return NULL; +} + +static void atl_refresh_rxf_desc(struct atl_nic *nic, + const struct atl_rxf_flt_desc *desc) +{ + int idx; + + atl_for_each_rxf_idx(desc, idx) + desc->update_rxf(nic, idx); + + atl_set_vlan_promisc(&nic->hw, (nic->ndev->flags & IFF_PROMISC) || + nic->rxf_vlan.promisc_count || + !nic->rxf_vlan.vlans_active); +} + +void atl_refresh_rxfs(struct atl_nic *nic) +{ + const struct atl_rxf_flt_desc *desc; + + atl_for_each_rxf_desc(desc) + atl_refresh_rxf_desc(nic, desc); + + atl_set_vlan_promisc(&nic->hw, (nic->ndev->flags & IFF_PROMISC) || + nic->rxf_vlan.promisc_count || + !nic->rxf_vlan.vlans_active); +} + +static bool atl_vlan_pull_from_promisc(struct atl_nic *nic, uint32_t idx) +{ + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + unsigned long *map; + int i; + long vid = -1; + + if (!vlan->promisc_count) + return false; + + map = kcalloc(ATL_VID_MAP_LEN, sizeof(*map), GFP_KERNEL); + if (!map) + return false; + + memcpy(map, vlan->map, ATL_VID_MAP_LEN * sizeof(*map)); + for (i = 0; i < vlan->available; i++) { + uint32_t cmd = vlan->cmd[i]; + + if (cmd & ATL_RXF_EN) + clear_bit(cmd & ATL_VLAN_VID_MASK, map); + } + + do { + idx &= ATL_VIDX_MASK; + vid = find_next_bit(map, BIT(12), vid + 1); + vlan->cmd[idx] = ATL_RXF_EN | ATL_RXF_ACT_TOHOST | vid; + atl_rxf_update_vlan(nic, idx); + __clear_bit(vid, map); + vlan->promisc_count--; + vlan->count++; + if (vlan->promisc_count == 0) + break; + + idx = atl_rxf_find_vid(nic, -1, false); + } while (idx & ATL_VIDX_FREE); + + kfree(map); + atl_set_vlan_promisc(&nic->hw, (nic->ndev->flags & IFF_PROMISC) || + vlan->promisc_count || + !vlan->vlans_active); + return true; +} + +static bool atl_rxf_del_vlan_override(const struct atl_rxf_flt_desc *desc, + struct atl_nic *nic, struct ethtool_rx_flow_spec *fsp) +{ + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + uint32_t *cmd = &vlan->cmd[atl_rxf_idx(desc, fsp)]; + uint16_t vid = *cmd & ATL_VLAN_VID_MASK; + + if (!test_bit(vid, vlan->map)) + return false; + + /* Trying to delete filter via ethtool while VLAN subdev still + * exists. Just drop queue assignment. */ + *cmd &= ~ATL_VLAN_RXQ; + return true; +} + +static int atl_set_rxf(struct atl_nic *nic, + struct ethtool_rx_flow_spec *fsp, bool delete) +{ + const struct atl_rxf_flt_desc *desc; + uint32_t *cmd; + int *count, ret, idx; + + desc = atl_rxf_desc(nic, fsp); + if (!desc) + return -EINVAL; + + count = atl_rxf_count(desc, nic); + + if (delete) { + idx = atl_rxf_idx(desc, fsp); + cmd = &atl_rxf_cmd(desc, nic)[idx]; + + if (!(*cmd & ATL_RXF_EN)) + /* Attempting to delete non-existent filter */ + return -EINVAL; + + if (desc->base == ATL_RXF_VLAN_BASE && + atl_rxf_del_vlan_override(desc, nic, fsp)) + goto done; + + *cmd = 0; + (*count)--; + + if (desc->base == ATL_RXF_VLAN_BASE && + atl_vlan_pull_from_promisc(nic, idx)) + /* Filter already updated by + * atl_vlan_pull_from_promisc(), can just + * return */ + return 0; + } else { + ret = desc->set_rxf(desc, nic, fsp); + if (ret < 0) + return ret; + + /* fsp->location may have been set in + * ->set_rxf(). Guaranteed to be valid now. */ + idx = atl_rxf_idx(desc, fsp); + *count += ret; + } + +done: + desc->update_rxf(nic, idx); + return 0; +} + +static void atl_get_rxf_count(struct atl_nic *nic, struct ethtool_rxnfc *rxnfc) +{ + int count = 0, max = 0; + const struct atl_rxf_flt_desc *desc; + + atl_for_each_rxf_desc(desc) { + count += *atl_rxf_count(desc, nic); + max += desc->max; + } + + rxnfc->rule_cnt = count; + rxnfc->data = max | RX_CLS_LOC_SPECIAL; +} + +static int atl_get_rxf_locs(struct atl_nic *nic, struct ethtool_rxnfc *rxnfc, + uint32_t *rule_locs) +{ + int count = 0; + int i; + const struct atl_rxf_flt_desc *desc; + + atl_for_each_rxf_desc(desc) + count += *atl_rxf_count(desc, nic); + + if (rxnfc->rule_cnt < count) + return -EMSGSIZE; + + atl_for_each_rxf_desc(desc) { + uint32_t *cmd = atl_rxf_cmd(desc, nic); + + atl_for_each_rxf_idx(desc, i) + if (cmd[i] & ATL_RXF_EN) + *rule_locs++ = i + desc->base; + + } + + rxnfc->rule_cnt = count; + return 0; +} + +static int atl_get_rxnfc(struct net_device *ndev, struct ethtool_rxnfc *rxnfc, + uint32_t *rule_locs) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct ethtool_rx_flow_spec *fsp = &rxnfc->fs; + int ret = -ENOTSUPP; + const struct atl_rxf_flt_desc *desc; + + switch (rxnfc->cmd) { + case ETHTOOL_GRXRINGS: + rxnfc->data = nic->nvecs; + return 0; + + case ETHTOOL_GRXCLSRLCNT: + atl_get_rxf_count(nic, rxnfc); + return 0; + + case ETHTOOL_GRXCLSRULE: + desc = atl_rxf_desc(nic, fsp); + if (!desc) + return -EINVAL; + + memset(&fsp->h_u, 0, sizeof(fsp->h_u)); + memset(&fsp->m_u, 0, sizeof(fsp->m_u)); + memset(&fsp->h_ext, 0, sizeof(fsp->h_ext)); + memset(&fsp->m_ext, 0, sizeof(fsp->m_ext)); + + ret = desc->get_rxf(desc, nic, fsp); + break; + + case ETHTOOL_GRXCLSRLALL: + ret = atl_get_rxf_locs(nic, rxnfc, rule_locs); + break; + + default: + break; + } + + return ret; +} + +static int atl_set_rxnfc(struct net_device *ndev, struct ethtool_rxnfc *rxnfc) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct ethtool_rx_flow_spec *fsp = &rxnfc->fs; + + switch (rxnfc->cmd) { + case ETHTOOL_SRXCLSRLINS: + return atl_set_rxf(nic, fsp, false); + + case ETHTOOL_SRXCLSRLDEL: + return atl_set_rxf(nic, fsp, true); + } + + return -ENOTSUPP; +} + +int atl_vlan_rx_add_vid(struct net_device *ndev, __be16 proto, u16 vid) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + int idx; + + atl_nic_dbg("Add vlan id %hd\n", vid); + + vid &= 0xfff; + if (__test_and_set_bit(vid, vlan->map)) + /* Already created -- shouldn't happen? */ + return 0; + + vlan->vlans_active++; + idx = atl_rxf_find_vid(nic, vid, false); + + if (idx == ATL_VIDX_NONE) { + /* VID not found and no unused filters */ + vlan->promisc_count++; + if (pm_runtime_active(&nic->hw.pdev->dev)) + atl_set_vlan_promisc(&nic->hw, + (ndev->flags & IFF_PROMISC) || + vlan->promisc_count || + !vlan->vlans_active); + return 0; + } + + if (idx & ATL_VIDX_FREE) { + /* VID not found, program unused filter */ + idx &= ATL_VIDX_MASK; + vlan->cmd[idx] = ATL_VLAN_EN | ATL_RXF_ACT_TOHOST | vid; + vlan->count++; + goto update_vlan; + } + + idx &= ATL_VIDX_MASK; + if (vlan->cmd[idx] & ATL_RXF_ACT_TOHOST) + /* VID already added via ethtool */ + return 0; + + /* Ethtool filter set to drop. Override. */ + atl_nic_warn("%s: Overriding VLAN filter for VID %hd @%d set to drop\n", + __func__, vid, idx); + + vlan->cmd[idx] = ATL_RXF_EN | ATL_RXF_ACT_TOHOST | vid; + +update_vlan: + atl_rxf_update_vlan(nic, idx); + if (pm_runtime_active(&nic->hw.pdev->dev)) + atl_set_vlan_promisc(&nic->hw, + (nic->ndev->flags & IFF_PROMISC) || + vlan->promisc_count || + !vlan->vlans_active); + return 0; +} + +int atl_vlan_rx_kill_vid(struct net_device *ndev, __be16 proto, u16 vid) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_rxf_vlan *vlan = &nic->rxf_vlan; + uint32_t cmd; + int idx; + + atl_nic_dbg("Kill vlan id %hd\n", vid); + + vid &= 0xfff; + if (!__test_and_clear_bit(vid, vlan->map)) + return -EINVAL; + + vlan->vlans_active--; + + idx = atl_rxf_find_vid(nic, vid, false); + if (!(idx & ATL_VIDX_FOUND)) { + /* VID not present in filters, decrease promisc count */ + vlan->promisc_count--; + goto update_vlan_promisc; + } + + idx &= ATL_VIDX_MASK; + cmd = vlan->cmd[idx]; + if (cmd & ATL_VLAN_RXQ) + /* Queue explicitly set via ethtool, leave the filter + * intact */ + return 0; + + /* Delete filter, maybe pull vid from promisc overflow */ + vlan->cmd[idx] = 0; + vlan->count--; + if (!atl_vlan_pull_from_promisc(nic, idx)) + atl_rxf_update_vlan(nic, idx); + +update_vlan_promisc: + if (pm_runtime_active(&nic->hw.pdev->dev)) + atl_set_vlan_promisc(&nic->hw, (ndev->flags & IFF_PROMISC) || + vlan->promisc_count || + !vlan->vlans_active); + return 0; +} + +static void atl_get_wol(struct net_device *ndev, struct ethtool_wolinfo *wol) +{ + struct atl_nic *nic = netdev_priv(ndev); + + wol->supported = ATL_WAKE_SUPPORTED; + wol->wolopts = nic->hw.wol_mode; +} + +static int atl_set_wol(struct net_device *ndev, struct ethtool_wolinfo *wol) +{ + int ret; + struct atl_nic *nic = netdev_priv(ndev); + + if (wol->wolopts & ~ATL_WAKE_SUPPORTED) { + atl_nic_err("%s: unsupported WoL mode %x\n", __func__, + wol->wolopts); + return -EINVAL; + } + + if (wol->wolopts) + nic->flags |= ATL_FL_WOL; + else + nic->flags &= ~ATL_FL_WOL; + + nic->hw.wol_mode = wol->wolopts; + + ret = device_set_wakeup_enable(&nic->hw.pdev->dev, + !!(nic->flags & ATL_FL_WOL)); + + if (ret == -EEXIST) + ret = 0; + if (ret) { + atl_nic_err("device_set_wakeup_enable failed: %d\n", -ret); + nic->flags &= ~ATL_FL_WOL; + nic->hw.wol_mode = 0; + } + + return ret; +} + +static int atl_ethtool_begin(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + pm_runtime_get_sync(&nic->hw.pdev->dev); + return 0; +} + +static void atl_ethtool_complete(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + pm_runtime_put(&nic->hw.pdev->dev); +} + +const struct ethtool_ops atl_ethtool_ops = { +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 7, 0) + .supported_coalesce_params = ETHTOOL_COALESCE_USECS | + ETHTOOL_COALESCE_MAX_FRAMES, +#endif + .get_link = atl_ethtool_get_link, +#ifndef ATL_HAVE_ETHTOOL_KSETTINGS + .get_settings = atl_ethtool_get_settings, + .set_settings = atl_ethtool_set_settings, +#else + .get_link_ksettings = atl_ethtool_get_ksettings, + .set_link_ksettings = atl_ethtool_set_ksettings, +#endif + + .get_rxfh_indir_size = atl_rss_tbl_size, + .get_rxfh_key_size = atl_rss_key_size, + .get_rxfh = atl_rss_get_rxfh, + .set_rxfh = atl_rss_set_rxfh, + .get_channels = atl_get_channels, + .set_channels = atl_set_channels, + .get_rxnfc = atl_get_rxnfc, + .set_rxnfc = atl_set_rxnfc, + .get_pauseparam = atl_get_pauseparam, + .set_pauseparam = atl_set_pauseparam, + .get_eee = atl_get_eee, + .set_eee = atl_set_eee, + .get_drvinfo = atl_get_drvinfo, + .nway_reset = atl_nway_reset, + .get_ringparam = atl_get_ringparam, + .set_ringparam = atl_set_ringparam, + .get_sset_count = atl_get_sset_count, + .get_strings = atl_get_strings, + .get_ethtool_stats = atl_get_ethtool_stats, + .get_priv_flags = atl_get_priv_flags, + .set_priv_flags = atl_set_priv_flags, + .get_coalesce = atl_get_coalesce, + .set_coalesce = atl_set_coalesce, + .get_ts_info = atl_get_ts_info, + .get_wol = atl_get_wol, + .set_wol = atl_set_wol, + .begin = atl_ethtool_begin, + .complete = atl_ethtool_complete, +}; diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.h new file mode 100644 index 000000000000..7fc88ea2c080 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ethtool.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_ETHTOOL_H_ +#define _ATL_ETHTOOL_H_ + +enum atl_rxf_type { + ATL_RXF_VLAN = 0, + ATL_RXF_ETYPE, + ATL_RXF_NTUPLE, + ATL_RXF_FLEX, +}; + +s8 atl_reserve_filter(enum atl_rxf_type type); +void atl_release_filter(enum atl_rxf_type type); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.c new file mode 100644 index 000000000000..8f90745a6f49 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.c @@ -0,0 +1,1215 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_common.h" +#include "atl_hw.h" +#include "atl_drviface.h" + +#define ATL_FW_CFG_DUMP_SIZE 2 + +struct atl_link_type atl_link_types[] = { +#define LINK_TYPE(_idx, _name, _speed, _duplex, _ethtl_idx, _fw1_bit, _fw2_bit) \ + [_idx] = { \ + .name = _name, \ + .speed = _speed, \ + .duplex = _duplex, \ + .ethtool_idx = _ethtl_idx, \ + .fw_bits = { \ + [0] = _fw1_bit, \ + [1] = _fw2_bit, \ + }, \ + }, + + LINK_TYPE(atl_link_type_idx_10m_half, + "10BaseTX-HD", 10, DUPLEX_HALF, + ETHTOOL_LINK_MODE_10baseT_Half_BIT, 0, 0) + LINK_TYPE(atl_link_type_idx_10m, + "10BaseTX-FD", 10, DUPLEX_FULL, + ETHTOOL_LINK_MODE_10baseT_Full_BIT, 0, BIT(1)) + LINK_TYPE(atl_link_type_idx_100m_half, + "100BaseTX-HD", 100, DUPLEX_HALF, + ETHTOOL_LINK_MODE_100baseT_Half_BIT, 0, 0) + LINK_TYPE(atl_link_type_idx_100m, + "100BaseTX-FD", 100, DUPLEX_FULL, + ETHTOOL_LINK_MODE_100baseT_Full_BIT, 0x20, BIT(5)) + LINK_TYPE(atl_link_type_idx_1g_half, + "1000BaseT-HD", 1000, DUPLEX_HALF, + ETHTOOL_LINK_MODE_1000baseT_Half_BIT, 0, 0) + LINK_TYPE(atl_link_type_idx_1g, + "1000BaseT-FD", 1000, DUPLEX_FULL, + ETHTOOL_LINK_MODE_1000baseT_Full_BIT, 0x10, BIT(8)) + LINK_TYPE(atl_link_type_idx_2p5g, + "2.5GBaseT-FD", 2500, DUPLEX_FULL, + ETHTOOL_LINK_MODE_2500baseT_Full_BIT, 8, BIT(9)) + LINK_TYPE(atl_link_type_idx_5g, + "5GBaseT-FD", 5000, DUPLEX_FULL, + ETHTOOL_LINK_MODE_5000baseT_Full_BIT, 2, BIT(10)) + LINK_TYPE(atl_link_type_idx_10g, + "10GBaseT-FD", 10000, DUPLEX_FULL, + ETHTOOL_LINK_MODE_10000baseT_Full_BIT, 1, BIT(11)) +}; +#define ATL_FW2_LINK_MSK (BIT(5) | BIT(8) | BIT(9) | BIT(10) | BIT(11)) + +const int atl_num_rates = ARRAY_SIZE(atl_link_types); + +/* fw lock must be held */ +static int __atl_fw1_wait_fw_init(struct atl_hw *hw) +{ + uint32_t hostData_addr; + uint32_t id, new_id; + int ret; + + mdelay(10); + + busy_wait(2000, mdelay(1), hostData_addr, + atl_read(hw, ATL_MCP_SCRATCH(FW_STAT_STRUCT)), + hostData_addr == 0); + + atl_dev_dbg("got hostData address: 0x%x\n", hostData_addr); + + ret = atl_read_mcp_mem(hw, hostData_addr + 4, &id, 4); + if (ret) + return ret; + + busy_wait(10000, mdelay(1), ret, + atl_read_mcp_mem(hw, hostData_addr + 4, &new_id, 4), + !ret && new_id == id); + if (ret) + return ret; + if (new_id == id) { + atl_dev_err("timeout waiting for FW to start (initial transactionId 0x%x, hostData addr 0x%x)\n", + id, hostData_addr); + return -EIO; + } + + /* return fw1_wait_drviface(hw, NULL); */ + return 0; +} + +/* fw lock must be held */ +static int __atl_fw2_wait_fw_init(struct atl_hw *hw) +{ + uint32_t reg; + int ret = 0; + + busy_wait(1000, mdelay(1), reg, atl_read(hw, ATL_GLOBAL_FW_IMAGE_ID), + !reg); + if (!reg) + return -EIO; + + hw->mcp.req_high = atl_read(hw, + ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH)); + + hw->mcp.fw_stat_addr = atl_read(hw, ATL_MCP_SCRATCH(FW_STAT_STRUCT)); + hw->mcp.rpc_addr = atl_read(hw, ATL_MCP_SCRATCH(FW2_RPC_DATA)); + + ret = atl_read_fwstat_word(hw, atl_fw2_stat_settings_addr, + &hw->mcp.fw_settings_addr); + if (ret) + return ret; + + ret = atl_read_fwstat_word(hw, atl_fw2_stat_settings_len, + &hw->mcp.fw_settings_len); + if (ret) + return ret; + + return 0; +} + +static struct atl_link_type *atl_parse_fw_bits(struct atl_hw *hw, + uint32_t low, uint32_t high, int fw_idx) +{ + struct atl_link_state *lstate = &hw->link_state; + unsigned int lp_adv = 0, adv = lstate->advertized; + struct atl_link_type *link; + bool eee = false; + int last = -1; + int i; + + atl_for_each_rate(i, link) { + uint32_t link_bit = link->fw_bits[fw_idx]; + + if (!(low & link_bit)) + continue; + + if (high & link_bit) + lp_adv |= BIT(i + ATL_EEE_BIT_OFFT); + + lp_adv |= BIT(i); + if (adv & BIT(i)) + last = i; + } + + lstate->lp_advertized = lp_adv; + + link = 0; + if (last >= 0) { + link = &atl_link_types[last]; + if ((lp_adv & BIT(last + ATL_EEE_BIT_OFFT)) && + (adv & BIT(last + ATL_EEE_BIT_OFFT))) + eee = true; + } + + lstate->link = link; + lstate->eee = eee; + return link; +} + +static struct atl_link_type *atl_fw1_check_link(struct atl_hw *hw) +{ + uint32_t reg; + struct atl_link_type *link; + + atl_lock_fw(hw); + reg = atl_read(hw, ATL_MCP_SCRATCH(FW1_LINK_STS)); + + if ((reg & 0xf) != 2) + reg = 0; + + reg = (reg >> 16) & 0xff; + + link = atl_parse_fw_bits(hw, reg, 0, 0); + + atl_unlock_fw(hw); + return link; +} + +static struct atl_link_type *atl_fw2_check_link(struct atl_hw *hw) +{ + struct atl_link_type *link; + struct atl_link_state *lstate = &hw->link_state; + enum atl_fc_mode fc = atl_fc_none; + uint32_t low; + uint32_t high; + bool alarm; + + low = atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_LOW)); + high = atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)); + + link = atl_parse_fw_bits(hw, low, high, 1); + + alarm = !!(low & atl_fw2_thermal_alarm); + atl_thermal_check(hw, alarm); + + /* Thermal check might have reset link due to throttling */ + link = lstate->link; + + if (link) { + if (high & atl_fw2_pause) + fc |= atl_fc_rx; + if (high & atl_fw2_asym_pause) + fc |= atl_fc_tx; + } + + lstate->fc.cur = fc; + + return link; +} + +/* fw lock must be held */ +static int __atl_fw1_get_link_caps(struct atl_hw *hw) +{ + return 0; +} + +/* fw lock must be held */ +static int __atl_fw2_get_link_caps(struct atl_hw *hw) +{ + struct atl_nic *nic = container_of(hw, struct atl_nic, hw); + struct atl_mcp *mcp = &hw->mcp; + uint32_t fw_stat_addr = mcp->fw_stat_addr; + struct atl_link_type *rate; + unsigned int supported = 0; + uint32_t caps[2], caps_ex; + uint32_t mask = atl_fw2_pause_mask | atl_fw2_link_drop; + int i, ret; + + atl_dev_dbg("Host data struct addr: %#x\n", fw_stat_addr); + ret = atl_read_mcp_mem(hw, fw_stat_addr + atl_fw2_stat_lcaps, + caps, 8); + if (ret) + return ret; + ret = atl_read_fwstat_word(hw, atl_fw2_stat_caps_ex, &caps_ex); + if (ret) + return ret; + + mcp->caps_low = caps[0]; + mcp->caps_high = caps[1]; + mcp->caps_ex = caps_ex; + mcp->wdog_disabled = !(mcp->caps_ex & atl_fw2_ex_caps_mac_heartbeat); + atl_dev_dbg("Got link caps: %#x %#x %#x\n", caps[0], caps[1], caps_ex); + + atl_for_each_rate(i, rate) { + uint32_t bit = rate->fw_bits[1]; + + if (bit & caps[0]) { + supported |= BIT(i); + if (bit & caps[1]) { + supported |= BIT(i + ATL_EEE_BIT_OFFT); + mask |= bit; + } + } + } + + mcp->req_high_mask = ~mask; + hw->link_state.supported = supported; + hw->link_state.lp_lowest = fls(supported) - 1; + + nic->rxf_flex.base_index = 0; + nic->rxf_flex.available = ATL_FLEX_FLT_NUM; + nic->rxf_mac.base_index = 0; + nic->rxf_mac.available = ATL_UC_FLT_NUM; + nic->rxf_etype.base_index = 0; + nic->rxf_etype.available = ATL_ETYPE_FLT_NUM - 1; /* 1 reserved by FW */ + nic->rxf_vlan.base_index = 0; + nic->rxf_vlan.available = ATL_VLAN_FLT_NUM; + nic->rxf_ntuple.l3_v4_base_index = 0; + nic->rxf_ntuple.l3_v4_available = ATL_NTUPLE_FLT_NUM; + nic->rxf_ntuple.l3_v6_base_index = 0; + nic->rxf_ntuple.l3_v6_available = ATL_NTUPLE_V6_FLT_NUM; + nic->rxf_ntuple.l4_base_index = 0; + nic->rxf_ntuple.l4_available = ATL_NTUPLE_FLT_NUM; + + return ret; +} + +static inline unsigned int atl_link_adv(struct atl_link_state *lstate) +{ + struct atl_hw *hw = container_of(lstate, struct atl_hw, link_state); + + if (lstate->thermal_throttled + && hw->thermal.flags & atl_thermal_throttle) + /* FW doesn't provide raw LP's advertized rates, only + * the rates adverized both by us and LP. Here we + * advertize not just the throttled_to rate, but also + * all the lower rates as well. That way if LP changes + * or dynamically starts to adverize a lower rate than + * throttled_to, we will notice that in + * atl_thermal_check() and switch to that lower + * rate there. + */ + return (lstate->advertized & ATL_EEE_MASK) | + (BIT(lstate->throttled_to + 1) - 1); + + return lstate->advertized; +} + +static inline bool atl_fw1_set_link_needed(struct atl_link_state *lstate) +{ + bool ret = false; + + if (atl_link_adv(lstate) != lstate->prev_advertized) { + ret = true; + lstate->prev_advertized = atl_link_adv(lstate); + } + + return ret; +} + +static inline bool atl_fw2_set_link_needed(struct atl_link_state *lstate) +{ + struct atl_fc_state *fc = &lstate->fc; + bool ret = false; + + if (fc->req != fc->prev_req) { + ret = true; + fc->prev_req = fc->req; + } + + return atl_fw1_set_link_needed(lstate) || ret; +} + +static uint64_t atl_set_fw_bits(struct atl_hw *hw, int fw_idx) +{ + unsigned int adv = atl_link_adv(&hw->link_state); + struct atl_link_type *ltype; + uint64_t link = 0; + int i; + + atl_for_each_rate(i, ltype) { + uint32_t bit = ltype->fw_bits[fw_idx]; + + if (adv & BIT(i)) { + link |= bit; + if (adv & BIT(i + ATL_EEE_BIT_OFFT)) + link |= (uint64_t)bit << 32; + } + } + + return link; +} + +static void atl_fw1_set_link(struct atl_hw *hw, bool force) +{ + uint32_t bits; + + if (!force && !atl_fw1_set_link_needed(&hw->link_state)) + return; + + atl_lock_fw(hw); + + bits = (atl_set_fw_bits(hw, 0) << 16) | 2; + atl_write(hw, ATL_MCP_SCRATCH(FW1_LINK_REQ), bits); + + atl_unlock_fw(hw); +} + +/* fw lock must be held */ +static void __atl_fw2_set_link(struct atl_hw *hw) +{ + struct atl_link_state *lstate = &hw->link_state; + uint32_t hi_bits; + uint64_t bits; + + hi_bits = hw->mcp.req_high & hw->mcp.req_high_mask; + + if (lstate->fc.req & atl_fc_rx) + hi_bits |= atl_fw2_pause | atl_fw2_asym_pause; + + if (lstate->fc.req & atl_fc_tx) + hi_bits ^= atl_fw2_asym_pause; + + bits = atl_set_fw_bits(hw, 1); + hi_bits |= bits >> 32; + + if (lstate->force_off) + hi_bits |= atl_fw2_link_drop; + + hw->mcp.req_high = hi_bits; + atl_write_mask_bits(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW), + ATL_FW2_LINK_MSK, bits); + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), hi_bits); +} + +static void atl_fw2_set_link(struct atl_hw *hw, bool force) +{ + if (!force && !atl_fw2_set_link_needed(&hw->link_state)) + return; + + atl_lock_fw(hw); + __atl_fw2_set_link(hw); + atl_unlock_fw(hw); +} + +static int atl_fw1_unsupported(struct atl_hw *hw) +{ + return -EOPNOTSUPP; +} + +static int __atl_fw2_restart_aneg(struct atl_hw *hw) +{ + /* Autoneg restart is self-clearing, no need to track via + * mcp->req_high */ + atl_set_bits(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), BIT(31)); + return 0; +} + +static int atl_fw2_restart_aneg(struct atl_hw *hw) +{ + atl_lock_fw(hw); + __atl_fw2_restart_aneg(hw); + atl_unlock_fw(hw); + return 0; +} + +static void atl_fw1_set_default_link(struct atl_hw *hw) +{ + struct atl_link_state *lstate = &hw->link_state; + + lstate->autoneg = true; + lstate->advertized = hw->link_state.supported; +} + +static void atl_fw2_set_default_link(struct atl_hw *hw) +{ + struct atl_link_state *lstate = &hw->link_state; + + atl_fw1_set_default_link(hw); + lstate->fc.req = atl_fc_full; + lstate->eee_enabled = 0; + lstate->advertized &= ~ATL_EEE_MASK; +} + +static int atl_fw1_enable_wol(struct atl_hw *hw, unsigned int wol_mode) +{ + return -EOPNOTSUPP; +} +static int atl_fw2_enable_wol(struct atl_hw *hw, unsigned int wol_mode) +{ + int ret = 0; + struct offloadInfo *info; + struct drvIface *msg = NULL; + uint32_t val, wol_bits = 0, req_high = hw->mcp.req_high; + uint32_t low_req, wol_ex_flags = 0; + + atl_lock_fw(hw); + + low_req = atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW)); + + if (wol_mode & atl_fw_wake_on_link) { + wol_bits |= atl_fw2_wake_on_link; + wol_ex_flags |= atl_fw2_wol_ex_wake_on_link_keep_rate; + low_req &= ~atl_fw2_wake_on_link_force; + } + + if (wol_mode & atl_fw_wake_on_link_rtpm) { + wol_bits |= atl_fw2_wake_on_link; + low_req |= atl_fw2_wake_on_link_force; + } + + if (wol_mode & atl_fw_wake_on_magic) { + wol_bits |= atl_fw2_nic_proxy | atl_fw2_wol; + wol_ex_flags |= atl_fw2_wol_ex_wake_on_magic_keep_rate; + + ret = -ENOMEM; + msg = kzalloc(sizeof(*msg), GFP_KERNEL); + if (!msg) + goto unlock; + + info = &msg->fw2xOffloads; + info->version = 0; + info->len = sizeof(*info); + memcpy(info->macAddr, hw->mac_addr, ETH_ALEN); + + ret = atl_write_mcp_mem(hw, 0, msg, + (info->len + offsetof(struct drvIface, fw2xOffloads) + + 3) & ~3, MCP_AREA_CONFIG); + if (ret) { + atl_dev_err("Failed to upload sleep proxy info to FW\n"); + goto unlock_free; + } + } + + if (hw->mcp.caps_ex & atl_fw2_ex_caps_wol_ex) { + ret = atl_write_fwsettings_word(hw, atl_fw2_setings_wol_ex, + wol_ex_flags); + if (ret) + goto unlock_free; + } + + req_high |= wol_bits; + req_high &= ~atl_fw2_link_drop; + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW), low_req); + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), req_high); + busy_wait(100, mdelay(1), val, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)), + (val & wol_bits) != wol_bits); + + ret = (val & wol_bits) == wol_bits ? 0 : -EIO; + if (ret) + atl_dev_err("Timeout waiting for WoL enable\n"); + +unlock_free: + kfree(msg); +unlock: + atl_unlock_fw(hw); + return ret; +} + +int atl_read_mcp_word(struct atl_hw *hw, uint32_t offt, uint32_t *val) +{ + int ret; + + ret = atl_read_mcp_mem(hw, offt & ~3, val, 4); + if (ret) + return ret; + + *val >>= 8 * (offt & 3); + return 0; +} + +/* fw lock must be held */ +static int __atl_fw2_get_phy_temperature(struct atl_hw *hw, int *temp) +{ + uint32_t req, res; + int ret = 0; + + if (test_bit(ATL_ST_RESETTING, &hw->state)) + return 0; + + hw->mcp.req_high ^= atl_fw2_phy_temp; + req = hw->mcp.req_high; + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), req); + + busy_wait(1000, udelay(10), res, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)), + ((res ^ req) & atl_fw2_phy_temp) != 0); + if (((res ^ req) & atl_fw2_phy_temp) != 0) { + atl_dev_err("Timeout waiting for PHY temperature\n"); + return -EIO; + } + + ret = atl_read_fwstat_word(hw, atl_fw2_stat_temp, &res); + if (ret) + return ret; + + *temp = (int16_t)(res & 0xFFFF) * 1000 / 256; + + return ret; +} + +static int atl_fw2_get_phy_temperature(struct atl_hw *hw, int *temp) +{ + int ret; + + atl_lock_fw(hw); + ret = __atl_fw2_get_phy_temperature(hw, temp); + atl_unlock_fw(hw); + return ret; +} + +static int atl_fw2_dump_cfg(struct atl_hw *hw) +{ + if (!hw->mcp.fw_cfg_dump) + hw->mcp.fw_cfg_dump = devm_kzalloc(&hw->pdev->dev, + ATL_FW_CFG_DUMP_SIZE, + GFP_KERNEL); + /* save link configuration */ + hw->mcp.fw_cfg_dump[0] = + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW)); + hw->mcp.fw_cfg_dump[1] = + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH)) & 0x400F18; + + return 0; +} + +static int atl_fw2_restore_cfg(struct atl_hw *hw) +{ + if (!hw->mcp.fw_cfg_dump) + return 0; + + /* restore link configuration */ + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW), + hw->mcp.fw_cfg_dump[0]); + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), + hw->mcp.fw_cfg_dump[1]); + + return 0; +} + +static int atl_fw2_set_phy_loopback(struct atl_nic *nic, u32 mode) +{ + bool on = !!(nic->priv_flags & BIT(mode)); + struct atl_hw *hw = &nic->hw; + + atl_lock_fw(hw); + + switch (mode) { + case ATL_PF_LPB_INT_PHY: + atl_write_bit(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), 27, on); + break; + case ATL_PF_LPB_EXT_PHY: + atl_write_bit(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), 26, on); + break; + } + + atl_unlock_fw(hw); + + return 0; +} + +static int atl_fw2_update_statistics(struct atl_hw *hw) +{ + uint32_t req; + int res = 0; + + hw->mcp.req_high ^= atl_fw2_statistics; + req = hw->mcp.req_high; + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), req); + + busy_wait(10000, udelay(10), res, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)), + ((res ^ req) & atl_fw2_statistics) != 0); + if (((res ^ req) & atl_fw2_statistics) != 0) { + atl_dev_err("Timeout waiting for statistics\n"); + return -EIO; + } + return 0; +} + +static int atl_fw2_set_mediadetect(struct atl_hw *hw, bool on) +{ + int ret = 0; + + if (hw->mcp.fw_rev < 0x0301005a) + return -EOPNOTSUPP; + + atl_lock_fw(hw); + + ret = atl_write_fwsettings_word(hw, atl_fw2_setings_media_detect, on); + if (ret) + goto unlock; + + /* request statistics just to force FW to read settings */ + ret = atl_fw2_update_statistics(hw); +unlock: + atl_unlock_fw(hw); + return ret; +} + +static int __atl_fw2x_apply_msm_settings(struct atl_hw *hw) +{ + + uint32_t msg_id = atl_fw2_msm_settings_apply; + uint32_t high_status, high_req = 0; + int ret = 0; + + if (!(hw->mcp.caps_ex & atl_fw2_ex_caps_msm_settings_apply)) + return __atl_fw2_restart_aneg(hw); + + ret = atl_write_mcp_mem(hw, 0, &msg_id, sizeof(msg_id), + MCP_AREA_CONFIG); + if (ret) { + atl_dev_err("Failed to upload macsec request: %d\n", ret); + atl_unlock_fw(hw); + return ret; + } + + high_req = atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH)); + high_req ^= atl_fw2_fw_request; + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), high_req); + + busy_wait(1000, mdelay(1), high_status, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)), + ((high_req ^ high_status) & atl_fw2_fw_request) != 0); + if (((high_req ^ high_status) & atl_fw2_fw_request) != 0) { + atl_dev_err("Timeout waiting for fw request\n"); + atl_unlock_fw(hw); + return -EIO; + } + + return ret; +} + +static int atl_fw2_set_pad_stripping(struct atl_hw *hw, bool on) +{ + uint32_t msm_opts; + int ret = 0; + + if (hw->mcp.fw_rev < 0x0300008e) + return -EOPNOTSUPP; + + atl_lock_fw(hw); + + ret = atl_read_fwsettings_word(hw, atl_fw2_setings_msm_opts, + &msm_opts); + if (ret) { + if (ret == -EINVAL) + ret = -EOPNOTSUPP; + + goto unlock; + } + + msm_opts &= ~atl_fw2_settings_msm_opts_strip_pad; + if (on) + msm_opts |= BIT(atl_fw2_settings_msm_opts_strip_pad_shift); + + ret = atl_write_fwsettings_word(hw, atl_fw2_setings_msm_opts, + msm_opts); + if (ret) + goto unlock; + + ret = __atl_fw2x_apply_msm_settings(hw); +unlock: + atl_unlock_fw(hw); + return ret; +} + +static int atl_fw2_send_macsec_request(struct atl_hw *hw, + struct macsec_msg_fw_request *req, + struct macsec_msg_fw_response *response) +{ + int ret = 0; + uint32_t low_status, low_req = 0; + + if (!req || !response) + return -EINVAL; + + if ((hw->mcp.caps_low & atl_fw2_macsec) == 0) + return -EOPNOTSUPP; + + atl_lock_fw(hw); + + /* Write macsec request to cfg memory */ + ret = atl_write_mcp_mem(hw, 0, req, (sizeof(*req) + 3) & ~3, + MCP_AREA_CONFIG); + if (ret) { + atl_dev_err("Failed to upload macsec request: %d\n", ret); + atl_unlock_fw(hw); + return ret; + } + + /* Toggle 0x368.CAPS_LO_MACSEC bit */ + low_req = atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW)); + low_req ^= atl_fw2_macsec; + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_LOW), low_req); + + busy_wait(1000, mdelay(1), low_status, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_LOW)), + ((low_req ^ low_status) & atl_fw2_macsec) != 0); + if (((low_req ^ low_status) & atl_fw2_macsec) != 0) { + atl_dev_err("Timeout waiting for macsec request\n"); + atl_unlock_fw(hw); + return -EIO; + } + + /* Read status of write operation */ + ret = atl_read_rpc_mem(hw, sizeof(u32), (u32 *)(void *)response, + sizeof(*response)); + + atl_unlock_fw(hw); + return ret; +} + +/* fw lock must be held */ +static int __atl_fw2_get_hbeat(struct atl_hw *hw, uint16_t *hbeat) +{ + int ret; + uint32_t val; + + ret = atl_read_fwstat_word(hw, atl_fw2_stat_phy_hbeat, &val); + if (ret) + atl_dev_err("FW watchdog: failure reading PHY heartbeat: %d\n", + -ret); + else + *hbeat = val & 0xffff; + + return ret; +} + +static int atl_fw1_get_mac_addr(struct atl_hw *hw, uint8_t *buf) +{ + uint32_t efuse_shadow_addr = + atl_read(hw, ATL_MCP_SCRATCH(FW1_EFUSE_SHADOW)); + uint8_t tmp[8]; + int ret; + + if (!efuse_shadow_addr) + return false; + + ret = atl_read_mcp_mem(hw, efuse_shadow_addr + 40 * 4, tmp, 8); + *(uint32_t *)buf = htonl(*(uint32_t *)tmp); + *(uint16_t *)&buf[4] = (uint16_t)htonl(*(uint32_t *)&tmp[4]); + + return ret; +} + +static int atl_fw2_get_mac_addr(struct atl_hw *hw, uint8_t *buf) +{ + uint32_t efuse_shadow_addr = + atl_read(hw, ATL_MCP_SCRATCH(FW2_EFUSE_SHADOW)); + uint8_t tmp[8]; + int ret; + + if (!efuse_shadow_addr) + return false; + + ret = atl_read_mcp_mem(hw, efuse_shadow_addr + 40 * 4, tmp, 8); + *(uint32_t *)buf = htonl(*(uint32_t *)tmp); + *(uint16_t *)&buf[4] = (uint16_t)htonl(*(uint32_t *)&tmp[4]); + + return ret; +} + +/* fw lock must be held */ +static int __atl_fw2_set_thermal_monitor(struct atl_hw *hw, bool enable) +{ + struct atl_mcp *mcp = &hw->mcp; + int ret; + uint32_t high; + + if (enable) { + struct atl_fw2_thermal_cfg cfg __attribute__((__aligned__(4))); + + cfg.msg_id = 0x17; + cfg.shutdown_temp = hw->thermal.crit; + cfg.high_temp = hw->thermal.high; + cfg.normal_temp = hw->thermal.low; + + ret = atl_write_mcp_mem(hw, 0, &cfg, (sizeof(cfg) + 3) & ~3, + MCP_AREA_CONFIG); + if (ret) { + atl_dev_err("Failed to upload thermal thresholds to firmware: %d\n", + ret); + return ret; + } + + mcp->req_high |= atl_fw2_set_thermal; + } else + mcp->req_high &= ~atl_fw2_set_thermal; + + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), mcp->req_high); + busy_wait(1000, udelay(10), high, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)), + !!(high & atl_fw2_set_thermal) != enable); + if (!!(high & atl_fw2_set_thermal) != enable) { + atl_dev_err("Timeout waiting for thermal monitoring FW request\n"); + return -EIO; + } + + return 0; +} + +static int atl_fw2_update_thermal(struct atl_hw *hw) +{ + bool enable = !!(hw->thermal.flags & atl_thermal_monitor); + struct atl_mcp *mcp = &hw->mcp; + int ret = 0; + bool alarm; + + if (!(mcp->caps_high & atl_fw2_set_thermal)) { + if (hw->thermal.flags & atl_thermal_monitor) + atl_dev_warn("Thermal monitoring not supported by firmware\n"); + hw->thermal.flags &= + ~(atl_thermal_monitor | atl_thermal_throttle); + return 0; + } + + atl_lock_fw(hw); + + if (!enable || (mcp->req_high & atl_fw2_set_thermal)) { + /* If monitoring is on and we need to change the + * thresholds, we need to temporarily disable thermal + * monitoring first. */ + ret = __atl_fw2_set_thermal_monitor(hw, false); + if (ret) { + atl_unlock_fw(hw); + return ret; + } + } + + if (enable) + ret = __atl_fw2_set_thermal_monitor(hw, true); + atl_unlock_fw(hw); + + /* Thresholds might have changed, recheck state. */ + alarm = !!(atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_LOW)) & + atl_fw2_thermal_alarm); + atl_thermal_check(hw, alarm); + + return ret; +} + +static int atl_fw2_send_ptp_request(struct atl_hw *hw, + struct ptp_msg_fw_request *msg) +{ + u32 high_req, high_status; + size_t size; + int ret = 0; + + if (!msg) + return -EINVAL; + + size = sizeof(msg->msg_id); + switch (msg->msg_id) { + case ptp_gpio_ctrl_msg: + size += sizeof(msg->gpio_ctrl); + break; + case ptp_adj_freq_msg: + size += sizeof(msg->adj_freq); + break; + case ptp_adj_clock_msg: + size += sizeof(msg->adj_clock); + break; + default: + return -EINVAL; + } + + atl_lock_fw(hw); + + /* Write ptp request to cfg memory */ + ret = atl_write_mcp_mem(hw, 0, msg, (size + 3) & ~3, MCP_AREA_CONFIG); + if (ret) { + atl_dev_err("Failed to upload ptp request: %d\n", ret); + goto err_exit; + } + + high_req = atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH)); + high_req ^= atl_fw2_fw_request; + atl_write(hw, ATL_MCP_SCRATCH(FW2_LINK_REQ_HIGH), high_req); + + busy_wait(1000, mdelay(1), high_status, + atl_read(hw, ATL_MCP_SCRATCH(FW2_LINK_RES_HIGH)), + ((high_req ^ high_status) & atl_fw2_fw_request) != 0); + if (((high_req ^ high_status) & atl_fw2_fw_request) != 0) { + atl_dev_err("Timeout waiting for fw request\n"); + ret = -EIO; + goto err_exit; + } + + /* Toggle statistics bit for FW to update */ + ret = atl_fw2_update_statistics(hw); + +err_exit: + atl_unlock_fw(hw); + return ret; +} + +static void atl_fw3_set_ptp(struct atl_hw *hw, bool on) +{ + u32 all_ptp_features = atl_fw2_ex_caps_phy_ptp_en | atl_fw2_ex_caps_ptp_gpio_en; + u32 ptp_opts; + + atl_lock_fw(hw); + ptp_opts = atl_read(hw, ATL_MCP_SCRATCH(FW3_EXT_RES)); + if (on) + ptp_opts |= all_ptp_features; + else + ptp_opts &= ~all_ptp_features; + + atl_write(hw, ATL_MCP_SCRATCH(FW3_EXT_REQ), ptp_opts); + atl_unlock_fw(hw); +} + +static struct atl_fw_ops atl_fw_ops[2] = { + [0] = { + .__wait_fw_init = __atl_fw1_wait_fw_init, + .set_link = atl_fw1_set_link, + .check_link = atl_fw1_check_link, + .__get_link_caps = __atl_fw1_get_link_caps, + .restart_aneg = atl_fw1_unsupported, + .set_default_link = atl_fw1_set_default_link, + .enable_wol = atl_fw1_enable_wol, + .get_phy_temperature = (void *)atl_fw1_unsupported, + .dump_cfg = atl_fw1_unsupported, + .restore_cfg = atl_fw1_unsupported, + .set_phy_loopback = (void *)atl_fw1_unsupported, + .set_mediadetect = (void *)atl_fw1_unsupported, + .send_macsec_req = (void *)atl_fw1_unsupported, + .set_pad_stripping = (void *)atl_fw1_unsupported, + .__get_hbeat = (void *)atl_fw1_unsupported, + .get_mac_addr = atl_fw1_get_mac_addr, + .update_thermal = atl_fw1_unsupported, + .deinit = atl_fw1_unsupported, + }, + [1] = { + .__wait_fw_init = __atl_fw2_wait_fw_init, + .set_link = atl_fw2_set_link, + .check_link = atl_fw2_check_link, + .__get_link_caps = __atl_fw2_get_link_caps, + .restart_aneg = atl_fw2_restart_aneg, + .set_default_link = atl_fw2_set_default_link, + .enable_wol = atl_fw2_enable_wol, + .get_phy_temperature = atl_fw2_get_phy_temperature, + .dump_cfg = atl_fw2_dump_cfg, + .restore_cfg = atl_fw2_restore_cfg, + .set_phy_loopback = atl_fw2_set_phy_loopback, + .set_mediadetect = atl_fw2_set_mediadetect, + .send_macsec_req = atl_fw2_send_macsec_request, + .set_pad_stripping = atl_fw2_set_pad_stripping, + .__get_hbeat = __atl_fw2_get_hbeat, + .get_mac_addr = atl_fw2_get_mac_addr, + .update_thermal = atl_fw2_update_thermal, + .send_ptp_req = atl_fw2_send_ptp_request, + .set_ptp = atl_fw3_set_ptp, + .deinit = atl_fw1_unsupported, + }, +}; + +struct atl_thermal_limit { + uintptr_t offset; + const char *name; + unsigned min; + unsigned max; +}; +#define atl_def_thermal_limit(_name, _field, _min, _max) \ +{ \ + .offset = offsetof(struct atl_thermal, _field), \ + .name = _name, \ + .min = _min, \ + .max = _max, \ +}, + +static struct atl_thermal_limit atl_thermal_limits[] = { + atl_def_thermal_limit("Shutdown", crit, 108, 118) + atl_def_thermal_limit("High", high, 90, 107) + atl_def_thermal_limit("Normal", low, 50, 85) +}; + +int atl_verify_thermal_limits(struct atl_hw *hw, struct atl_thermal *thermal) +{ + int i; + bool ignore = !!(thermal->flags & atl_thermal_ignore_lims); + + for (i = 0; i < ARRAY_SIZE(atl_thermal_limits); i++) { + struct atl_thermal_limit *lim = &atl_thermal_limits[i]; + unsigned val = *((uint8_t *)thermal + lim->offset); + + if (val >= lim->min && val <= lim->max) + continue; + + if (ignore) { + atl_dev_init_warn("%s temperature threshold out of range (%d - %d): %d, allowing anyway\n", + lim->name, lim->min, lim->max, val); + continue; + } else { + atl_dev_init_err("%s temperature threshold out of range (%d - %d): %d\n", + lim->name, lim->min, lim->max, val); + return -EINVAL; + } + } + + return 0; +} + +int atl_update_thermal(struct atl_hw *hw) +{ + int ret; + + ret = atl_verify_thermal_limits(hw, &hw->thermal); + if (ret) + return ret; + + if (test_bit(ATL_ST_RESETTING, &hw->state)) + /* After reset, atl_fw_init() will apply the settings + * skipped here */ + return 0; + + ret = hw->mcp.ops->update_thermal(hw); + + return ret; +} + +int atl_update_thermal_flag(struct atl_hw *hw, int bit, bool val) +{ + struct atl_thermal *thermal = &hw->thermal; + unsigned flags, changed; + int ret = 0; + + flags = thermal->flags; + + switch (bit) { + case atl_thermal_monitor_shift: + if (!val) + /* Disable throttling along with monitoring */ + flags &= ~atl_thermal_throttle; + else + if (!(hw->mcp.caps_high & atl_fw2_set_thermal)) { + atl_dev_err("Thermal monitoring not supported by firmware\n"); + ret = -EINVAL; + } + break; + + case atl_thermal_throttle_shift: + if (val && !(flags & atl_thermal_monitor)) { + atl_dev_err("Thermal monitoring needs to be enabled before enabling throttling\n"); + ret = -EINVAL; + } + break; + + case atl_thermal_ignore_lims_shift: + break; + + default: + ret = -EINVAL; + break; + } + if (ret) + return ret; + + flags &= ~BIT(bit); + flags |= val << bit; + + changed = flags ^ thermal->flags; + thermal->flags = flags; + + if (test_bit(ATL_ST_RESETTING, &hw->state)) + /* After reset, atl_fw_init() will apply the settings + * skipped here */ + return ret; + + if (changed & atl_thermal_monitor) { + ret = hw->mcp.ops->update_thermal(hw); + } else if (changed & atl_thermal_throttle && + hw->link_state.thermal_throttled) + hw->mcp.ops->set_link(hw, true); + + if (ret) + /* __atl_fw2_update_thermal() failed. Revert flag + * changes */ + thermal->flags ^= changed; + + return ret; +} + +static unsigned int atl_wdog_period = 1100; +module_param_named(wdog_period, atl_wdog_period, uint, 0644); + +int atl_fw_init(struct atl_hw *hw) +{ + uint32_t tries, reg, major; + int ret; + struct atl_mcp *mcp = &hw->mcp; + + tries = busy_wait(10000, mdelay(1), reg, atl_read(hw, 0x18), !reg); + if (!reg) { + atl_dev_err("Timeout waiting for FW version\n"); + return -EIO; + } + atl_dev_dbg("FW startup took %d ms\n", tries); + + major = (reg >> 24) & 0xff; + if (!major || major > 3) { + atl_dev_err("Unsupported FW major version: %u\n", major); + return -EINVAL; + } + if (major > 2) + major--; + mcp->ops = &atl_fw_ops[major - 1]; + mcp->fw_rev = reg; + + ret = mcp->ops->__wait_fw_init(hw); + if (ret) + return ret; + + ret = mcp->ops->__get_hbeat(hw, &mcp->phy_hbeat); + if (ret) + return ret; + mcp->next_wdog = jiffies + 2 * HZ; + + ret = mcp->ops->__get_link_caps(hw); + + return ret; +} + +void atl_fw_watchdog(struct atl_hw *hw) +{ + struct atl_mcp *mcp = &hw->mcp; + int ret; + uint16_t hbeat; + + if (mcp->wdog_disabled || !time_after(jiffies, mcp->next_wdog)) + return; + + if (test_bit(ATL_ST_RESETTING, &hw->state) || + !test_bit(ATL_ST_ENABLED, &hw->state)) + return; + + atl_lock_fw(hw); + + ret = mcp->ops->__get_hbeat(hw, &hbeat); + if (ret) { + atl_dev_err("FW watchdog: failure reading PHY heartbeat: %d\n", + -ret); + goto out; + } + + if (hbeat == mcp->phy_hbeat) { + atl_dev_err("FW watchdog: FW hang (PHY heartbeat stuck at %hd), resetting\n", hbeat); + set_bit(ATL_ST_RESET_NEEDED, &hw->state); + } + + mcp->phy_hbeat = hbeat; + +out: + mcp->next_wdog = jiffies + atl_wdog_period * HZ / 1000; + atl_unlock_fw(hw); +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.h new file mode 100644 index 000000000000..9115933157ff --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fw.h @@ -0,0 +1,304 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_FW_H_ +#define _ATL_FW_H_ + +#include +#include +#include + +struct atl_hw; +struct atl_nic; + +struct atl_mcp { + uint32_t fw_rev; + struct atl_fw_ops *ops; + uint32_t fw_stat_addr; + uint32_t rpc_addr; + uint32_t fw_settings_addr; + uint32_t fw_settings_len; + uint32_t req_high; + uint32_t req_high_mask; /* Clears link rate-dependend bits */ + uint32_t caps_low; + uint32_t caps_high; + uint32_t caps_ex; + struct mutex lock; + unsigned long next_wdog; + bool wdog_disabled; + uint16_t phy_hbeat; + uint32_t *fw_cfg_dump; +}; + +struct atl_link_type { + unsigned speed; + bool duplex; + unsigned ethtool_idx; + uint32_t fw_bits[2]; + const char *name; +}; + +enum atl_link_type_index { + atl_link_type_idx_10m, + atl_link_type_idx_100m, + atl_link_type_idx_1g, + atl_link_type_idx_2p5g, + atl_link_type_idx_5g, + atl_link_type_idx_10g, + atl_link_type_idx_10m_half, + atl_link_type_idx_100m_half, + atl_link_type_idx_1g_half, +}; + +extern struct atl_link_type atl_link_types[]; +extern const int atl_num_rates; + +struct atl_fw2_thermal_cfg { + uint32_t msg_id; + uint8_t shutdown_temp; + uint8_t high_temp; + uint8_t normal_temp; +}; + +#define atl_for_each_rate(idx, type) \ + for (idx = 0, type = atl_link_types; \ + idx < atl_num_rates; \ + idx++, type++) + +#define atl_define_bit(_name, _bit) \ + _name ## _shift = (_bit), \ + _name = BIT(_name ## _shift), + +enum atl_fw2_opts { + atl_define_bit(atl_fw2_pause, 3) + atl_define_bit(atl_fw2_asym_pause, 4) + atl_fw2_pause_mask = atl_fw2_pause | atl_fw2_asym_pause, + atl_define_bit(atl_fw2_fw_request, 12) + atl_define_bit(atl_fw2_macsec, 15) + atl_define_bit(atl_fw2_wake_on_link, 16) + atl_define_bit(atl_fw2_wake_on_link_force, 17) + atl_define_bit(atl_fw2_phy_temp, 18) + atl_define_bit(atl_fw2_set_thermal, 21) + atl_define_bit(atl_fw2_link_drop, 22) + atl_define_bit(atl_fw2_nic_proxy, 0x17) + atl_define_bit(atl_fw2_wol, 0x18) + atl_define_bit(atl_fw2_thermal_alarm, 29) + atl_define_bit(atl_fw2_statistics, 30) +}; + +enum atl_fw2_ex_caps { + atl_define_bit(atl_fw2_ex_caps_phy_ptp_en, 16) + atl_define_bit(atl_fw2_ex_caps_ptp_gpio_en, 20) + atl_define_bit(atl_fw2_ex_caps_phy_ctrl_ts_pin, 22) + atl_define_bit(atl_fw2_ex_caps_wol_ex, 23) + atl_define_bit(atl_fw2_ex_caps_mac_heartbeat, 25) + atl_define_bit(atl_fw2_ex_caps_msm_settings_apply, 26) +}; + +enum atl_fw2_wol_ex { + atl_define_bit(atl_fw2_wol_ex_wake_on_link_keep_rate, 0) + atl_define_bit(atl_fw2_wol_ex_wake_on_magic_keep_rate, 1) +}; + +enum atl_fw2_stat_offt { + atl_fw2_stat_phy_hbeat = 0x4c, + atl_fw2_stat_temp = 0x50, + atl_fw2_stat_ptp_offset = 0x64, + atl_fw2_stat_lcaps = 0x84, + atl_fw2_stat_settings_addr = 0x10c, + atl_fw2_stat_settings_len = 0x110, + atl_fw2_stat_caps_ex = 0x114, + atl_fw2_stat_gpio_pin = 0x118, +}; + +enum atl_fw2_settings_offt { + atl_fw2_setings_msm_opts = 0x90, + atl_fw2_setings_media_detect = 0x98, + atl_fw2_setings_wol_ex = 0x9c, +}; + +enum atl_fw2_msm_opts { + atl_define_bit(atl_fw2_settings_msm_opts_strip_pad, 0) +}; + +enum atl_fw2_fw_request { + atl_fw2_msm_settings_apply = 0x20, +}; + +enum atl_fc_mode { + atl_fc_none = 0, + atl_define_bit(atl_fc_rx, 0) + atl_define_bit(atl_fc_tx, 1) + atl_fc_full = atl_fc_rx | atl_fc_tx, +}; + +enum atl_thermal_flags { + atl_define_bit(atl_thermal_monitor, 0) + atl_define_bit(atl_thermal_throttle, 1) + atl_define_bit(atl_thermal_ignore_lims, 2) +}; + +struct atl_fc_state { + enum atl_fc_mode req; + enum atl_fc_mode prev_req; + enum atl_fc_mode cur; +}; + +enum atl_wake_flags { + atl_fw_wake_on_link = WAKE_PHY, + atl_fw_wake_on_magic = WAKE_MAGIC, + atl_fw_wake_on_link_rtpm = BIT(10), +}; + +#define ATL_EEE_BIT_OFFT 16 +#define ATL_EEE_MASK ~(BIT(ATL_EEE_BIT_OFFT) - 1) + +struct atl_link_state{ + /* The following three bitmaps use alt_link_types[] indices + * as link bit positions. Conversion to/from ethtool bits is + * done in atl_ethtool.c. */ + unsigned supported; + unsigned advertized; + unsigned lp_advertized; + unsigned prev_advertized; + int lp_lowest; /* Idx of lowest rate advertized by + * link partner in atl_link_types[] */ + int throttled_to; /* Idx of the rate we're throttled to */ + bool force_off; + bool thermal_throttled; + bool autoneg; + bool eee; + bool eee_enabled; + bool ptp_available; + bool ptp_datapath_up; + struct atl_link_type *link; + struct atl_fc_state fc; +}; + +enum macsec_msg_type { + macsec_cfg_msg = 0, + macsec_add_rx_sc_msg, + macsec_add_tx_sc_msg, + macsec_add_rx_sa_msg, + macsec_add_tx_sa_msg, + macsec_get_stats_msg, +}; + +struct __packed macsec_cfg_request { + u32 enabled; + u32 egress_threshold; + u32 ingress_threshold; + u32 interrupts_enabled; +}; + +struct __packed macsec_msg_fw_request { + u32 msg_id; /* not used */ + u32 msg_type; + + struct macsec_cfg_request cfg; +}; + +struct __packed macsec_msg_fw_response { + u32 result; +}; + +enum atl_gpio_pin_function { + GPIO_PIN_FUNCTION_NC, + GPIO_PIN_FUNCTION_VAUX_ENABLE, + GPIO_PIN_FUNCTION_EFUSE_BURN_ENABLE, + GPIO_PIN_FUNCTION_SFP_PLUS_DETECT, + GPIO_PIN_FUNCTION_TX_DISABLE, + GPIO_PIN_FUNCTION_RATE_SEL_0, + GPIO_PIN_FUNCTION_RATE_SEL_1, + GPIO_PIN_FUNCTION_TX_FAULT, + GPIO_PIN_FUNCTION_PTP0, + GPIO_PIN_FUNCTION_PTP1, + GPIO_PIN_FUNCTION_PTP2, + GPIO_PIN_FUNCTION_SIZE +}; + +struct __packed atl_ptp_offset_info { + u16 ingress_100; + u16 egress_100; + u16 ingress_1000; + u16 egress_1000; + u16 ingress_2500; + u16 egress_2500; + u16 ingress_5000; + u16 egress_5000; + u16 ingress_10000; + u16 egress_10000; +}; + +enum ptp_msg_type { + ptp_gpio_ctrl_msg = 0x11, + ptp_adj_freq_msg = 0x12, + ptp_adj_clock_msg = 0x13, +}; + +struct __packed ptp_gpio_ctrl { + u32 index; + u32 period; + u64 start; +}; + +struct __packed ptp_adj_freq { + u32 ns_mac; + u32 fns_mac; + u32 ns_phy; + u32 fns_phy; + u32 mac_ns_adj; + u32 mac_fns_adj; +}; + +struct __packed ptp_adj_clock { + u32 ns; + u32 sec; + int sign; +}; + +struct __packed ptp_msg_fw_request { + u32 msg_id; + union { + struct ptp_gpio_ctrl gpio_ctrl; + struct ptp_adj_freq adj_freq; + struct ptp_adj_clock adj_clock; + }; +}; + +struct atl_fw_ops { + void (*set_link)(struct atl_hw *hw, bool force); + struct atl_link_type *(*check_link)(struct atl_hw *hw); + int (*__wait_fw_init)(struct atl_hw *hw); + int (*__get_link_caps)(struct atl_hw *hw); + int (*restart_aneg)(struct atl_hw *hw); + void (*set_default_link)(struct atl_hw *hw); + int (*enable_wol)(struct atl_hw *hw, unsigned int wol_mode); + int (*get_phy_temperature)(struct atl_hw *hw, int *temp); + int (*dump_cfg)(struct atl_hw *hw); + int (*restore_cfg)(struct atl_hw *hw); + int (*set_phy_loopback)(struct atl_nic *nic, u32 mode); + int (*set_mediadetect)(struct atl_hw *hw, bool on); + int (*set_pad_stripping)(struct atl_hw *hw, bool on); + int (*send_macsec_req)(struct atl_hw *hw, + struct macsec_msg_fw_request *msg, + struct macsec_msg_fw_response *resp); + int (*__get_hbeat)(struct atl_hw *hw, uint16_t *hbeat); + int (*get_mac_addr)(struct atl_hw *hw, uint8_t *buf); + int (*update_thermal)(struct atl_hw *hw); + int (*send_ptp_req)(struct atl_hw *hw, struct ptp_msg_fw_request *msg); + void (*set_ptp)(struct atl_hw *hw, bool on); + int (*deinit)(struct atl_hw *hw); +}; + +int atl_read_mcp_word(struct atl_hw *hw, uint32_t offt, uint32_t *val); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.c new file mode 100644 index 000000000000..2b5e899c6bf8 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.c @@ -0,0 +1,905 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2018 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include "atl_common.h" +#include "atl_desc.h" + +static const char *atl_fwd_dir_str(struct atl_fwd_ring *ring) +{ + return ring->flags & ATL_FWR_TX ? "Tx" : "Rx"; +} + +static int atl_fwd_ring_tx(struct atl_fwd_ring *ring) +{ + return !!(ring->flags & ATL_FWR_TX); +} + +static void *atl_fwd_frag_vaddr(struct atl_fwd_buf_frag *frag, + struct atl_fwd_mem_ops *ops) +{ + if (ops->alloc_buf) + return frag->buf; + + return page_to_virt(frag->page); +} + +static int atl_fwd_get_frag(struct atl_fwd_ring *ring, int idx) +{ + struct page *pg; + dma_addr_t daddr = 0; + struct device *dev = &ring->nic->hw.pdev->dev; + struct atl_fwd_bufs *bufs = ring->bufs; + struct atl_fwd_buf_frag *frag = &bufs->frags[idx]; + struct atl_fwd_mem_ops *ops = ring->mem_ops; + + if (ops->alloc_buf) { + void *buf = ops->alloc_buf(dev, bufs->frag_size, + &daddr, GFP_KERNEL, ops); + + if (!IS_ERR_OR_NULL(buf)) { + frag->buf = buf; + frag->daddr = daddr; + return 0; + } else + return PTR_ERR(buf); + } + + pg = __dev_alloc_pages(GFP_KERNEL, bufs->order); + if (!pg) + return -ENOMEM; + + if (!(ring->flags & ATL_FWR_DONT_DMA_MAP)) { + daddr = dma_map_page(dev, pg, 0, PAGE_SIZE << bufs->order, + DMA_FROM_DEVICE); + + if (dma_mapping_error(dev, daddr)) { + __free_pages(pg, bufs->order); + return -ENOMEM; + } + } + + frag->daddr = daddr; + frag->page = pg; + + return 0; +} + +static void atl_fwd_free_bufs(struct atl_fwd_ring *ring) +{ + struct atl_nic *nic = ring->nic; + struct device *dev = &nic->hw.pdev->dev; + struct atl_fwd_bufs *bufs = ring->bufs; + struct atl_fwd_mem_ops *ops = ring->mem_ops; + int ring_size = ring->hw.size; + int order; + size_t frag_size; + int i; + + if (!bufs) + return; + + order = bufs->order; + frag_size = bufs->frag_size; + + if (bufs->daddr_vec) + dma_free_coherent(dev, ring_size * sizeof(dma_addr_t), + bufs->daddr_vec, bufs->daddr_vec_base); + + if (ring->flags & ATL_FWR_WANT_VIRT_BUF_VEC) + kfree(bufs->vaddr_vec); + + for (i = 0; i < bufs->num_pages; i++) { + struct atl_fwd_buf_frag *frag = &bufs->frags[i]; + + if (ops->free_buf) { + if (frag->buf) + ops->free_buf(frag->buf, dev, frag_size, + frag->daddr, ops); + continue; + } + + if (frag->page) { + if (!(ring->flags & ATL_FWR_DONT_DMA_MAP)) + dma_unmap_page(dev, frag->daddr, + PAGE_SIZE << order, + DMA_FROM_DEVICE); + __free_pages(frag->page, order); + } + } + + kfree(bufs); + ring->bufs = NULL; +} + +static int atl_fwd_alloc_bufs(struct atl_fwd_ring *ring, int order) +{ + struct atl_nic *nic = ring->nic; + int flags = ring->flags; + int ring_size = ring->hw.size; + int buf_size = ring->buf_size; + struct device *dev = &nic->hw.pdev->dev; + struct atl_fwd_mem_ops *ops = ring->mem_ops; + struct atl_fwd_buf_frag *frag; + struct atl_fwd_bufs *bufs; + int num_pages, i; + int ret; + unsigned int pg_off = 0; + bool want_dvec = !!(flags & ATL_FWR_WANT_DMA_BUF_VEC); + bool want_vvec = !!(flags & ATL_FWR_WANT_VIRT_BUF_VEC); + size_t frag_size; + + if (!(flags & ATL_FWR_ALLOC_BUFS)) + return 0; + + if (flags & ATL_FWR_CONTIG_BUFS) { + frag_size = buf_size * ring_size; + order = get_order(frag_size); + num_pages = 1; + } else { + int bufs_per_page; + + frag_size = PAGE_SIZE << order; + bufs_per_page = frag_size / buf_size; + num_pages = ring_size / bufs_per_page + + !!(ring_size % bufs_per_page); + } + + bufs = kzalloc(sizeof(*bufs) + + sizeof(struct atl_fwd_buf_frag) * num_pages, + GFP_KERNEL); + if (!bufs) { + atl_nic_err("%s: couldn't alloc buffers structure\n", __func__); + return -ENOMEM; + } + + ring->bufs = bufs; + bufs->num_pages = num_pages; + bufs->order = order; + bufs->frag_size = frag_size; + + for (i = 0; i < num_pages; i++) { + ret = atl_fwd_get_frag(ring, i); + if (ret) { + atl_nic_err("%s: couldn't alloc buffer page (order %d)\n", + __func__, order); + goto free; + } + } + + frag = &bufs->frags[0]; + + if (want_dvec) { + ret = -ENOMEM; + bufs->daddr_vec = dma_alloc_coherent(dev, + ring_size * sizeof(dma_addr_t), + &bufs->daddr_vec_base, GFP_KERNEL); + if (!bufs->daddr_vec) { + atl_nic_err("%s: couldn't alloc DMA addr table\n", + __func__); + goto free; + } + } else + bufs->daddr_vec_base = frag[0].daddr; + + if (want_vvec) { + ret = -ENOMEM; + bufs->vaddr_vec = kcalloc(ring_size, sizeof(void *), + GFP_KERNEL); + if (!bufs->vaddr_vec) { + atl_nic_err("%s: couldn't alloc virtual addr table\n", + __func__); + goto free; + } + } else + bufs->vaddr_vec = atl_fwd_frag_vaddr(frag, ops); + + if (!(want_dvec || want_vvec)) + return 0; + + for (i = 0; i < ring_size; i++) { + dma_addr_t daddr = frag->daddr + pg_off; + + if (want_dvec) + bufs->daddr_vec[i] = daddr; + if (want_vvec) + bufs->vaddr_vec[i] = atl_fwd_frag_vaddr(frag, ops) + + pg_off; + + pg_off += buf_size; + if (pg_off + buf_size <= frag_size) + continue; + + frag++; + pg_off = 0; + } + + return 0; + +free: + atl_fwd_free_bufs(ring); + return ret; +} + +static void atl_fwd_update_im(struct atl_fwd_ring *ring) +{ + struct atl_hw *hw = &ring->nic->hw; + int idx = ring->idx; + uint32_t addr, tx_reg; + + if (hw->chip_id == ATL_ANTIGUA) + tx_reg = ATL2_TX_INTR_MOD_CTRL(idx); + else + tx_reg = ATL_TX_INTR_MOD_CTRL(idx); + addr = atl_fwd_ring_tx(ring) ? tx_reg : + ATL_RX_INTR_MOD_CTRL(idx); + + atl_write(hw, addr, (ring->intr_mod_max / 2) << 0x10 | + (ring->intr_mod_min / 2) << 8 | 2); +} + +static void atl_fwd_init_descr(struct atl_fwd_ring *fwd_ring) +{ + struct atl_hw_ring *ring = &fwd_ring->hw; + int dir_tx = atl_fwd_ring_tx(fwd_ring); + struct atl_fwd_buf_frag *frag = NULL; + int buf_size = fwd_ring->buf_size; + int ring_size = ring->size; + size_t frag_size = 0; + unsigned int pg_off; + int i; + + memset(ring->descs, 0, ring_size * sizeof(*ring->descs)); + + if (!(fwd_ring->flags & ATL_FWR_ALLOC_BUFS)) + return; + + frag = &fwd_ring->bufs->frags[0]; + frag_size = fwd_ring->bufs->frag_size; + + if (!(fwd_ring->flags & ATL_FWR_DONT_DMA_MAP)) { + for (pg_off = 0, i = 0; i < ring_size; i++) { + union atl_desc *desc = &ring->descs[i]; + dma_addr_t daddr = frag->daddr + pg_off; + + if (dir_tx) { + /* init both daddr and dd for both cases: + * daddr for head pointer writeback + * dd for the descriptor writeback */ + desc->tx.daddr = daddr; + desc->tx.dd = 1; + } else { + desc->rx.daddr = daddr; + } + + pg_off += buf_size; + if (pg_off + buf_size <= frag_size) + continue; + + frag++; + pg_off = 0; + } + } +} + +static void atl_fwd_init_ring(struct atl_fwd_ring *fwd_ring) +{ + struct atl_hw *hw = &fwd_ring->nic->hw; + struct atl_hw_ring *ring = &fwd_ring->hw; + unsigned int flags = fwd_ring->flags; + int dir_tx = atl_fwd_ring_tx(fwd_ring); + int idx = fwd_ring->idx; + int lxo_bit = !!(flags & ATL_FWR_LXO); + + /* Reinit descriptors as they could be stale after hardware reset */ + atl_fwd_init_descr(fwd_ring); + + atl_write(hw, ATL_RING_BASE_LSW(ring), ring->daddr); + atl_write(hw, ATL_RING_BASE_MSW(ring), upper_32_bits(ring->daddr)); + + if (dir_tx) { + atl_write(hw, ATL_TX_RING_THRESH(ring), + 8 << 8 | 8 << 0x10 | 24 << 0x18); + atl_write(hw, ATL_TX_RING_CTL(ring), ring->size); + + atl_write_bit(hw, ATL_TX_LSO_CTRL, idx, lxo_bit); + } else { + uint32_t ctrl = ring->size | + !!(flags & ATL_FWR_VLAN) << 29; + + atl_write(hw, ATL_RX_RING_BUF_SIZE(ring), + fwd_ring->buf_size / 1024); + atl_write(hw, ATL_RX_RING_THRESH(ring), + 8 << 0x10 | 24 << 0x18); + atl_write(hw, ATL_RX_RING_TAIL(ring), ring->size - 1); + atl_write(hw, ATL_RX_RING_CTL(ring), ctrl); + + if (lxo_bit) + atl_write_bits(hw, ATL_RX_LRO_PKT_LIM(idx), + (idx & 7) * 4, 2, 3); + + atl_write_bit(hw, ATL_RX_LRO_CTRL1, idx, lxo_bit); + atl_write_bit(hw, ATL_INTR_RSC_EN, idx, lxo_bit); + } + + atl_fwd_update_im(fwd_ring); +} + +void atl_fwd_release_ring(struct atl_fwd_ring *ring) +{ + struct atl_nic *nic = ring->nic; + int idx = ring->idx; + int dir_tx = atl_fwd_ring_tx(ring); + struct atl_fwd *fwd = &nic->fwd; + unsigned long *map = &fwd->ring_map[dir_tx]; + struct atl_fwd_ring **rings = fwd->rings[dir_tx]; + struct atl_fwd_mem_ops *ops = ring->mem_ops; + struct device *dev = &nic->hw.pdev->dev; + struct atl_hw_ring *hwring = &ring->hw; + + atl_fwd_disable_ring(ring); + + if (ring->evt) { + atl_fwd_disable_event(ring->evt); + atl_fwd_release_event(ring->evt); + } + + atl_do_reset(nic); + + __clear_bit(idx, map); + rings[idx - ATL_FWD_RING_BASE] = NULL; + atl_fwd_free_bufs(ring); + if (ops->free_descs) + ops->free_descs(hwring->descs, dev, + hwring->size * sizeof(*hwring->descs), hwring->daddr, + ops); + else + atl_free_descs(nic, &ring->hw, 0); + kfree(ring); +} +EXPORT_SYMBOL(atl_fwd_release_ring); + +static unsigned int atl_fwd_rx_mod_max = 25, atl_fwd_rx_mod_min = 15, + atl_fwd_tx_mod_max = 25, atl_fwd_tx_mod_min = 15; +atl_module_param(fwd_rx_mod_max, uint, 0644); +atl_module_param(fwd_rx_mod_min, uint, 0644); +atl_module_param(fwd_tx_mod_max, uint, 0644); +atl_module_param(fwd_tx_mod_min, uint, 0644); + +static struct atl_fwd_mem_ops null_ops; + +struct atl_fwd_ring *atl_fwd_request_ring(struct net_device *ndev, + int flags, int ring_size, int buf_size, int page_order, + struct atl_fwd_mem_ops *ops) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_fwd *fwd = &nic->fwd; + int dir_tx = !!(flags & ATL_FWR_TX); + unsigned long *map = &fwd->ring_map[dir_tx]; + struct atl_fwd_ring **rings = fwd->rings[dir_tx], *ring; + struct device *dev = &nic->hw.pdev->dev; + struct atl_hw_ring *hwring; + int ret = -ENOMEM; + int idx; + + if (ring_size & 7 || ring_size > ATL_MAX_RING_SIZE) { + atl_nic_err("%s: bad ring size %d, must be no more than %d and a multiple of 8\n", + __func__, ring_size, ATL_MAX_RING_SIZE); + return ERR_PTR(-EINVAL); + } + + if (buf_size & 1023 || buf_size > 16 * 1024) { + atl_nic_err("%s: bad buffer size %d, must be no more than 16k and a multiple of 1024\n", + __func__, buf_size); + return ERR_PTR(-EINVAL); + } + + if (!ops) + ops = &null_ops; + + if ((ops->alloc_buf && !ops->free_buf) || + (ops->free_buf && !ops->alloc_buf)) { + atl_nic_err("%s: must provide either both buffer allocator and deallocator or none\n", + __func__); + return ERR_PTR(-EINVAL); + } + + if ((ops->alloc_descs && !ops->free_descs) || + (ops->free_descs && !ops->alloc_descs)) { + atl_nic_err("%s: must provide either both descriptor ring allocator and deallocator or none\n", + __func__); + return ERR_PTR(-EINVAL); + } + + if ((flags & (ATL_FWR_WANT_DMA_BUF_VEC | ATL_FWR_DONT_DMA_MAP)) == + (ATL_FWR_WANT_DMA_BUF_VEC | ATL_FWR_DONT_DMA_MAP)) { + atl_nic_err("%s: ATL_FWR_WANT_DMA_BUF_VEC and ATL_FWR_DONT_DMA_MAP flags are mutually exclusive\n", + __func__); + return ERR_PTR(-EINVAL); + } + + idx = find_next_zero_bit(map, ATL_FWD_RING_BASE + ATL_NUM_FWD_RINGS, + ATL_FWD_RING_BASE); + if (idx >= ATL_FWD_RING_BASE + ATL_NUM_FWD_RINGS) { + atl_nic_err("%s: no more rings available\n", __func__); + return ERR_PTR(ret); + } + + ring = kzalloc(sizeof(*ring), GFP_KERNEL); + if (!ring) { + atl_nic_err("%s: couldn't alloc ring structure\n", __func__); + return ERR_PTR(ret); + } + + ring->nic = nic; + ring->idx = idx; + ring->flags = flags; + hwring = &ring->hw; + hwring->size = ring_size; + ring->buf_size = buf_size; + ring->mem_ops = ops; + + if (ops->alloc_descs) { + void *descs; + dma_addr_t daddr; + + descs = ops->alloc_descs(dev, + ring_size * sizeof(*hwring->descs), &daddr, GFP_KERNEL, + ops); + if (!IS_ERR_OR_NULL(descs)) { + hwring->descs = descs; + hwring->daddr = daddr; + ret = 0; + } else + ret = PTR_ERR(descs); + } else + ret = atl_alloc_descs(nic, hwring, 0); + + if (ret) { + atl_nic_err("%s: couldn't alloc the ring\n", __func__); + goto free_ring; + } + + hwring->reg_base = dir_tx ? ATL_TX_RING(idx) : ATL_RX_RING(idx); + + ret = atl_fwd_alloc_bufs(ring, page_order); + if (ret) + goto free_descs; + + __set_bit(idx, map); + rings[idx - ATL_FWD_RING_BASE] = ring; + + if (dir_tx) { + ring->intr_mod_max = atl_fwd_tx_mod_max; + ring->intr_mod_min = atl_fwd_tx_mod_min; + } else { + ring->intr_mod_max = atl_fwd_rx_mod_max; + ring->intr_mod_min = atl_fwd_rx_mod_min; + } + + atl_fwd_init_ring(ring); + return ring; + +free_descs: + if (ops->free_descs) + ops->free_descs(hwring->descs, dev, + hwring->size * sizeof(*hwring->descs), hwring->daddr, + ops); + else + atl_free_descs(nic, hwring, 0); + +free_ring: + kfree(ring); + return ERR_PTR(ret); +} +EXPORT_SYMBOL(atl_fwd_request_ring); + +int atl_fwd_set_ring_intr_mod(struct atl_fwd_ring *ring, int min, int max) +{ + struct atl_nic *nic = ring->nic; + if (atl_fwd_ring_tx(ring) && ring->evt && + ring->evt->flags & ATL_FWD_EVT_TXWB) { + struct atl_nic *nic = ring->nic; + + atl_nic_err("%s: Interrupt moderation not supported for head pointer writeback events\n", + __func__); + return -EINVAL; + } + + if (min >= 0) { + if (min > 511) { + atl_nic_err("%s: min delay out of range (0..511): %d\n", + __func__, min); + return -EINVAL; + } + ring->intr_mod_min = min; + } + + if (max >= 0) { + if (max > 1023) { + atl_nic_err("%s: max delay out of range (0..1023): %d\n", + __func__, max); + return -EINVAL; + } + ring->intr_mod_max = max; + } + + atl_fwd_update_im(ring); + return 0; +} +EXPORT_SYMBOL(atl_fwd_set_ring_intr_mod); + +void atl_fwd_release_rings(struct atl_nic *nic) +{ + struct atl_fwd_ring *ring; + int i; + + for (i = 0; i < ATL_NUM_FWD_RINGS * ATL_FWDIR_NUM; i++) { + ring = nic->fwd.rings[i % ATL_FWDIR_NUM][i / ATL_FWDIR_NUM]; + if (ring) + atl_fwd_release_ring(ring); + } +} + +int atl_fwd_enable_ring(struct atl_fwd_ring *ring) +{ + struct atl_hw *hw = &ring->nic->hw; + + atl_set_bits(hw, ATL_RING_CTL(&ring->hw), BIT(31)); + atl_clear_bits(hw, ATL_RING_CTL(&ring->hw), BIT(30)); + ring->state |= ATL_FWR_ST_ENABLED; + + return 0; +} +EXPORT_SYMBOL(atl_fwd_enable_ring); + +void atl_fwd_disable_ring(struct atl_fwd_ring *ring) +{ + struct atl_hw *hw = &ring->nic->hw; + + if (!(ring->state & ATL_FWR_ST_ENABLED)) + return; + + atl_clear_bits(hw, ATL_RING_CTL(&ring->hw), BIT(31)); + atl_set_bits(hw, ATL_RING_CTL(&ring->hw), BIT(30)); + ring->state &= ~ATL_FWR_ST_ENABLED; +} +EXPORT_SYMBOL(atl_fwd_disable_ring); + +static void __iomem *atl_msix_bar(struct atl_nic *nic) +{ + struct pci_dev *pdev = nic->hw.pdev; + struct msi_desc *msi; + + if (!pdev->msix_enabled) + return NULL; + + msi = list_first_entry(dev_to_msi_list(&pdev->dev), + struct msi_desc, list); + return msi->mask_base; +} + +static int atl_fwd_set_msix_vec(struct atl_nic *nic, struct atl_fwd_event *evt) +{ + int idx = evt->idx; + uint64_t addr = evt->msi_addr; + uint32_t data = evt->msi_data; + uint32_t ctrl; + void __iomem *desc = atl_msix_bar(nic); + + if (!desc) + return -EIO; + + desc += idx * PCI_MSIX_ENTRY_SIZE; + + /* MSI-X table updates must be atomic, so mask first */ + ctrl = readl(desc + PCI_MSIX_ENTRY_VECTOR_CTRL); + writel(ctrl | PCI_MSIX_ENTRY_CTRL_MASKBIT, + desc + PCI_MSIX_ENTRY_VECTOR_CTRL); + + /* Program the vector */ + writel(addr & (BIT_ULL(32) - 1), desc + PCI_MSIX_ENTRY_LOWER_ADDR); + writel(addr >> 32, desc + PCI_MSIX_ENTRY_UPPER_ADDR); + writel(data, desc + PCI_MSIX_ENTRY_DATA); + + /* Unmask */ + writel(ctrl & ~PCI_MSIX_ENTRY_CTRL_MASKBIT, + desc + PCI_MSIX_ENTRY_VECTOR_CTRL); + + return 0; +} + +void atl_fwd_release_event(struct atl_fwd_event *evt) +{ + struct atl_fwd_ring *ring = evt->ring; + struct atl_nic *nic = ring->nic; + unsigned long *map = &nic->fwd.msi_map; + int idx = evt->idx; + + if (ring->evt != evt) { + atl_nic_err("%s: attempt to release unset event\n", __func__); + return; + } + + atl_fwd_disable_event(evt); + + ring->evt = NULL; + + if (evt->flags & ATL_FWD_EVT_TXWB) + return; + + __clear_bit(idx, map); + atl_set_intr_bits(&nic->hw, ring->idx, + atl_fwd_ring_tx(ring) ? -1 : ATL_NUM_MSI_VECS, + atl_fwd_ring_tx(ring) ? ATL_NUM_MSI_VECS : -1); +} +EXPORT_SYMBOL(atl_fwd_release_event); + +static int atl_fwd_init_event(struct atl_fwd_event *evt) +{ + struct atl_fwd_ring *ring = evt->ring; + int dir_tx = atl_fwd_ring_tx(ring); + struct atl_nic *nic = ring->nic; + struct atl_hw *hw = &nic->hw; + bool tx_wb = !!(evt->flags & ATL_FWD_EVT_TXWB); + int idx; + int ret; + + if (tx_wb) { + struct atl_hw_ring *hwring = &ring->hw; + + atl_write(hw, ATL_TX_RING_HEAD_WB_LSW(hwring), + evt->tx_head_wrb); + atl_write(hw, ATL_TX_RING_HEAD_WB_MSW(hwring), + upper_32_bits(evt->tx_head_wrb)); + return 0; + } + + idx = evt->idx; + + ret = atl_fwd_set_msix_vec(nic, evt); + if (ret) + return ret; + + atl_set_intr_bits(&nic->hw, ring->idx, + dir_tx ? -1 : idx, + dir_tx ? idx : -1); + + atl_write_bit(hw, ATL_INTR_AUTO_CLEAR, idx, 1); + atl_write_bit(hw, ATL_INTR_AUTO_MASK, idx, + !!(evt->flags & ATL_FWD_EVT_AUTOMASK)); + + return 0; +} + +int atl_fwd_request_event(struct atl_fwd_event *evt) +{ + struct atl_fwd_ring *ring = evt->ring; + struct atl_nic *nic = ring->nic; + unsigned long *map = &nic->fwd.msi_map; + bool tx_wb = !!(evt->flags & ATL_FWD_EVT_TXWB); + int idx; + int ret; + + if (ring->evt) { + atl_nic_err("%s: event already set for %s ring %d\n", + __func__, atl_fwd_dir_str(ring), ring->idx); + return -EEXIST; + } + + if (!tx_wb && !(nic->flags & ATL_FL_MULTIPLE_VECTORS)) { + atl_nic_err("%s: MSI-X interrupts are disabled\n", __func__); + return -EINVAL; + } + + if (tx_wb && !atl_fwd_ring_tx(ring)) { + atl_nic_err("%s: head pointer writeback events supported " + "on Tx rings only\n", __func__); + return -EINVAL; + } + + if ((evt->flags & (ATL_FWD_EVT_TXWB | ATL_FWD_EVT_AUTOMASK)) == + (ATL_FWD_EVT_TXWB | ATL_FWD_EVT_AUTOMASK)) { + atl_nic_err("%s: event automasking supported " + "for MSI events only\n", __func__); + return -EINVAL; + } + + ring->evt = evt; + + if (tx_wb) { + ret = atl_fwd_init_event(evt); + if (ret) + goto fail; + + return 0; + } + + idx = find_next_zero_bit(map, ATL_NUM_MSI_VECS, ATL_FWD_MSI_BASE); + if (idx >= ATL_NUM_MSI_VECS) { + atl_nic_err("%s: no MSI vectors left\n", __func__); + ret = -ENOMEM; + goto fail; + } + + evt->idx = idx; + + ret = atl_fwd_init_event(evt); + if (ret) + goto fail; + + __set_bit(idx, map); + + return 0; + +fail: + ring->evt = NULL; + return ret; +} +EXPORT_SYMBOL(atl_fwd_request_event); + +int atl_fwd_enable_event(struct atl_fwd_event *evt) +{ + struct atl_fwd_ring *ring = evt->ring; + struct atl_hw *hw = &ring->nic->hw; + + if (evt->flags & ATL_FWD_EVT_TXWB) { + if (ring->state & ATL_FWR_ST_ENABLED) + return -EINVAL; + + atl_write_bit(hw, ATL_TX_RING_CTL(&ring->hw), 28, 1); + ring->state |= ATL_FWR_ST_EVT_ENABLED; + return 0; + } + + atl_intr_enable(hw, BIT(evt->idx)); + ring->state |= ATL_FWR_ST_EVT_ENABLED; + return 0; +} +EXPORT_SYMBOL(atl_fwd_enable_event); + +int atl_fwd_disable_event(struct atl_fwd_event *evt) +{ + struct atl_fwd_ring *ring = evt->ring; + struct atl_hw *hw = &ring->nic->hw; + + if (evt->flags & ATL_FWD_EVT_TXWB) { + if (ring->state & ATL_FWR_ST_ENABLED) + return -EINVAL; + + atl_write_bit(hw, ATL_TX_RING_CTL(&ring->hw), 28, 0); + ring->state &= ~ATL_FWR_ST_EVT_ENABLED; + return 0; + } + + atl_intr_disable(hw, BIT(evt->idx)); + ring->state &= ~ATL_FWR_ST_EVT_ENABLED; + return 0; +} +EXPORT_SYMBOL(atl_fwd_disable_event); + +int atl_fwd_receive_skb(struct net_device *ndev, struct sk_buff *skb) +{ + struct atl_nic *nic = netdev_priv(ndev); + + nic->stats.rx_fwd.packets++; + nic->stats.rx_fwd.bytes += skb->len; + + skb->protocol = eth_type_trans(skb, ndev); + return netif_rx(skb); +} +EXPORT_SYMBOL(atl_fwd_receive_skb); + +int atl_fwd_napi_receive_skb(struct net_device *ndev, struct sk_buff *skb) +{ + struct atl_nic *nic = netdev_priv(ndev); + + nic->stats.rx_fwd.packets++; + nic->stats.rx_fwd.bytes += skb->len; + + skb->protocol = eth_type_trans(skb, ndev); + return netif_receive_skb(skb); +} +EXPORT_SYMBOL(atl_fwd_napi_receive_skb); + +int atl_fwd_transmit_skb(struct net_device *ndev, struct sk_buff *skb) +{ + skb->dev = ndev; + return dev_queue_xmit(skb); +} +EXPORT_SYMBOL(atl_fwd_transmit_skb); + +int atl_fwd_register_notifier(struct net_device *ndev, + struct notifier_block *n) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_fwd *fwd = &nic->fwd; + + return blocking_notifier_chain_register(&fwd->nh_clients, n); +} +EXPORT_SYMBOL(atl_fwd_register_notifier); + +int atl_fwd_unregister_notifier(struct net_device *ndev, + struct notifier_block *n) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_fwd *fwd = &nic->fwd; + + return blocking_notifier_chain_unregister(&fwd->nh_clients, n); +} +EXPORT_SYMBOL(atl_fwd_unregister_notifier); + +void atl_fwd_notify(struct atl_nic *nic, enum atl_fwd_notify notif, void *data) +{ + blocking_notifier_call_chain(&nic->fwd.nh_clients, notif, data); +} + +int atl_fwd_reconfigure_rings(struct atl_nic *nic) +{ + struct atl_fwd_ring *ring; + int i; + int ret; + + for (i = 0; i < ATL_NUM_FWD_RINGS * ATL_FWDIR_NUM; i++) { + ring = nic->fwd.rings[i % ATL_FWDIR_NUM][i / ATL_FWDIR_NUM]; + + if (!ring) + continue; + + atl_fwd_init_ring(ring); + + if (ring->evt) { + ret = atl_fwd_init_event(ring->evt); + if (ret) + return ret; + + if (ring->state & ATL_FWR_ST_EVT_ENABLED) { + ret = atl_fwd_enable_event(ring->evt); + if (ret) + return ret; + } + } + } + + return 0; +} + +int atl_fwd_suspend_rings(struct atl_nic *nic) +{ + atl_fwd_notify(nic, ATL_FWD_NOTIFY_RESET_PREPARE, NULL); + + return 0; +} + +int atl_fwd_resume_rings(struct atl_nic *nic) +{ + struct atl_fwd_ring *ring; + int i; + int ret; + + ret = atl_fwd_reconfigure_rings(nic); + if (ret) + goto err; + + atl_fwd_notify(nic, ATL_FWD_NOTIFY_RESET_COMPLETE, NULL); + + for (i = 0; i < ATL_NUM_FWD_RINGS * ATL_FWDIR_NUM; i++) { + ring = nic->fwd.rings[i % ATL_FWDIR_NUM][i / ATL_FWDIR_NUM]; + + if (!ring) + continue; + + if ((ring->state & ATL_FWR_ST_ENABLED)) + atl_fwd_enable_ring(ring); + } +err: + return ret; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.h new file mode 100644 index 000000000000..57b31320be5a --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwd.h @@ -0,0 +1,408 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2018 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_FWD_H_ +#define _ATL_FWD_H_ + +#include "atl_common.h" + +/* Each incompatible API change bumps the API version */ +#define ATL_FWD_API_VERSION 3 + +struct atl_fwd_event; + +struct atl_fwd_buf_frag { + union { + void *buf; /* Valid when custom allocator is + * used */ + struct page *page; /* Valid otherwise */ + }; + dma_addr_t daddr; +}; + +/** + * atl_fwd_rxbufs - offload engine's ring's Rx buffers + * + * Buffers are allocated by the driver when a ring is created + * + * The entire buffer space for the ring may optionally be + * allocated as a single physically-contiguous block. + * + * Descriptors are overwritten with the write-back descriptor + * format on Rx and optionally on Tx. To simplify Rx descriptor + * refill by the offload engine, vectors containing virtual + * addresses and DMA-addresses of each buffer are provided in + * @vaddr_vec and @daddr_vec respectively if + * @ATL_FWR_WANT_VIRT_BUF_VEC or @ATL_FWR_WANT_DMA_BUF_VEC + * flags are set on @atl_fwd_request_ring(). + * + * If the resepective buffer vector request flags are not set, + * @daddr_vec_base contains the DMA address of the first buffer + * page and @vaddr_vec contains its virtual address. + * + * @daddr_vec_base: DMA address of the base of the @daddr_vec + * @daddr_vec: A vector of buffers' DMA ddresses + * @vaddr_vec: A vector of buffers' virtual addresses + * or first buffer's virtual address + * depending on ring flags + */ +struct atl_fwd_bufs { + dma_addr_t daddr_vec_base; + dma_addr_t *daddr_vec; + void **vaddr_vec; + + /* The following is not part of API and subject to change */ + int num_pages; + int order; + size_t frag_size; + struct atl_fwd_buf_frag frags[0]; +}; + +union atl_desc; + +/** + * atl_hw_ring - low leverl descriptor ring structure + * + * @descs: Pointer to the descriptor ring + * @size: Number of descriptors in the ring + * @reg_base: Offset of ring's register block from start of + * BAR 0 + * @daddr: DMA address of the ring + */ +/* atl_hw_ring defined in "atl_hw.h" */ + +/** + * + * atl_fwd_mem_ops - memory management handlers for the ring + * + * Custom memory allocators may be provided to + * @atl_fwd_request_ring() to allocate the memory for the + * descriptor ring and buffers. + * + * Unless @ATL_FWR_DONT_DMA_MAP is included in flags on + * @atl_fwd_request_ring() call, the buffer allocator must also + * DMA-map the memory they allocate. If @ATL_FWR_DONT_DMA_MAP is + * set, @daddr parameter may be ignored by the buffer allocator + * and should be ignored by the deallocator. + * + * The ring allocator must DMA-map the memory unconditionally + * because the mapping is required to program the hardware ring + * registers. + * + * A private data pointer @private is reserved for offload + * engine's use. A pointer to the ops struct is provided to each + * allocator / deallocator so they can get to the private + * data. The @ops argument can be ignored by allocator / + * deallocator functions if access to private data is not needed. + * + * Either both allocator and deallocator or none of them must be + * provided for each memory type. + * + * Deallocator functions may ignore the size parameter. + */ +struct atl_fwd_mem_ops { + void *(*alloc_descs)(struct device *dev, size_t size, dma_addr_t *daddr, + gfp_t gfp, struct atl_fwd_mem_ops *ops); + void *(*alloc_buf)(struct device *dev, size_t size, dma_addr_t *daddr, + gfp_t gfp, struct atl_fwd_mem_ops *ops); + void (*free_descs)(void *buf, struct device *dev, size_t size, + dma_addr_t daddr, struct atl_fwd_mem_ops *ops); + void (*free_buf)(void *buf, struct device *dev, size_t size, + dma_addr_t daddr, struct atl_fwd_mem_ops *ops); + void *private; +}; + +/** + * atl_fwd_ring - Offload engine-controlled ring + * + * Buffer space is allocated by the driver on ring creation. + * + * @hw: Low-level ring information + * @evt: Ring's event, either an MSI-X vector (either + * Tx or Rx) or head pointer writeback address + * (Tx ring only). NULL on ring allocation, set + * by atl_fwd_request_event() + * @bufs: Ring's buffers. Allocated only if + * @ATL_FWR_ALLOC_BUFS flag is set on ring + * request. + * @nic: struct atl_nic backreference + * @idx: Ring index + * @mem_ops: Memory allocators provided at ring creation + */ +struct atl_fwd_ring { + struct atl_hw_ring hw; + struct atl_fwd_event *evt; + struct atl_fwd_bufs *bufs; + struct atl_nic *nic; + int idx; + struct atl_fwd_mem_ops *mem_ops; + + /* The following is not part of API and subject to change */ + unsigned int flags; + unsigned long state; + int buf_size; + unsigned intr_mod_min; + unsigned intr_mod_max; +}; + +enum atl_fwd_event_flags { + ATL_FWD_EVT_TXWB = BIT(0), /* Event type: 0 for MSI, 1 for Tx + * head WB */ + ATL_FWD_EVT_AUTOMASK = BIT(1), /* Disable event after + * raising, MSI only. */ +}; + +/** + * atl_fwd_event - Ring's notification event + * + * @flags Event type and flags + * @ring Ring backreference + * @msi_addr MSI message address + * @msi_data MSI message data + * @idx MSI index (0 .. 31) + * @tx_head_wrb Tx head writeback location + */ +struct atl_fwd_event { + enum atl_fwd_event_flags flags; + struct atl_fwd_ring *ring; + union { + struct { + dma_addr_t msi_addr; + uint32_t msi_data; + int idx; + }; + dma_addr_t tx_head_wrb; + }; +}; + +enum atl_fwd_ring_flags { + ATL_FWR_TX = BIT(0), /* Direction: 0 for Rx, 1 for Tx */ + ATL_FWR_VLAN = BIT(1), /* Enable VLAN tag stripping / insertion */ + ATL_FWR_LXO = BIT(2), /* Enable LRO / LSO */ + ATL_FWR_ALLOC_BUFS = BIT(3), /* Allocate buffers */ + ATL_FWR_CONTIG_BUFS = BIT(4), /* Alloc buffers as physically + * contiguous. May fail if + * total buffer space required + * is larger than a max-order + * compound page. */ + ATL_FWR_WANT_VIRT_BUF_VEC = BIT(5), /* Alloc and fill + * per-buffer virtual address + * vectors. If unset, first + * buffer's vaddr is provided + * in ring's @vaddr_vec */ + ATL_FWR_WANT_DMA_BUF_VEC = BIT(6), /* Alloc and fill + * per-buffer DMA address + * vectors. If unset, first + * buffer's daddr is provided + * in ring's + * @daddr_vec_base */ + ATL_FWR_DONT_DMA_MAP = BIT(7), /* Don't DMA-map buffers, the + * caller will take care of + * that. Mutually exclusive + * with + * @ATL_FWR_WANT_DMA_BUF_VECS + * If an external buffer + * allocator is provided to + * @atl_fwd_request_ring() + * along with this flag, the + * allocator need not provide + * the DMA address. */ +}; + +/** + * atl_fwd_request_ring() - Create a ring for an offload engine + * + * @ndev: network device + * @flags: ring flags + * @ring_size: number of descriptors + * @buf_size: individual buffer's size + * @page_order: page order to use when @ATL_FWR_CONTIG_BUFS is + * not set + * @mem_ops: optional custom memory allocators, may be NULL + * + * atl_fwd_request_ring() creates a ring for an offload engine, + * allocates buffer memory if @ATL_FWR_ALLOC_BUFS flag is set, + * initializes ring's registers, and, unless the @ATL_FWR_DONT_DMA_MAP + * flag is set, fills the address fields in descriptors. Ring is + * inactive until explicitly enabled via atl_fwd_enable_ring(). + * + * Buffers can be allocated either as a single physically-contiguous + * compound page, or as a sequence of compound pages of @page_order + * order. In the latter case, depending on the requested buffer size, + * tweaking the page order allows to pack buffers into buffer pages + * with less wasted space. + * + * Returns the ring pointer on success, ERR_PTR(error code) on failure + */ +struct atl_fwd_ring *atl_fwd_request_ring(struct net_device *ndev, + int flags, int ring_size, int buf_size, int page_order, + struct atl_fwd_mem_ops *mem_ops); + +/** + * atl_fwd_release_ring() - Free offload engine's ring + * + * @ring: ring to be freed + * + * Stops the ring, frees buffers if they were allocated, disables and + * releases ring's event if non-NULL, and frees the ring. + */ +void atl_fwd_release_ring(struct atl_fwd_ring *ring); + +/** + * atl_fwd_set_ring_intr_mod() - Set ring's interrupt moderation + * delays + * + * @ring: ring + * @min: min delay (0 - 511 uS) + * @max: max delay (0 - 1023 uS) + * + * Each ring has two configurable interrupt moderation timers. When an + * interrupt condition occurs (write-back of the final descriptor of a + * packet on receive, or writeback on a transmit descriptor with WB + * bit set), the min timer is restarted unconditionally and max timer + * is started only if it's not running yet. When any of the timers + * expires, the interrupt is signalled. + * + * Thus if a single interrupt event occurs it will be subjected to min + * delay. If subsequent events keep occuring with intervals less than + * min_delay between each other, the interrupt will be triggered + * max_delay after the initial event. + * + * Delays are internally represented in units of 2 microseconds, so + * the values supplied are rounded down to an even value. + * + * When called with negative @min or @max, the corresponding setting + * is left unchanged. + * + * Interrupt moderation is only supported for MSI-X vectors, not head + * pointer writeback events. + * + * Returns 0 on success or -EINVAL on attempt to set moderation delays + * for a ring with attached Tx WB event or when a requested delay is + * out of range. + */ +int atl_fwd_set_ring_intr_mod(struct atl_fwd_ring *ring, int min, int max); + +/** + * atl_fwd_enable_channel() - Enable offload engine's ring + * + * @ring: ring to be enabled + * + * Starts the ring. Returns 0 on success or negative error code. + */ +int atl_fwd_enable_ring(struct atl_fwd_ring *ring); +/** + * atl_fwd_disable_channel() - Disable offload engine's ring + * + * @ring: ring to be disabled + * + * Stops the ring. + */ +void atl_fwd_disable_ring(struct atl_fwd_ring *ring); + +/** + * atl_fwd_request_event() - Creates and attaches a ring notification + * event + * + * @evt: event structure + * + * Caller must allocate a struct atl_fwd_event and fill the @flags, + * @ring and either @tx_head_wrb or @msi_addr and @msi_data depending + * on the type bit in @flags. Event is created in disabled state. + * + * For an MSI event type, an MSI vector table slot is + * allocated and programmed, and it's index is saved in @evt->idx. + * + * @evt is then attached to the ring. + * + * Returns 0 on success or negative error code. + */ +int atl_fwd_request_event(struct atl_fwd_event *evt); + +/** + * atl_fwd_release_event() - Release a ring notification event + * + * @evt: event structure + * + * Disables the event if enabled, frees the MSI vector for an MSI-type + * event and detaches @evt from the ring. The @evt structure itself is + * not freed. + */ +void atl_fwd_release_event(struct atl_fwd_event *evt); + +/** + * atl_fwd_enable_event() - Enable a ring event + * + * @evt: event structure + * + * Enables the event. + * + * Returns 0 on success or negative error code. + */ +int atl_fwd_enable_event(struct atl_fwd_event *evt); + +/** + * atl_fwd_disable_event() - Disable a ring event + * + * @evt: event structure + * + * Disables the event. + * + * Returns 0 on success or negative error code. + */ +int atl_fwd_disable_event(struct atl_fwd_event *evt); + +int atl_fwd_receive_skb(struct net_device *ndev, struct sk_buff *skb); +int atl_fwd_transmit_skb(struct net_device *ndev, struct sk_buff *skb); + +/** + * atl_fwd_napi_receive_skb() - post skb to the network stack + * + * @ndev: network device + * @skb: buffer to post + * + * This function may only be called from softirq context and interrupts + * should be enabled. + */ +int atl_fwd_napi_receive_skb(struct net_device *ndev, struct sk_buff *skb); + +/** + * atl_fwd_register_notifier() - Register notifier for reset of device + * + * @ndev: network device + * @n: notifier block + * + * Register for notification on reset of device. The notifier callback + * receives a pointer to the affected device. Notification callback is + * expected to be synchronous. The receiver shall perform the expected actions + * upon the notification according to the notification type. + * + * Returns 0 on success or negative error code. + */ +int atl_fwd_register_notifier(struct net_device *ndev, + struct notifier_block *n); +int atl_fwd_unregister_notifier(struct net_device *ndev, + struct notifier_block *n); +enum atl_fwd_notify { + ATL_FWD_NOTIFY_RESET_PREPARE, /* receiver shall stop traffic and */ + /* disable rings */ + ATL_FWD_NOTIFY_RESET_COMPLETE, /* receiver shall refill descriptors and */ + /* enable rings */ + ATL_FWD_NOTIFY_MACSEC_ON, + ATL_FWD_NOTIFY_MACSEC_OFF, +}; + +enum atl_fwd_ring_state { + ATL_FWR_ST_ENABLED = BIT(0), + ATL_FWR_ST_EVT_ENABLED = BIT(1), +}; + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.c new file mode 100644 index 000000000000..3e2a98296319 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.c @@ -0,0 +1,1815 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "atl_compat.h" +#include "atl_fwdnl.h" + +#include "atl_common.h" +#include "atl_fwdnl_params.h" +#include "atl_ring.h" +#include "atl_trace.h" + +#define ATL_FWDNL_PREFIX "atl_fwdnl: " +/* Forward declaration, actual definition is at the bottom of the file */ +static struct genl_family atlfwd_nl_family; +static bool is_tx_ring_index(const int ring_index); +static bool is_rx_ring_index(const int ring_index); +static bool is_tx_ring(const struct atl_fwd_ring *ring); +static bool is_rx_ring(const struct atl_fwd_ring *ring); +static struct atl_fwd_ring *get_fwd_ring(struct net_device *netdev, + const int ring_index, + struct genl_info *info, + const bool err_if_released); +static int enable_fwd_queue(struct atl_nic *nic); +static int atlfwd_nl_transmit_skb_ring(struct atl_fwd_ring *ring, + struct sk_buff *skb); +static void atlfwd_nl_tx_head_poll(struct work_struct *); +static void atlfwd_nl_rx_poll(struct timer_list *); +static int release_ring(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring); +static int disable_ring(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring); +static int release_event(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring); + +/* workqueue data to perform tx cleanup */ +struct atlfwd_nl_tx_cleanup_workdata { + /* NB! delayed_work must be the first field in this structure */ + struct delayed_work dwork; + struct net_device *ndev; +}; + +/* data used to perform rx polling */ +struct atlfwd_nl_rx_poll_workdata { + struct atl_fwd_ring *ring; + struct timer_list timer; +}; + +/* Register generic netlink family upon module initialization */ +int __init atlfwd_nl_init(void) +{ + return genl_register_family(&atlfwd_nl_family); +} + +/* Populate private fwdnl data on probe */ +void atlfwd_nl_on_probe(struct net_device *ndev) +{ + struct atlfwd_nl_rx_poll_workdata *timer = NULL; + struct atlfwd_nl_tx_cleanup_workdata *wq = NULL; + struct atl_nic *nic = netdev_priv(ndev); + int i = 0; + + nic->fwdnl.force_icmp_via = S32_MIN; + nic->fwdnl.force_tx_via = S32_MIN; + + wq = devm_kzalloc(&ndev->dev, sizeof(*wq), GFP_KERNEL); + /* Memory allocation failure is not critical here. + * Yes, we won't be able to poll, but we still have other means + * of performing the cleanup. + */ + if (likely(wq)) { + INIT_DELAYED_WORK(&wq->dwork, atlfwd_nl_tx_head_poll); + wq->ndev = ndev; + nic->fwdnl.tx_cleanup_wq = &wq->dwork; + } + + for (i = 0; i != ARRAY_SIZE(nic->fwdnl.ring_desc); i++) { + nic->fwdnl.ring_desc[i].nic = nic; + + if (is_tx_ring_index(i)) + continue; + + timer = devm_kzalloc(&ndev->dev, sizeof(*timer), GFP_KERNEL); + if (likely(timer)) { + timer_setup(&timer->timer, atlfwd_nl_rx_poll, 0); + nic->fwdnl.ring_desc[i].rx_poll_timer = &timer->timer; + } else + pr_warn(ATL_FWDNL_PREFIX "RX timer creation failed!"); + } +} + +/* Cancel all pending work on remove request */ +void atlfwd_nl_on_remove(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_fwd_ring *ring = NULL; + int i = 0; + + if (nic->fwdnl.tx_cleanup_wq) + cancel_delayed_work_sync(nic->fwdnl.tx_cleanup_wq); + + for (i = 0; i != ATL_NUM_FWD_RINGS * 2; i++) { + ring = get_fwd_ring(ndev, i, NULL, false); + if (ring) + release_ring(ndev, NULL, ring); + } + + for (i = 0; i != ARRAY_SIZE(nic->fwdnl.ring_desc); i++) + nic->fwdnl.ring_desc[i].rx_poll_timer = NULL; +} + +/* Enables FWD egress queue, if necessary */ +int atlfwd_nl_on_open(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + + if (nic->fwdnl.force_icmp_via != S32_MIN || + nic->fwdnl.force_tx_via != S32_MIN) + return enable_fwd_queue(nic); + + return 0; +} + +/* Unregister generic netlink family upon module exit */ +void __exit atlfwd_nl_exit(void) +{ + genl_unregister_family(&atlfwd_nl_family); +} + +void atl_fwd_get_ring_stats(struct atl_fwd_ring *ring, + struct atl_ring_stats *stats) +{ + struct atl_desc_ring *desc = atlfwd_nl_get_fwd_ring_desc(ring); + + if (likely(desc != NULL)) { + unsigned int start; + + do { + start = u64_stats_fetch_begin_irq(&desc->syncp); + memcpy(stats, &desc->stats, sizeof(*stats)); + } while (u64_stats_fetch_retry_irq(&desc->syncp, start)); + } else { + memset(stats, 0, sizeof(*stats)); + } +} + +/* Returns true, if skb should be sent via FWD. false otherwise. */ +static bool atlfwd_nl_should_redirect(const struct sk_buff *skb, + struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + + if (unlikely(nic->fwdnl.force_tx_via != S32_MIN)) { + /* Redirect everything, but: + * . LSO is not supported at the moment + * . VLAN is not supported at the moment + */ + return (skb_shinfo(skb)->gso_size == 0 && + !skb_vlan_tag_present(skb)); + } + + if (unlikely(nic->fwdnl.force_icmp_via != S32_MIN)) { + uint8_t l4_proto = 0; + + switch (skb->protocol) { + case htons(ETH_P_IP): + l4_proto = ip_hdr(skb)->protocol; + break; + case htons(ETH_P_IPV6): + l4_proto = ipv6_hdr(skb)->nexthdr; + break; + } + + switch (l4_proto) { + case IPPROTO_ICMP: + return true; + } + } + + return false; +} + +netdev_tx_t atlfwd_nl_xmit(struct sk_buff *skb, struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + const bool err_if_released = false; + struct atl_fwd_ring *ring = NULL; + int ring_index = S32_MIN; + + if (nic->fwdnl.force_tx_via != S32_MIN) + ring_index = nic->fwdnl.force_tx_via; + else if (nic->fwdnl.force_icmp_via != S32_MIN) + ring_index = nic->fwdnl.force_icmp_via; + else + return -EFAULT; + + ring = get_fwd_ring(ndev, ring_index, NULL, err_if_released); + if (unlikely(ring == NULL)) + return -EFAULT; + + return atlfwd_nl_transmit_skb_ring(ring, skb); +} + +u16 atlfwd_nl_select_queue_fallback(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev, + select_queue_fallback_t fallback) +{ + static atomic_t fwd_idx_collisions = ATOMIC_INIT(0); + struct atl_nic *nic = netdev_priv(dev); +#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) && \ + RHEL_RELEASE_CODE < RHEL_RELEASE_VERSION(8, 0) + u16 idx = fallback(dev, skb); +#else + u16 idx = fallback(dev, skb, sb_dev); +#endif + + if (likely(dev->real_num_tx_queues == nic->nvecs)) + /* no FWD queues enabled, only standard ones */ + return idx; + + if (atlfwd_nl_should_redirect(skb, dev)) + /* FWD queue enabled, skb meets redirection criteria */ + return nic->nvecs; + + /* FWD queue enabled, but skb doesn't meet redirection criteria. + * Send via standard ring. + */ + if (idx >= nic->nvecs) { + atomic_inc_return(&fwd_idx_collisions); + idx = 0; + } + + return idx; +} + +/* Returns true, if a given TX FWD ring is created/requested. + * Ring index argument is 0-based, ATL_FWD_RING_BASE is added automatically. + */ +bool atlfwd_nl_is_tx_fwd_ring_created(struct net_device *ndev, + const int fwd_ring_index) +{ + const int ring_index = ATL_FWD_RING_BASE + fwd_ring_index; + struct atl_nic *nic = netdev_priv(ndev); + + return test_bit(ring_index, &nic->fwd.ring_map[ATL_FWDIR_TX]); +} + +/* Returns true, if a given RX FWD ring is created/requested. + * Ring index argument is 0-based, ATL_FWD_RING_BASE is added automatically. + */ +bool atlfwd_nl_is_rx_fwd_ring_created(struct net_device *ndev, + const int fwd_ring_index) +{ + const int ring_index = ATL_FWD_RING_BASE + fwd_ring_index; + struct atl_nic *nic = netdev_priv(ndev); + + return test_bit(ring_index, &nic->fwd.ring_map[ATL_FWDIR_RX]); +} + +struct atl_fwd_ring *atlfwd_nl_get_fwd_ring(struct net_device *ndev, + const int ring_index) +{ + return get_fwd_ring(ndev, ring_index, NULL, false); +} + +/* Set the netlink error message + * + * Before Linux 4.12 we could only put it in kernel logs. + * Starting with 4.12 we can also use extack to pass the message to user-mode. + */ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 12, 0) +#define ATLFWD_NL_SET_ERR_MSG(info, msg) \ + do { \ + if (info) \ + GENL_SET_ERR_MSG(info, msg); \ + else \ + pr_warn(ATL_FWDNL_PREFIX "%s\n", msg); \ + } while (0) +#else +#define ATLFWD_NL_SET_ERR_MSG(info, msg) pr_warn(ATL_FWDNL_PREFIX "%s\n", msg) +#endif + +/* Returns true, if the given net_device was allocated by FWD driver. + */ +bool is_atlfwd_device(const struct net_device *dev) +{ + static size_t atl_len; + + if (unlikely(atl_len == 0)) + atl_len = strlen(atl_driver_name); + + if (likely(dev && dev->dev.parent)) { + const char *driver_name = dev_driver_string(dev->dev.parent); + const size_t len = min_t(size_t, atl_len, strlen(driver_name)); + + return (len == atl_len) && + !strncmp(driver_name, atl_driver_name, len); + } + + return false; +} + +/* Attempt to auto-deduce the device + * + * This is possible if and only if there's exactly 1 ATL device in the system. + * + * Returns NULL on error, net_device pointer otherwise. + */ +static struct net_device *atlfwd_nl_auto_deduce_dev(struct genl_info *info) +{ + struct net *net = genl_info_net(info); + struct net_device *ndev = NULL; + bool found = false; + + rcu_read_lock(); + for_each_netdev_rcu (net, ndev) { + if (is_atlfwd_device(ndev)) { + dev_hold(ndev); + found = true; + break; + } + } + rcu_read_unlock(); + + if (!found) + ATLFWD_NL_SET_ERR_MSG( + info, "No ATL devices found or a wrong driver is used"); + + return found ? ndev : NULL; +} + +/* Get net_device by name + * + * The name is usually provided by the user-mode tool via the IFNAME attribute. + * But it can also be null, if the tool hasn't provided the attribute and wants + * us to attempt to auto deduce the device. + * + * Returns NULL on error, net_device pointer otherwise. + */ +static struct net_device *atlfwd_nl_get_dev_by_name(const char *dev_name, + struct genl_info *info) +{ + struct net_device *netdev = NULL; + + if (dev_name == NULL) { + /* No dev_name provided in request, try to auto-deduce. */ + return atlfwd_nl_auto_deduce_dev(info); + } + + netdev = dev_get_by_name(genl_info_net(info), dev_name); + if (unlikely(netdev == NULL)) { + ATLFWD_NL_SET_ERR_MSG(info, "No matching device found"); + return NULL; + } + + if (unlikely(!is_atlfwd_device(netdev))) { + ATLFWD_NL_SET_ERR_MSG( + info, + "Requested device is not an ATL device or a wrong driver is used"); + goto err_devput; + } + + return netdev; + +err_devput: + dev_put(netdev); + return NULL; +} + +/* Helper function to obtain the netlink attribute string data + * + * Returns NULL, if the attribute doesn't exist. + */ +static const char * +atlfwd_attr_to_str_or_null(struct genl_info *info, + const enum atlfwd_nl_attribute attr) +{ + if (likely(!info->attrs[attr])) + return NULL; + + return (char *)nla_data(info->attrs[attr]); +} + +/* Helper function to obtain the netlink attribute s32 data + * + * Returns S32_MIN, if attribute is missing. Actual attribute value otherwise. + */ +static int atlfwd_attr_to_s32_priv(struct genl_info *info, + const enum atlfwd_nl_attribute attr, + const bool optional) +{ + if (info->attrs[attr]) + return nla_get_s32(info->attrs[attr]); + + if (!optional) { + pr_warn(ATL_FWDNL_PREFIX + "attribute %d check is missing in pre_doit\n", + attr); + ATLFWD_NL_SET_ERR_MSG( + info, + "Required attribute is missing (and internal error)"); + } + return S32_MIN; +} + +static int atlfwd_attr_to_s32(struct genl_info *info, + const enum atlfwd_nl_attribute attr) +{ + return atlfwd_attr_to_s32_priv(info, attr, false); +} + +/* Similar to previous function, but doesn't produce warnings, + * because attribute is optional. + */ +static int atlfwd_attr_to_s32_optional(struct genl_info *info, + const enum atlfwd_nl_attribute attr) +{ + return atlfwd_attr_to_s32_priv(info, attr, true); +} + +static bool is_tx_ring_index(const int ring_index) +{ + return ring_index % 2; +} + +static bool is_rx_ring_index(const int ring_index) +{ + return !(is_tx_ring_index(ring_index)); +} + +static bool is_tx_ring(const struct atl_fwd_ring *ring) +{ + return !!(ring->flags & ATL_FWR_TX); +} + +static bool is_rx_ring(const struct atl_fwd_ring *ring) +{ + return !(ring->flags & ATL_FWR_TX); +} + +/* Converts regular ring index to "normalized" internal representation. + * This "normalized" value is the only thing known to user-mode. + */ +static int nl_ring_index(const struct atl_fwd_ring *ring) +{ + int idx = 0; + + if (ring->idx < ATL_FWD_RING_BASE) { + pr_warn(ATL_FWDNL_PREFIX + "Got an unexpected ring index %d (was expecting %d or greater)\n", + ring->idx, ATL_FWD_RING_BASE); + return S32_MIN; + } + + /* Expose a 0-based index to the user-mode */ + idx = ring->idx - ATL_FWD_RING_BASE; + + /* Use even (0,2,...) indexes for RX rings, odd - for TX. */ + idx *= 2; + if (is_tx_ring(ring)) + idx++; + pr_debug(ATL_FWDNL_PREFIX "Normalized ring index %d\n", idx); + + return idx; +} + +/* Checks if a given ring index (obtained from user-mode => "normalized") + * is valid. + */ +static bool is_valid_ring_index(const int ring_index) +{ + return (ring_index >= 0 && ring_index < ATL_NUM_FWD_RINGS * 2); +} + +static struct sk_buff *nl_reply_create(void) +{ + return nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); +} + +static void *nl_reply_init(struct sk_buff *msg, struct genl_info *info) +{ + void *hdr = NULL; + + if (unlikely(msg == NULL)) + return NULL; + + hdr = genlmsg_put(msg, info->snd_portid, info->snd_seq, + &atlfwd_nl_family, 0, info->genlhdr->cmd); + if (hdr == NULL) { + ATLFWD_NL_SET_ERR_MSG(info, "Reply message creation failed"); + nlmsg_free(msg); + } + + return hdr; +} + +static bool nl_reply_add_attr(struct sk_buff *msg, void *hdr, + struct genl_info *info, + const enum atlfwd_nl_attribute attr, + const int value) +{ + if (unlikely(msg == NULL)) + return false; + + if (nla_put_s32(msg, attr, value) != 0) { + ATLFWD_NL_SET_ERR_MSG(info, + "Failed to put the reply attribute"); + genlmsg_cancel(msg, hdr); + nlmsg_free(msg); + return false; + } + + return true; +} + +static int nl_reply_send(struct sk_buff *msg, void *hdr, struct genl_info *info) +{ + genlmsg_end(msg, hdr); + return genlmsg_reply(msg, info); +} + +static int atlfwd_nl_send_reply(struct genl_info *info, + const enum atlfwd_nl_attribute attr, + const int value) +{ + struct sk_buff *msg = nl_reply_create(); + void *hdr = nl_reply_init(msg, info); + + if (unlikely(msg == NULL)) + return -ENOBUFS; + if (unlikely(hdr == NULL)) + return -EMSGSIZE; + + if (unlikely(!nl_reply_add_attr(msg, hdr, info, attr, value))) + return -EMSGSIZE; + + return nl_reply_send(msg, hdr, info); +} + +/* Get the FWD ring descriptor for a given FWD ring. + * + * Returns NULL on error. + */ +struct atl_desc_ring *atlfwd_nl_get_fwd_ring_desc(struct atl_fwd_ring *ring) +{ + int ring_index = S32_MIN; + + if (unlikely(ring == NULL)) + return NULL; + + ring_index = nl_ring_index(ring); + if (unlikely(ring_index == S32_MIN)) + return NULL; + + return &ring->nic->fwdnl.ring_desc[ring_index]; +} + +/* Get the FWD ring by index. + * + * This function obtains the FWD ring descriptor/pointer from the private + * part of net_device data. + * + * Returns NULL on error. + */ +static struct atl_fwd_ring *get_fwd_ring(struct net_device *netdev, + const int ring_index, + struct genl_info *info, + const bool err_if_released) +{ + const int dir_tx = is_tx_ring_index(ring_index); + const int norm_index = (dir_tx ? ring_index - 1 : ring_index) / 2; + struct atl_nic *nic = netdev_priv(netdev); + struct atl_fwd_ring *ring = nic->fwd.rings[dir_tx][norm_index]; + + pr_debug(ATL_FWDNL_PREFIX "index=%d/n%d\n", ring_index, norm_index); + + if (unlikely(norm_index < 0 || norm_index >= ATL_NUM_FWD_RINGS)) { + ATLFWD_NL_SET_ERR_MSG(info, "Ring index is out of bounds"); + return NULL; + } + + if (unlikely(ring == NULL && err_if_released)) + ATLFWD_NL_SET_ERR_MSG(info, + "Requested ring is NULL / released"); + + return ring; +} + +static uint32_t atlfwd_nl_ring_hw_head(struct atl_fwd_ring *ring) +{ + return atl_read(&ring->nic->hw, ATL_RING_HEAD(&ring->hw)) & 0x1fff; +} + +static uint32_t atlfwd_nl_ring_hw_tail(struct atl_fwd_ring *ring) +{ + return atl_read(&ring->nic->hw, ATL_RING_TAIL(&ring->hw)) & 0x1fff; +} + +static int atlfwd_nl_ring_occupied(struct atl_fwd_ring *ring, u32 sw_tail) +{ + int busy = sw_tail - atlfwd_nl_ring_hw_head(ring); + + if (busy < 0) + busy += ring->hw.size; + + return busy; +} + +/* Returns true if the space (number of free elements) is sufficient to store + * 'needed' amount of elements. + */ +static bool atlfwd_nl_tx_full(struct atl_desc_ring *ring, + const unsigned int needed) +{ + int space = ring_space(ring); + + pr_debug(ATL_FWDNL_PREFIX "Needed %u, actual %d\n", needed, space); + if (likely(space >= needed)) + return false; + + return true; +} + +static bool atlfwd_nl_queue_stopped(struct atl_nic *nic, + struct atl_desc_ring *ring_desc) +{ + return __netif_subqueue_stopped(nic->ndev, nic->nvecs); +} + +static void atlfwd_nl_stop_queue(struct atl_nic *nic, + struct atl_desc_ring *ring_desc) +{ + pr_debug(ATL_FWDNL_PREFIX "Stopping TX queue %d\n", nic->nvecs); + netif_stop_subqueue(nic->ndev, nic->nvecs); +} + +static void atlfwd_nl_restart_queue(struct atl_nic *nic, + struct atl_desc_ring *ring_desc) +{ + pr_debug(ATL_FWDNL_PREFIX "Restarting TX queue %d\n", nic->nvecs); + netif_start_subqueue(nic->ndev, nic->nvecs); + atl_update_ring_stat(ring_desc, tx.tx_restart, 1); +} + +static int enable_fwd_queue(struct atl_nic *nic) +{ + if (unlikely(!test_bit(ATL_ST_UP, &nic->hw.state))) + return 0; + + if (nic->ndev->real_num_tx_queues == nic->nvecs) + return netif_set_real_num_tx_queues(nic->ndev, nic->nvecs + 1); + + return 0; +} + +static void disable_fwd_queue(struct atl_nic *nic) +{ + if (unlikely(!test_bit(ATL_ST_UP, &nic->hw.state))) + return; + + if (nic->ndev->real_num_tx_queues != nic->nvecs) + netif_set_real_num_tx_queues(nic->ndev, nic->nvecs); +} + +/* Returns true if the space is sufficient after TX queue stop, + * e.g. if another CPU has freed some space. + */ +static bool atlfwd_nl_tx_full_after_stop(struct atl_nic *nic, + struct atl_desc_ring *ring, + const unsigned int needed) +{ + atlfwd_nl_stop_queue(nic, ring); + + smp_mb(); + + /* Check if another CPU has freed some space */ + return atlfwd_nl_tx_full(ring, needed); +} + +/* Returns true, if the number of occupied elements is above a given threshold. + * + * E.g. if threshold is 4, then we'll return true here, if number of occupied + * elements is greater than 1/4 of the total ring size. + */ +static bool atlfwd_nl_tx_above_threshold(struct atl_desc_ring *ring, + const unsigned int threshold_frac) +{ + if (unlikely(threshold_frac == 0)) + return false; + + return ring->hw.size - ring_space(ring) > + ring->hw.size / threshold_frac; +} + +/* Returns checksum offload flags for TX descriptor */ +static unsigned int +atlfwd_nl_skb_checksum_offload_cmd(const struct sk_buff *skb) +{ + unsigned int cmd = 0; + + if (skb->ip_summed == CHECKSUM_PARTIAL) { + /* Checksum offload has been requested by the stack */ + uint8_t l4_proto = 0; + + switch (skb->protocol) { + case htons(ETH_P_IP): + cmd |= tx_desc_cmd_ipv4cs; + l4_proto = ip_hdr(skb)->protocol; + break; + case htons(ETH_P_IPV6): + l4_proto = ipv6_hdr(skb)->nexthdr; + break; + } + + switch (l4_proto) { + case IPPROTO_TCP: + case IPPROTO_UDP: + cmd |= tx_desc_cmd_l4cs; + break; + } + } + + return cmd; +} + +static unsigned int atlfwd_nl_num_txd_for_skb(struct sk_buff *skb) +{ + unsigned int len = skb_headlen(skb); + unsigned int num_txd = + (len - len % ATL_DATA_PER_TXD) / ATL_DATA_PER_TXD + 1; + unsigned int frag; + + for (frag = 0; frag != skb_shinfo(skb)->nr_frags; frag++) { + len = skb_frag_size(&skb_shinfo(skb)->frags[frag]); + num_txd += + (len - len % ATL_DATA_PER_TXD) / ATL_DATA_PER_TXD + 1; + } + + return num_txd; +} + +/* Returns true, if HW has completed processing (head is equal to tail). + * Returns false, if there some work is still pending. + */ +static bool atlfwd_nl_tx_head_poll_ring(struct atl_fwd_ring *ring) +{ + struct atl_desc_ring *desc = atlfwd_nl_get_fwd_ring_desc(ring); + uint32_t budget = atl_tx_clean_budget; + unsigned int free_high = 0; + struct device *dev = NULL; + unsigned int packets = 0; + unsigned int bytes = 0; + uint32_t hw_head = 0; + uint32_t sw_head = 0; + + if (unlikely(desc == NULL)) + return true; + + dev = &ring->nic->hw.pdev->dev; + hw_head = atlfwd_nl_ring_hw_head(ring); + sw_head = READ_ONCE(desc->head); + pr_debug(ATL_FWDNL_PREFIX "hw_head=%d, sw_head=%d\n", hw_head, sw_head); + + do { + struct atl_txbuf *txbuf = &desc->txbufs[sw_head]; + + if (sw_head == hw_head) + break; + + bytes += txbuf->bytes; + packets += txbuf->packets; + + if (dma_unmap_len(txbuf, len)) { + dma_unmap_single(dev, dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), + DMA_TO_DEVICE); + dma_unmap_len_set(txbuf, len, 0); + } + + if (txbuf->skb) { + napi_consume_skb(txbuf->skb, budget); + txbuf->skb = NULL; + } + + bump_ptr(sw_head, desc, 1); + } while (budget--); + + desc->head = sw_head; + pr_debug(ATL_FWDNL_PREFIX "bytes=%u, packets=%u, sw_head=%d\n", bytes, + packets, sw_head); + + u64_stats_update_begin(&desc->syncp); + desc->stats.tx.bytes += bytes; + desc->stats.tx.packets += packets; + u64_stats_update_end(&desc->syncp); + + free_high = min_t(typeof(atl_tx_free_high), atl_tx_free_high, + ring->hw.size - 1); + if (unlikely(atlfwd_nl_queue_stopped(ring->nic, desc)) && + likely(test_bit(ATL_ST_RINGS_RUNNING, &ring->nic->hw.state)) && + likely(!atlfwd_nl_tx_full(desc, free_high))) { + atlfwd_nl_restart_queue(ring->nic, desc); + } + + return (hw_head == READ_ONCE(desc->tail)); +} + +static void atlfwd_nl_tx_head_poll(struct work_struct *work) +{ + struct atlfwd_nl_tx_cleanup_workdata *data = + (struct atlfwd_nl_tx_cleanup_workdata *)work; + struct net_device *ndev = data->ndev; + struct atl_nic *nic = netdev_priv(ndev); + const bool err_if_released = false; + struct atl_fwd_ring *ring = NULL; + bool poll_finished = true; + int ring_index = S32_MIN; + + ring_index = nic->fwdnl.force_tx_via; + if (ring_index != S32_MIN) { + ring = get_fwd_ring(ndev, ring_index, NULL, err_if_released); + if (!atlfwd_nl_tx_head_poll_ring(ring)) + poll_finished = false; + } + + if (ring_index != nic->fwdnl.force_icmp_via) { + ring_index = nic->fwdnl.force_icmp_via; + if (ring_index != S32_MIN) { + ring = get_fwd_ring(ndev, ring_index, NULL, + err_if_released); + if (!atlfwd_nl_tx_head_poll_ring(ring)) + poll_finished = false; + } + } + + if (!poll_finished && likely(atlfwd_nl_tx_clean_threshold_msec != 0)) + schedule_delayed_work( + nic->fwdnl.tx_cleanup_wq, + msecs_to_jiffies(atlfwd_nl_tx_clean_threshold_msec)); +} + +static int atlfwd_nl_receive_skb(struct atl_desc_ring *ring, + struct sk_buff *skb) +{ + struct net_device *ndev = ring->nic->ndev; + + return atl_fwd_receive_skb(ndev, skb); +} + +static void atlfwd_nl_rx_poll(struct timer_list *timer) +{ + struct atlfwd_nl_rx_poll_workdata *data = + from_timer(data, timer, timer); + struct atl_desc_ring *desc = atlfwd_nl_get_fwd_ring_desc(data->ring); + int budget = atlfwd_nl_rx_clean_budget; + + atl_clean_rx(desc, budget, atlfwd_nl_receive_skb); + + if (likely(altfwd_nl_rx_poll_interval_msec != 0)) { + timer->expires = + jiffies + + msecs_to_jiffies(altfwd_nl_rx_poll_interval_msec); + add_timer(timer); + } +} + +static void atl_fwd_txbuf_free(struct device *dev, struct atl_txbuf *txbuf) +{ + if (dma_unmap_len(txbuf, len)) { + dma_unmap_single(dev, dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), DMA_TO_DEVICE); + } + + memset(txbuf, 0, sizeof(*txbuf)); +} + +static int atlfwd_nl_transmit_skb_ring(struct atl_fwd_ring *ring, + struct sk_buff *skb) +{ + const unsigned int num_txd = atlfwd_nl_num_txd_for_skb(skb); + unsigned int frags = skb_shinfo(skb)->nr_frags; + struct atl_desc_ring *ring_desc = NULL; + unsigned int frag_len = skb_headlen(skb); + struct atl_txbuf *first_buf = NULL; + struct atl_nic *nic = ring->nic; + u32 bunch = nic->fwdnl.tx_bunch; + struct atl_txbuf *txbuf = NULL; + struct atl_hw *hw = &nic->hw; + struct device *dev = &hw->pdev->dev; + unsigned int free_low = 0; + dma_addr_t frag_daddr = 0; + skb_frag_t *frag = NULL; + struct atl_tx_desc desc; + uint32_t desc_idx; + + ring_desc = atlfwd_nl_get_fwd_ring_desc(ring); + if (unlikely(ring_desc == NULL)) + return -EFAULT; + + desc_idx = ring_desc->tail; + + if (atlfwd_nl_tx_above_threshold(ring_desc, + atlfwd_nl_tx_clean_threshold_frac)) { + cancel_delayed_work_sync(nic->fwdnl.tx_cleanup_wq); + atlfwd_nl_tx_head_poll(&nic->fwdnl.tx_cleanup_wq->work); + } + + if (unlikely(atlfwd_nl_tx_full(ring_desc, num_txd))) { + if (atlfwd_nl_tx_full_after_stop(nic, ring_desc, num_txd)) { + atl_update_ring_stat(ring_desc, tx.tx_busy, 1); + return NETDEV_TX_BUSY; + } + + atlfwd_nl_restart_queue(nic, ring_desc); + } + + memset(&desc, 0, sizeof(desc)); + desc.cmd = tx_desc_cmd_fcs; + desc.cmd |= atlfwd_nl_skb_checksum_offload_cmd(skb); + desc.ct_en = 0; + desc.type = tx_desc_type_desc; + desc.pay_len = skb->len; + + first_buf = &ring_desc->txbufs[desc_idx]; + txbuf = first_buf; + memset(txbuf, 0, sizeof(*txbuf)); + + frag_daddr = dma_map_single(dev, skb->data, frag_len, DMA_TO_DEVICE); + for (frag = &skb_shinfo(skb)->frags[0];; frag++) { + dma_addr_t daddr = frag_daddr; + unsigned int len = frag_len; + + if (dma_mapping_error(dev, daddr)) + goto err_dma; + + desc.daddr = cpu_to_le64(daddr); + while (len > ATL_DATA_PER_TXD) { + desc.len = cpu_to_le16(ATL_DATA_PER_TXD); + txbuf->bytes = ATL_DATA_PER_TXD; + ring->hw.descs[desc_idx].tx = desc; + bump_ptr(desc_idx, ring_desc, 1); + txbuf = &ring_desc->txbufs[desc_idx]; + memset(txbuf, 0, sizeof(*txbuf)); + daddr += ATL_DATA_PER_TXD; + len -= ATL_DATA_PER_TXD; + desc.daddr = cpu_to_le64(daddr); + } + desc.len = cpu_to_le16(len); + txbuf->bytes = len; /* populate bytes for each descriptor */ + /* populate DMA addr/len for last descriptor of the fragment + * (it's safe to unmap once HW processes this descriptor) + */ + dma_unmap_len_set(txbuf, len, frag_len); + dma_unmap_addr_set(txbuf, daddr, frag_daddr); + + if (!frags) + break; + + ring->hw.descs[desc_idx].tx = desc; + bump_ptr(desc_idx, ring_desc, 1); + txbuf = &ring_desc->txbufs[desc_idx]; + memset(txbuf, 0, sizeof(*txbuf)); + frag_len = skb_frag_size(frag); + frag_daddr = + skb_frag_dma_map(dev, frag, 0, frag_len, DMA_TO_DEVICE); + + frags--; + } + + /* Last descriptor */ + desc.eop = 1; +#if defined(ATL_TX_DESC_WB) || defined(ATL_TX_HEAD_WB) + desc.cmd |= tx_desc_cmd_wb; +#endif + /* populate skb for each packet + * (it's safe to consume skb once HW processes this descriptor) + */ + txbuf->packets = 1; + txbuf->skb = skb; + ring->hw.descs[desc_idx].tx = desc; + bump_ptr(desc_idx, ring_desc, 1); + ring_desc->tail = desc_idx; + + if (likely(atlfwd_nl_tx_clean_threshold_msec != 0)) + schedule_delayed_work( + nic->fwdnl.tx_cleanup_wq, + msecs_to_jiffies(atlfwd_nl_tx_clean_threshold_msec)); + + /* Stop the queue, if there is no space for another packet */ + free_low = min_t(typeof(atl_tx_free_low), atl_tx_free_low, + ring->hw.size / 2); + if (unlikely(atlfwd_nl_tx_full(ring_desc, free_low))) { + if (atlfwd_nl_speculative_queue_stop) + atlfwd_nl_stop_queue(nic, ring_desc); + } else if (skb_xmit_more(skb)) { + /* Delay bumping the HW tail if another packet is pending */ + return NETDEV_TX_OK; + } + + wmb(); + + if (atlfwd_nl_ring_occupied(ring, desc_idx) > bunch) + atl_write(hw, ATL_TX_RING_TAIL(ring), desc_idx); + + return NETDEV_TX_OK; + +err_dma: + dev_err(dev, "%s failed\n", __func__); + for (;;) { + atl_fwd_txbuf_free(dev, txbuf); + if (txbuf == first_buf) + break; + bump_ptr(desc_idx, ring_desc, -1); + txbuf = &ring_desc->txbufs[desc_idx]; + } + atl_update_ring_stat(ring_desc, tx.dma_map_failed, 1); + return -EFAULT; +} + +static struct net_device * +get_ndev_or_null(struct genl_info *info, + const enum atlfwd_nl_attribute ifname_attr) +{ + const char *ifname = atlfwd_attr_to_str_or_null(info, ifname_attr); + + return atlfwd_nl_get_dev_by_name(ifname, info); +} + +typedef int (*no_attr_handler)(struct net_device *dev, struct genl_info *info); +static int cmd_with_no_attr(struct sk_buff *skb, struct genl_info *info, + no_attr_handler handler) +{ + struct net_device *ndev = get_ndev_or_null(info, ATL_FWD_ATTR_IFNAME); + int result = 0; + + if (ndev == NULL) + return -ENODEV; + + result = handler(ndev, info); + + dev_put(ndev); + return result; +} + +#define MANDATORY_ATTR true +#define OPTIONAL_ATTR false +typedef int (*s32_attr_handler)(struct net_device *dev, struct genl_info *info, + const int value); +static int cmd_with_s32_attr(struct sk_buff *skb, struct genl_info *info, + const enum atlfwd_nl_attribute s32_attr, + const bool mandatory, s32_attr_handler handler) +{ + struct net_device *ndev = get_ndev_or_null(info, ATL_FWD_ATTR_IFNAME); + int value = S32_MIN; + int result = 0; + + if (ndev == NULL) + return -ENODEV; + + if (mandatory) { + value = atlfwd_attr_to_s32(info, s32_attr); + if (unlikely(value == S32_MIN)) { + result = -EINVAL; + goto err_netdev; + } + } else { + value = atlfwd_attr_to_s32_optional(info, s32_attr); + } + + result = handler(ndev, info, value); + +err_netdev: + dev_put(ndev); + return result; +} + +typedef int (*ring_attr_handler)(struct net_device *dev, struct genl_info *info, + struct atl_fwd_ring *ring); +static int cmd_with_ring_index_attr(struct sk_buff *skb, struct genl_info *info, + ring_attr_handler handler) +{ + struct net_device *ndev = get_ndev_or_null(info, ATL_FWD_ATTR_IFNAME); + const bool err_if_released = true; + struct atl_fwd_ring *ring = NULL; + int ring_index = S32_MIN; + int result = 0; + + if (ndev == NULL) + return -ENODEV; + + ring_index = atlfwd_attr_to_s32(info, ATL_FWD_ATTR_RING_INDEX); + if (unlikely(ring_index == S32_MIN)) { + result = -EINVAL; + goto err_netdev; + } + + ring = get_fwd_ring(ndev, ring_index, info, err_if_released); + if (unlikely(ring == NULL)) { + result = -EINVAL; + goto err_netdev; + } + + result = handler(ndev, info, ring); + +err_netdev: + dev_put(ndev); + return result; +} + +/* ATL_FWD_CMD_REQUEST_RING handler */ +static int request_ring(struct net_device *ndev, struct genl_info *info) +{ + const int page_order = + atlfwd_attr_to_s32(info, ATL_FWD_ATTR_PAGE_ORDER); + const int ring_size = atlfwd_attr_to_s32(info, ATL_FWD_ATTR_RING_SIZE); + const int buf_size = atlfwd_attr_to_s32(info, ATL_FWD_ATTR_BUF_SIZE); + const int flags = atlfwd_attr_to_s32(info, ATL_FWD_ATTR_FLAGS); + struct atl_desc_ring *ring_desc = NULL; + struct atl_fwd_ring *ring = NULL; + int ring_index = S32_MIN; + int result = 0; + + if (unlikely(flags == S32_MIN || ring_size == S32_MIN || + buf_size == S32_MIN || page_order == S32_MIN)) + return -EINVAL; + + pr_debug(ATL_FWDNL_PREFIX "flags=%d\n", flags); + pr_debug(ATL_FWDNL_PREFIX "ring_size=%d\n", ring_size); + pr_debug(ATL_FWDNL_PREFIX "buf_size=%d\n", buf_size); + pr_debug(ATL_FWDNL_PREFIX "page_order=%d\n", page_order); + + ring = atl_fwd_request_ring(ndev, flags, ring_size, buf_size, + page_order, NULL); + if (IS_ERR_OR_NULL(ring)) + return PTR_ERR(ring); + + pr_debug(ATL_FWDNL_PREFIX "Got ring %d (%p)\n", ring->idx, ring); + + ring_desc = atlfwd_nl_get_fwd_ring_desc(ring); + ring_index = nl_ring_index(ring); + if (unlikely(ring_desc == NULL || ring_index == S32_MIN)) { + ATLFWD_NL_SET_ERR_MSG(info, "Internal error"); + result = -EFAULT; + goto err_relring; + } + + memcpy(&ring_desc->hw, &ring->hw, sizeof(ring_desc->hw)); + + if (is_tx_ring(ring)) { + ring_desc->txbufs = kcalloc( + ring_size, sizeof(*ring_desc->txbufs), GFP_KERNEL); + if (unlikely(ring_desc->txbufs == NULL)) { + result = -ENOMEM; + goto err_relring; + } + + result = atl_init_tx_ring(ring_desc); + } else { + ring_desc->rxbufs = kcalloc( + ring_size, sizeof(*ring_desc->rxbufs), GFP_KERNEL); + if (unlikely(ring_desc->rxbufs == NULL)) { + result = -ENOMEM; + goto err_relring; + } + + result = atl_init_rx_ring(ring_desc); + } + if (result) + goto err_relring; + + u64_stats_init(&ring_desc->syncp); + memset(&ring_desc->stats, 0, sizeof(ring_desc->stats)); + + return atlfwd_nl_send_reply(info, ATL_FWD_ATTR_RING_INDEX, ring_index); + +err_relring: + atl_fwd_release_ring(ring); + return result; +} + +static int doit_request_ring(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_no_attr(skb, info, request_ring); +} + +/* ATL_FWD_CMD_RELEASE_RING handler */ +static int release_ring(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct atl_desc_ring *ring_desc = atlfwd_nl_get_fwd_ring_desc(ring); + + if (unlikely(ring_desc == NULL)) { + ATLFWD_NL_SET_ERR_MSG(info, "Internal error"); + return -EFAULT; + } + + disable_ring(ndev, info, ring); + release_event(ndev, info, ring); + + memset(&ring_desc->hw, 0, sizeof(ring_desc->hw)); + if (is_tx_ring(ring)) { + kfree(ring_desc->txbufs); + ring_desc->txbufs = NULL; + } else { + kfree(ring_desc->rxbufs); + ring_desc->rxbufs = NULL; + } + + pr_debug(ATL_FWDNL_PREFIX "Releasing ring %d (%p)\n", + nl_ring_index(ring), ring); + atl_fwd_release_ring(ring); + + return 0; +} + +static int doit_release_ring(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, release_ring); +} + +/* ATL_FWD_CMD_ENABLE_RING handler */ +static int enable_ring(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct atl_desc_ring *desc = NULL; + + pr_debug(ATL_FWDNL_PREFIX "Enabling ring %d (%p)\n", + nl_ring_index(ring), ring); + + if (is_rx_ring(ring)) { + struct atlfwd_nl_rx_poll_workdata *data = NULL; + + desc = atlfwd_nl_get_fwd_ring_desc(ring); + if (unlikely(desc == NULL)) { + ATLFWD_NL_SET_ERR_MSG(info, "Internal error"); + return -EFAULT; + } + + if (unlikely(desc->rx_poll_timer == NULL)) { + ATLFWD_NL_SET_ERR_MSG( + info, + "RX polling timer doesn't exist. The ring might not function properly."); + goto err_enablering; + } + + data = from_timer(data, desc->rx_poll_timer, timer); + data->ring = ring; + if (likely(altfwd_nl_rx_poll_interval_msec != 0)) { + desc->rx_poll_timer->expires = + jiffies + + msecs_to_jiffies( + altfwd_nl_rx_poll_interval_msec); + add_timer(desc->rx_poll_timer); + } + } + +err_enablering: + return atl_fwd_enable_ring(ring); +} + +static int doit_enable_ring(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, enable_ring); +} + +/* ATL_FWD_CMD_DISABLE_RING handler */ +static int disable_ring(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct atl_desc_ring *desc = NULL; + + pr_debug(ATL_FWDNL_PREFIX "Disabling ring %d (%p)\n", + nl_ring_index(ring), ring); + atl_fwd_disable_ring(ring); + + if (is_tx_ring(ring)) + return 0; + + // RX ring + desc = atlfwd_nl_get_fwd_ring_desc(ring); + if (unlikely(desc == NULL || desc->rx_poll_timer == NULL)) + return 0; + + del_timer_sync(desc->rx_poll_timer); + atl_clear_rx_bufs(desc); + + return 0; +} + +static int doit_disable_ring(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, disable_ring); +} + +/* ATL_FWD_CMD_DUMP_RING handler */ +static int dump_ring(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + const uint32_t head = atlfwd_nl_ring_hw_head(ring); + const uint32_t tail = atlfwd_nl_ring_hw_tail(ring); + bool dir_tx = is_tx_ring(ring); + int j; + + pr_debug(ATL_FWDNL_PREFIX "Dumping ring %d (%p)\n", nl_ring_index(ring), + ring); + + pr_info("[%d] head=%d tail=%d", ring->idx, head, tail); + + for (j = 0; j != ring->hw.size; j++) { + if (dir_tx) { + pr_info("[%d] [%d] 0x%llx, DD=%d", ring->idx, j, + ring->hw.descs[j].tx.daddr, + ring->hw.descs[j].tx.dd); + trace_atl_tx_descr(ring->idx, j, + (u64 *)ring->hw.descs[j].raw); + } else { + pr_info("[%d] [%d] 0x%llx, DD=%d", ring->idx, j, + ring->hw.descs[j].rx.daddr, + ring->hw.descs[j].rx.dd); + trace_atl_rx_descr(ring->idx, j, + (u64 *)ring->hw.descs[j].raw); + } + } + + return 0; +} + +static int doit_dump_ring(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, dump_ring); +} + +/* ATL_FWD_CMD_SET_TX_BUNCH handler */ +static int set_tx_bunch(struct net_device *ndev, struct genl_info *info, + const int bunch) +{ + struct atl_nic *nic = netdev_priv(ndev); + + pr_debug(ATL_FWDNL_PREFIX "Set TX bunch %d\n", bunch); + nic->fwdnl.tx_bunch = bunch; + + return 0; +} + +static int doit_set_tx_bunch(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_s32_attr(skb, info, ATL_FWD_ATTR_TX_BUNCH_SIZE, + MANDATORY_ATTR, set_tx_bunch); +} + +/* ATL_FWD_CMD_REQUEST_EVENT handler */ +static int request_event(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct atl_desc_ring *desc = atlfwd_nl_get_fwd_ring_desc(ring); + int flags; + + if (is_tx_ring(ring)) { + if (unlikely(ring->evt)) { + ATLFWD_NL_SET_ERR_MSG(info, "Event exists already."); + return -EFAULT; + } + + desc->tx_evt = kzalloc(sizeof(*desc->tx_evt), GFP_KERNEL); + if (unlikely(!desc->tx_evt)) + return -ENOBUFS; + + /* right now we support head pointer writeback only */ + flags = ATL_FWD_EVT_TXWB; + desc->tx_evt->ring = ring; + desc->tx_evt->flags = flags; + if (desc->tx_evt->flags & ATL_FWD_EVT_TXWB) + desc->tx_evt->tx_head_wrb = dma_map_single( + &ring->nic->hw.pdev->dev, &desc->tx_hw_head, + sizeof(desc->tx_hw_head), DMA_FROM_DEVICE); + + pr_debug(ATL_FWDNL_PREFIX "Requesting event for ring %d (%p)\n", + nl_ring_index(ring), ring); + + return atl_fwd_request_event(desc->tx_evt); + } + + return 0; +} + +static int doit_request_event(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, request_event); +} + +/* ATL_FWD_CMD_RELEASE_EVENT handler */ +static int release_event(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + if (is_tx_ring(ring) && ring->evt) { + struct atl_desc_ring *desc = atlfwd_nl_get_fwd_ring_desc(ring); + + pr_debug(ATL_FWDNL_PREFIX "Releasing event for ring %d (%p)\n", + nl_ring_index(ring), ring); + atl_fwd_release_event(ring->evt); + + if (desc->tx_evt->flags & ATL_FWD_EVT_TXWB) + dma_unmap_single(&ring->nic->hw.pdev->dev, + desc->tx_evt->tx_head_wrb, + sizeof(desc->tx_hw_head), + DMA_FROM_DEVICE); + + kfree(desc->tx_evt); + desc->tx_evt = NULL; + } + + return 0; +} + +static int doit_release_event(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, release_event); +} + +/* ATL_FWD_CMD_ENABLE_EVENT handler */ +static int enable_event(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + if (is_tx_ring(ring) && ring->evt) { + pr_debug(ATL_FWDNL_PREFIX "Enabling event for ring %d (%p)\n", + nl_ring_index(ring), ring); + + return atl_fwd_enable_event(ring->evt); + } + + return 0; +} +static int doit_enable_event(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, enable_event); +} + +/* ATL_FWD_CMD_DISABLE_EVENT handler */ +static int disable_event(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + if (is_tx_ring(ring) && ring->evt) { + pr_debug(ATL_FWDNL_PREFIX "Disabling event for ring %d (%p)\n", + nl_ring_index(ring), ring); + + return atl_fwd_disable_event(ring->evt); + } + + return 0; +} + +static int doit_disable_event(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, disable_event); +} + +/* ATL_FWD_CMD_DISABLE_REDIRECTIONS handler */ +static int disable_redirections(struct net_device *ndev, struct genl_info *info) +{ + struct atl_nic *nic = netdev_priv(ndev); + + pr_debug(ATL_FWDNL_PREFIX "All forced redirections are disabled now\n"); + disable_fwd_queue(nic); + nic->fwdnl.force_icmp_via = S32_MIN; + nic->fwdnl.force_tx_via = S32_MIN; + + return 0; +} + +static int doit_disable_redirections(struct sk_buff *skb, + struct genl_info *info) +{ + return cmd_with_no_attr(skb, info, disable_redirections); +} + +/* ATL_FWD_CMD_FORCE_ICMP_TX_VIA handler */ +static int force_icmp_tx_via(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct atl_nic *nic = netdev_priv(ndev); + int ring_index = 0; + + if (unlikely(!is_tx_ring(ring))) { + ATLFWD_NL_SET_ERR_MSG(info, "Expected TX ring"); + return -EINVAL; + } + + ring_index = nl_ring_index(ring); + pr_debug(ATL_FWDNL_PREFIX + "All egress ICMP traffic is now forced via ring %d (%p)\n", + ring_index, ring); + nic->fwdnl.force_icmp_via = ring_index; + enable_fwd_queue(nic); + + return 0; +} + +static int doit_force_icmp_tx_via(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, force_icmp_tx_via); +} + +/* ATL_FWD_CMD_FORCE_TX_VIA handler */ +static int force_tx_via(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct atl_nic *nic = netdev_priv(ndev); + int ring_index = 0; + + if (unlikely(!is_tx_ring(ring))) { + ATLFWD_NL_SET_ERR_MSG(info, "Expected TX ring"); + return -EINVAL; + } + + ring_index = nl_ring_index(ring); + pr_debug(ATL_FWDNL_PREFIX + "All egress traffic is now forced via ring %d (%p)\n", + ring_index, ring); + nic->fwdnl.force_tx_via = ring_index; + enable_fwd_queue(nic); + + return 0; +} + +static int doit_force_tx_via(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, force_tx_via); +} + +/* ATL_FWD_CMD_RING_STATUS processor */ +static int atlfwd_nl_add_ring_status(struct net_device *netdev, + struct sk_buff *msg, void *hdr, + struct genl_info *info, + const int ring_index) +{ + const bool err_if_released = false; + const struct atl_fwd_ring *ring = + get_fwd_ring(netdev, ring_index, info, err_if_released); + + pr_debug(ATL_FWDNL_PREFIX "Ring %d (%p) status\n", ring_index, ring); + + if (unlikely(!nl_reply_add_attr(msg, hdr, info, ATL_FWD_ATTR_RING_INDEX, + ring_index))) + return -EMSGSIZE; + + if (unlikely(!nl_reply_add_attr(msg, hdr, info, ATL_FWD_ATTR_RING_IS_TX, + is_tx_ring_index(ring_index)))) + return -EMSGSIZE; + + if (ring != NULL) { + if (unlikely(!nl_reply_add_attr( + msg, hdr, info, ATL_FWD_ATTR_RING_STATUS, + (!(ring->state & ATL_FWR_ST_ENABLED) ? + ATL_FWD_RING_STATUS_CREATED_DISABLED : + ATL_FWD_RING_STATUS_ENABLED)))) + return -EMSGSIZE; + if (unlikely(!nl_reply_add_attr(msg, hdr, info, + ATL_FWD_ATTR_RING_SIZE, + ring->hw.size))) + return -EMSGSIZE; + if (unlikely(!nl_reply_add_attr(msg, hdr, info, + ATL_FWD_ATTR_RING_FLAGS, + ring->flags))) + return -EMSGSIZE; + } else { + if (unlikely(!nl_reply_add_attr(msg, hdr, info, + ATL_FWD_ATTR_RING_STATUS, + ATL_FWD_RING_STATUS_RELEASED))) + return -EMSGSIZE; + } + + return 0; +} + +static int ring_status(struct net_device *ndev, struct genl_info *info, + const int ring_index) +{ + struct sk_buff *msg = nl_reply_create(); + void *hdr = nl_reply_init(msg, info); + int last_idx = ATL_NUM_FWD_RINGS * 2; + int first_idx = 0; + int result = 0; + int idx = 0; + + if (unlikely(msg == NULL)) + return -ENOBUFS; + if (unlikely(hdr == NULL)) + return -EMSGSIZE; + + if (ring_index != S32_MIN) { + if (unlikely(!is_valid_ring_index(ring_index))) + return -EINVAL; + + first_idx = ring_index; + last_idx = ring_index + 1; + } + + for (idx = first_idx; idx != last_idx; idx++) { + result = atlfwd_nl_add_ring_status(ndev, msg, hdr, info, idx); + if (unlikely(result < 0)) + return result; + } + + return nl_reply_send(msg, hdr, info); +} + +static int doit_ring_status(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_s32_attr(skb, info, ATL_FWD_ATTR_RING_INDEX, + OPTIONAL_ATTR, ring_status); +} + +/* ATL_FWD_CMD_GET_RX_QUEUE/ATL_FWD_CMD_GET_TX_QUEUE processor */ +static int get_queue_index(struct net_device *ndev, struct genl_info *info, + struct atl_fwd_ring *ring) +{ + struct sk_buff *msg = nl_reply_create(); + void *hdr = nl_reply_init(msg, info); + + if (unlikely(msg == NULL)) + return -ENOBUFS; + if (unlikely(hdr == NULL)) + return -EMSGSIZE; + + if (unlikely(!nl_reply_add_attr(msg, hdr, info, + ATL_FWD_ATTR_QUEUE_INDEX, ring->idx))) + return -EMSGSIZE; + + return nl_reply_send(msg, hdr, info); +} +static int doit_get_queue(struct sk_buff *skb, struct genl_info *info) +{ + return cmd_with_ring_index_attr(skb, info, get_queue_index); +} + +/* This handler is called before the actual command handler. + * + * Returns 0 on success, error otherwise. + */ +static int atlfwd_nl_pre_doit(const struct genl_ops *ops, struct sk_buff *skb, + struct genl_info *info) +{ + enum atlfwd_nl_attribute missing_attr = ATL_FWD_ATTR_INVALID; + int ring_index = S32_MIN; + int ret = 0; + + /* First, check that all mandatory attributes are present */ + switch (ops->cmd) { + case ATL_FWD_CMD_REQUEST_RING: + if (!info->attrs[ATL_FWD_ATTR_FLAGS]) + missing_attr = ATL_FWD_ATTR_FLAGS; + else if (!info->attrs[ATL_FWD_ATTR_RING_SIZE]) + missing_attr = ATL_FWD_ATTR_RING_SIZE; + else if (!info->attrs[ATL_FWD_ATTR_BUF_SIZE]) + missing_attr = ATL_FWD_ATTR_BUF_SIZE; + else if (!info->attrs[ATL_FWD_ATTR_PAGE_ORDER]) + missing_attr = ATL_FWD_ATTR_PAGE_ORDER; + break; + case ATL_FWD_CMD_RELEASE_RING: + case ATL_FWD_CMD_ENABLE_RING: + case ATL_FWD_CMD_DISABLE_RING: + case ATL_FWD_CMD_REQUEST_EVENT: + case ATL_FWD_CMD_RELEASE_EVENT: + case ATL_FWD_CMD_ENABLE_EVENT: + case ATL_FWD_CMD_DISABLE_EVENT: + case ATL_FWD_CMD_DUMP_RING: + case ATL_FWD_CMD_FORCE_ICMP_TX_VIA: + case ATL_FWD_CMD_FORCE_TX_VIA: + case ATL_FWD_CMD_GET_RX_QUEUE: + case ATL_FWD_CMD_GET_TX_QUEUE: + if (!info->attrs[ATL_FWD_ATTR_RING_INDEX]) + missing_attr = ATL_FWD_ATTR_RING_INDEX; + break; + case ATL_FWD_CMD_RING_STATUS: + /* the only attribute is optional => nothing to check */ + break; + case ATL_FWD_CMD_SET_TX_BUNCH: + if (!info->attrs[ATL_FWD_ATTR_TX_BUNCH_SIZE]) + missing_attr = ATL_FWD_ATTR_TX_BUNCH_SIZE; + break; + case ATL_FWD_CMD_DISABLE_REDIRECTIONS: + /* no attributes => nothing to check */ + break; + default: + ATLFWD_NL_SET_ERR_MSG(info, "Unknown command"); + return -EINVAL; + } + + if (unlikely(missing_attr != ATL_FWD_ATTR_INVALID)) { + pr_warn(ATL_FWDNL_PREFIX "Required attribute is missing: %d\n", + missing_attr); + ATLFWD_NL_SET_ERR_MSG(info, "Required attribute is missing"); + return -EINVAL; + } + + /* Now, check additional pre-conditions (if any) */ + if (info->attrs[ATL_FWD_ATTR_RING_INDEX]) + ring_index = nla_get_s32(info->attrs[ATL_FWD_ATTR_RING_INDEX]); + + switch (ops->cmd) { + case ATL_FWD_CMD_GET_RX_QUEUE: + if (!is_rx_ring_index(ring_index)) { + ATLFWD_NL_SET_ERR_MSG(info, "Expected RX ring."); + return -EINVAL; + } + break; + case ATL_FWD_CMD_GET_TX_QUEUE: + if (!is_tx_ring_index(ring_index)) { + ATLFWD_NL_SET_ERR_MSG(info, "Expected TX ring."); + return -EINVAL; + } + break; + } + + return ret; +} + +/* netlink-specific structures */ +static const struct nla_policy atlfwd_nl_policy[NUM_ATL_FWD_ATTR] = { + [ATL_FWD_ATTR_IFNAME] = { .type = NLA_NUL_STRING, .len = IFNAMSIZ - 1 }, + [ATL_FWD_ATTR_FLAGS] = { .type = NLA_S32 }, + [ATL_FWD_ATTR_RING_SIZE] = { .type = NLA_S32 }, + [ATL_FWD_ATTR_BUF_SIZE] = { .type = NLA_S32 }, + [ATL_FWD_ATTR_PAGE_ORDER] = { .type = NLA_S32 }, + [ATL_FWD_ATTR_RING_INDEX] = { .type = NLA_S32 }, + [ATL_FWD_ATTR_TX_BUNCH_SIZE] = { .type = NLA_S32 }, + [ATL_FWD_ATTR_QUEUE_INDEX] = { .type = NLA_S32 }, +}; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 2, 0) +#define ATLFWD_NL_OP_POLICY(op_policy) .policy = op_policy +#else +#define ATLFWD_NL_OP_POLICY(op_policy) +#endif + +static const struct genl_ops atlfwd_nl_ops[] = { + { .cmd = ATL_FWD_CMD_REQUEST_RING, + .doit = doit_request_ring, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_RELEASE_RING, + .doit = doit_release_ring, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_ENABLE_RING, + .doit = doit_enable_ring, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_DISABLE_RING, + .doit = doit_disable_ring, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_DUMP_RING, + .doit = doit_dump_ring, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_SET_TX_BUNCH, + .doit = doit_set_tx_bunch, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_REQUEST_EVENT, + .doit = doit_request_event, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_RELEASE_EVENT, + .doit = doit_release_event, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_ENABLE_EVENT, + .doit = doit_enable_event, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_DISABLE_EVENT, + .doit = doit_disable_event, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_DISABLE_REDIRECTIONS, + .doit = doit_disable_redirections, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_FORCE_ICMP_TX_VIA, + .doit = doit_force_icmp_tx_via, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_FORCE_TX_VIA, + .doit = doit_force_tx_via, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_RING_STATUS, + .doit = doit_ring_status, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_GET_RX_QUEUE, + .doit = doit_get_queue, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, + { .cmd = ATL_FWD_CMD_GET_TX_QUEUE, + .doit = doit_get_queue, + ATLFWD_NL_OP_POLICY(atlfwd_nl_policy) }, +}; + +static struct genl_family atlfwd_nl_family = { + .hdrsize = 0, /* no private header */ + .name = ATL_FWD_GENL_NAME, /* have users key off the name instead */ + .version = 1, /* no particular meaning now */ + .maxattr = ATL_FWD_ATTR_MAX, + .netnsok = false, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 2, 0) + .policy = atlfwd_nl_policy, +#endif + .pre_doit = atlfwd_nl_pre_doit, + .ops = atlfwd_nl_ops, + .n_ops = ARRAY_SIZE(atlfwd_nl_ops), + .module = THIS_MODULE, +}; diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.h new file mode 100644 index 000000000000..743063dcea86 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl.h @@ -0,0 +1,122 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_FWDNL_H_ +#define _ATL_FWDNL_H_ + +/* family name */ +#define ATL_FWD_GENL_NAME "atl_fwd" + +/* commands */ +enum atlfwd_nl_command { + ATL_FWD_CMD_UNSPEC, + + ATL_FWD_CMD_REQUEST_RING, + ATL_FWD_CMD_RELEASE_RING, + ATL_FWD_CMD_ENABLE_RING, + ATL_FWD_CMD_DISABLE_RING, + + ATL_FWD_CMD_DISABLE_REDIRECTIONS, + ATL_FWD_CMD_FORCE_ICMP_TX_VIA, + ATL_FWD_CMD_FORCE_TX_VIA, + + ATL_FWD_CMD_RING_STATUS, + + ATL_FWD_CMD_DUMP_RING, + ATL_FWD_CMD_SET_TX_BUNCH, + + ATL_FWD_CMD_REQUEST_EVENT, + ATL_FWD_CMD_RELEASE_EVENT, + ATL_FWD_CMD_ENABLE_EVENT, + ATL_FWD_CMD_DISABLE_EVENT, + + ATL_FWD_CMD_GET_RX_QUEUE, + ATL_FWD_CMD_GET_TX_QUEUE, + + /* keep last */ + NUM_ATL_FWD_CMD, + ATL_FWD_CMD_MAX = NUM_ATL_FWD_CMD - 1 +}; + +enum atlfwd_nl_attribute { + ATL_FWD_ATTR_INVALID, + + ATL_FWD_ATTR_IFNAME, + + /* REQUEST_RING attributes */ + ATL_FWD_ATTR_FLAGS, + ATL_FWD_ATTR_RING_SIZE, + ATL_FWD_ATTR_BUF_SIZE, + ATL_FWD_ATTR_PAGE_ORDER, + + /* RELEASE_RING attributes */ + ATL_FWD_ATTR_RING_INDEX, + /* ENABLE_RING / DISABLE_RING use RING_INDEX attribute above */ + + /* RING_STATUS reply attributes */ + ATL_FWD_ATTR_RING_STATUS, + ATL_FWD_ATTR_RING_IS_TX, + ATL_FWD_ATTR_RING_FLAGS, + + /* ATL_FWD_CMD_SET_TX_BUNCH atributes */ + ATL_FWD_ATTR_TX_BUNCH_SIZE, + + /* ATL_FWD_CMD_GET_(RX|TX)_QUEUE attributes */ + ATL_FWD_ATTR_QUEUE_INDEX, + + /* keep last */ + NUM_ATL_FWD_ATTR, + ATL_FWD_ATTR_MAX = NUM_ATL_FWD_ATTR - 1 +}; + +enum atlfwd_nl_ring_status { + ATL_FWD_RING_STATUS_INVALID, + + ATL_FWD_RING_STATUS_RELEASED, + ATL_FWD_RING_STATUS_CREATED_DISABLED, + ATL_FWD_RING_STATUS_ENABLED, + + /* keep last */ + NUM_ATL_FWD_RING_STATUS, + ATL_FWD_RING_STATUS_MAX = NUM_ATL_FWD_RING_STATUS - 1 +}; + +#ifdef __KERNEL__ +#include +#include + +struct atl_fwd_ring; +struct atl_desc_ring; + +int atlfwd_nl_init(void); +void atlfwd_nl_on_probe(struct net_device *ndev); +void atlfwd_nl_on_remove(struct net_device *ndev); +int atlfwd_nl_on_open(struct net_device *ndev); +void atlfwd_nl_exit(void); + +netdev_tx_t atlfwd_nl_xmit(struct sk_buff *skb, struct net_device *ndev); +u16 atlfwd_nl_select_queue_fallback(struct net_device *dev, struct sk_buff *skb, + struct net_device *arg3, + select_queue_fallback_t fallback); + +bool atlfwd_nl_is_tx_fwd_ring_created(struct net_device *ndev, + const int fwd_ring_index); +bool atlfwd_nl_is_rx_fwd_ring_created(struct net_device *ndev, + const int fwd_ring_index); + +struct atl_fwd_ring *atlfwd_nl_get_fwd_ring(struct net_device *ndev, + const int ring_index); +struct atl_desc_ring *atlfwd_nl_get_fwd_ring_desc(struct atl_fwd_ring *ring); + +bool is_atlfwd_device(const struct net_device *dev); +#endif + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.c new file mode 100644 index 000000000000..73b01c15e442 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.c @@ -0,0 +1,150 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include + +#include "atl_common.h" + +unsigned int atlfwd_nl_tx_clean_threshold_msec = 200; +unsigned int atlfwd_nl_tx_clean_threshold_frac = 8; +bool atlfwd_nl_speculative_queue_stop = true; +unsigned int altfwd_nl_rx_poll_interval_msec = 200; +int atlfwd_nl_rx_clean_budget = NAPI_POLL_WEIGHT; + +typedef void (*for_each_func_t)(struct net_device *ndev); +static void for_each_atlfwd_device(for_each_func_t f) +{ + struct net_device *ndev; + struct net *ns; + + rcu_read_lock(); + for_each_net_rcu (ns) { + for_each_netdev_rcu (ns, ndev) { + if (!is_atlfwd_device(ndev)) + continue; + + f(ndev); + } + } + rcu_read_unlock(); +} + +static void force_tx_housekeeping(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + + /* Schedule for ASAP execution */ + schedule_delayed_work(nic->fwdnl.tx_cleanup_wq, 0); +} + +static void force_rx_polling(struct net_device *ndev) +{ + struct atl_desc_ring *desc = NULL; + struct atl_fwd_ring *ring = NULL; + int i; + + for (i = 0; i != ATL_NUM_FWD_RINGS; i++) { + if (!atlfwd_nl_is_rx_fwd_ring_created(ndev, i)) + continue; + + ring = atlfwd_nl_get_fwd_ring(ndev, i); + if (unlikely(!(ring->state & ATL_FWR_ST_ENABLED))) + continue; + + desc = atlfwd_nl_get_fwd_ring_desc(ring); + if (likely(desc != NULL && desc->rx_poll_timer != NULL)) { + desc->rx_poll_timer->expires = + jiffies + + msecs_to_jiffies( + altfwd_nl_rx_poll_interval_msec); + add_timer(desc->rx_poll_timer); + } + } +} + +static int tx_clean_threshold_msec_set(const char *val, + const struct kernel_param *kp) +{ + unsigned int msec; + int ret = kstrtouint(val, 10, &msec); + + if (unlikely(ret != 0)) + return -EINVAL; + + if (msec != 0 && atlfwd_nl_tx_clean_threshold_msec == 0) { + /* Force extra TX housekeeping, if threshold is re-enabled */ + for_each_atlfwd_device(force_tx_housekeeping); + } + + return param_set_uint(val, kp); +} + +static const struct kernel_param_ops fwdnl_tx_clean_threshold_msec_ops = { + .set = tx_clean_threshold_msec_set, + .get = param_get_uint, +}; + +static int tx_clean_threshold_frac_set(const char *val, + const struct kernel_param *kp) +{ + unsigned int frac; + int ret = kstrtouint(val, 10, &frac); + + if (unlikely(ret != 0)) + return -EINVAL; + + if (frac != 0 && atlfwd_nl_tx_clean_threshold_frac == 0) { + /* Force extra TX housekeeping, if threshold is re-enabled */ + for_each_atlfwd_device(force_tx_housekeeping); + } + + return param_set_uint(val, kp); +} + +static const struct kernel_param_ops fwdnl_tx_clean_threshold_frac_ops = { + .set = tx_clean_threshold_frac_set, + .get = param_get_uint, +}; + +static int rx_poll_interval_msec_set(const char *val, + const struct kernel_param *kp) +{ + unsigned int msec; + int ret = kstrtouint(val, 10, &msec); + + if (unlikely(ret != 0)) + return -EINVAL; + + if (msec != 0 && altfwd_nl_rx_poll_interval_msec == 0) { + /* Kick off RX polling, if polling is re-enabled */ + for_each_atlfwd_device(force_rx_polling); + } + + return param_set_uint(val, kp); +} + +static const struct kernel_param_ops fwdnl_rx_poll_interval_msec_ops = { + .set = rx_poll_interval_msec_set, + .get = param_get_uint, +}; + +module_param_cb(fwdnl_tx_clean_threshold_msec, + &fwdnl_tx_clean_threshold_msec_ops, + &atlfwd_nl_tx_clean_threshold_msec, 0644); +module_param_cb(fwdnl_tx_clean_threshold_frac, + &fwdnl_tx_clean_threshold_frac_ops, + &atlfwd_nl_tx_clean_threshold_frac, 0644); +module_param_named(fwdnl_speculative_queue_stop, + atlfwd_nl_speculative_queue_stop, bool, 0644); +module_param_cb(fwdnl_rx_poll_interval_msec, &fwdnl_rx_poll_interval_msec_ops, + &altfwd_nl_rx_poll_interval_msec, 0644); +module_param_named(fwdnl_rx_clean_budget, atlfwd_nl_rx_clean_budget, int, 0644); diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.h new file mode 100644 index 000000000000..8a30ab9c545c --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_fwdnl_params.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_FWDNL_PARAMS_H_ +#define _ATL_FWDNL_PARAMS_H_ + +extern unsigned int atlfwd_nl_tx_clean_threshold_msec; +extern unsigned int atlfwd_nl_tx_clean_threshold_frac; +extern bool atlfwd_nl_speculative_queue_stop; +extern unsigned int altfwd_nl_rx_poll_interval_msec; +extern int atlfwd_nl_rx_clean_budget; + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.c new file mode 100644 index 000000000000..da6fe59f363a --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.c @@ -0,0 +1,1602 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include + +#include "atl_common.h" +#include "atl_hw.h" +#include "atl_ptp.h" +#include "atl_ring.h" +#include "atl2_fw.h" + +static void atl_unplugged(struct atl_hw *hw) +{ + if (!hw->regs) + return; + hw->regs = 0; + dev_err(&hw->pdev->dev, "Device removed\n"); +} + +void atl_check_unplug(struct atl_hw *hw, uint32_t addr) +{ + uint32_t val; + + if (addr == ATL_GLOBAL_MIF_ID) { + atl_unplugged(hw); + return; + } + + val = atl_read(hw, ATL_GLOBAL_MIF_ID); + if (val == 0xffffffff) + atl_unplugged(hw); +} + +int atl_read_mcp_mem(struct atl_hw *hw, uint32_t mcp_addr, void *host_addr, + unsigned int size) +{ + uint32_t *addr = (uint32_t *)host_addr; + int ret; + + ret = atl_hwsem_get(hw, ATL_MCP_SEM_MEM); + if (ret) + return ret; + + size = (size + 3) & ~3u; + atl_write(hw, ATL_GLOBAL_MBOX_ADDR, mcp_addr); + while (size) { + uint32_t next; + + atl_write(hw, ATL_GLOBAL_MBOX_CTRL, 0x8000); + + if (hw->chip_rev == 0xb1) { + busy_wait(100, udelay(10), next, + atl_read(hw, ATL_GLOBAL_MBOX_ADDR), + next == mcp_addr); + if (next == mcp_addr) { + atl_dev_err("mcp mem read timed out (%d remaining)\n", + size); + ret = -EIO; + goto err_exit; + } + } else + busy_wait(100, udelay(10), next, + atl_read(hw, ATL_GLOBAL_MBOX_CTRL), + next & BIT(8)); + + *addr = atl_read(hw, ATL_GLOBAL_MBOX_DATA); + mcp_addr += 4; + addr++; + size -= 4; + } +err_exit: + atl_hwsem_put(hw, ATL_MCP_SEM_MEM); + return ret; +} + + +static inline void atl_glb_soft_reset(struct atl_hw *hw) +{ + atl_write_bit(hw, ATL_GLOBAL_STD_CTRL, 14, 0); + atl_write_bit(hw, ATL_GLOBAL_STD_CTRL, 15, 1); +} + +static inline void atl_glb_soft_reset_full(struct atl_hw *hw) +{ + atl_write_bit(hw, ATL_TX_CTRL1, 29, 0); + atl_write_bit(hw, ATL_RX_CTRL1, 29, 0); + atl_write_bit(hw, ATL_INTR_CTRL, 29, 0); + atl_write_bit(hw, ATL_MPI_CTRL1, 29, 0); + atl_glb_soft_reset(hw); +} + +static void atl2_hw_new_rx_filter_vlan_promisc(struct atl_hw *hw, bool promisc); +static void atl2_hw_new_rx_filter_promisc(struct atl_hw *hw, bool promisc, + bool allmulti); +static void atl2_hw_init_new_rx_filters(struct atl_hw *hw); + +static void atl_set_promisc(struct atl_hw *hw, bool enabled, bool allmulti) +{ + atl_write_bit(hw, ATL_RX_FLT_CTRL1, 3, enabled); + if (hw->new_rpf) + atl2_hw_new_rx_filter_promisc(hw, enabled, allmulti); + else { + atl_write_bit(hw, ATL_RX_MC_FLT_MSK, 14, allmulti); + atl_write(hw, ATL_RX_MC_FLT(0), allmulti ? 0x80010FFF : 0x00010FFF); + } +} + +void atl_set_vlan_promisc(struct atl_hw *hw, int promisc) +{ + atl_write_bit(hw, ATL_RX_VLAN_FLT_CTRL1, 1, !!promisc); + if (hw->new_rpf) + atl2_hw_new_rx_filter_vlan_promisc(hw, !!promisc); +} + +static inline void atl_enable_dma_net_lpb_mode(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + atl_set_vlan_promisc(hw, 1); + atl_set_promisc(hw, 1, 0); + atl_write_bit(hw, ATL_TX_PBUF_CTRL1, 4, 0); + atl_write_bit(hw, ATL_TX_CTRL1, 4, 1); + atl_write_bit(hw, ATL_RX_CTRL1, 4, 1); +} +/* entered with fw lock held */ +static int atl_hw_reset_nonrbl(struct atl_hw *hw) +{ + uint32_t tries; + uint32_t reg = atl_read(hw, ATL_GLOBAL_DAISY_CHAIN_STS1); + int ret; + + bool daisychain_running = (reg & 0x30) != 0x30; + + if (daisychain_running) + atl_dev_dbg("AQDBG: daisychain running (0x18: %#x)\n", + atl_read(hw, ATL_GLOBAL_FW_IMAGE_ID)); + + atl_write(hw, 0x404, 0x40e1); + mdelay(50); + + atl_write(hw, 0x534, 0xa0); + atl_write(hw, 0x100, 0x9f); + atl_write(hw, 0x100, 0x809f); + mdelay(50); + + atl_glb_soft_reset(hw); + + atl_write(hw, 0x404, 0x80e0); + atl_write(hw, 0x32a8, 0); + atl_write(hw, 0x520, 1); + mdelay(50); + atl_write(hw, 0x404, 0x180e0); + + tries = busy_wait(10000, mdelay(1), reg, atl_read(hw, 0x704), + !(reg & 0x10)); + if (!(reg & 0x10)) { + atl_dev_err("FLB kickstart timed out: %#x\n", reg); + ret = -EIO; + goto unlock; + } + atl_dev_dbg("FLB kickstart took %d ms\n", tries); + + atl_write(hw, 0x404, 0x40e1); + mdelay(50); + atl_write(hw, 0x3a0, 1); + + atl_glb_soft_reset_full(hw); + + if (hw->mcp.ops) + hw->mcp.ops->restore_cfg(hw); + + /* unstall FW*/ + atl_write(hw, 0x404, 0x40e0); + + ret = atl_fw_init(hw); + +unlock: + atl_unlock_fw(hw); + + if (ret) + set_bit(ATL_ST_RESET_NEEDED, &hw->state); + else + set_bit(ATL_ST_GLOBAL_CONF_NEEDED, &hw->state); + + return ret; +} + + +#define ATL2_FW_READ_DONE BIT(0x10) +#define ATL2_FW_READ_REQ BIT(0x11) +#define ATL2_BOOT_STARTED BIT(0x18) +#define ATL2_CRASH_INIT BIT(0x1B) +#define ATL2_BOOT_CODE_FAILED BIT(0x1C) +#define ATL2_FW_INIT_FAILED BIT(0x1D) +#define ATL2_FW_INIT_COMP_SUCCESS BIT(0x1F) +#define ATL2_FW_BOOT_FAILED_MASK (ATL2_CRASH_INIT | \ + ATL2_BOOT_CODE_FAILED | \ + ATL2_FW_INIT_FAILED) +#define ATL2_FW_BOOT_COMPLETE_MASK (ATL2_FW_BOOT_FAILED_MASK | \ + ATL2_FW_READ_REQ | \ + ATL2_FW_INIT_COMP_SUCCESS) + +#define ATL2_FW_BOOT_REQ_REBOOT BIT(0x0) +#define ATL2_FW_BOOT_REQ_HOST_BOOT BIT(0x8) +#define ATL2_FW_BOOT_REQ_MAC_FAST_BOOT BIT(0xA) +#define ATL2_FW_BOOT_REQ_PHY_FAST_BOOT BIT(0xB) +#define ATL2_FW_BOOT_REQ_HOST_ITI BIT(0xC) + +static bool atl2_mcp_boot_complete(struct atl_hw *hw) +{ + u32 rbl_status; + + rbl_status = atl_read(hw, ATL2_MIF_BOOT_REG_ADR); + if (rbl_status & ATL2_FW_BOOT_COMPLETE_MASK) + return true; + + /* Host boot requested */ + if (atl_read(hw, ATL2_MCP_HOST_REQ_INT) & 0x1) + return true; + + return false; +} + +static int atl2_hw_reset(struct atl_hw *hw) +{ + bool rbl_complete = false; + u32 rbl_status = 0; + u32 rbl_request; + int err = 0; + + atl_lock_fw(hw); + + busy_wait(50, mdelay(1), rbl_status, + atl_read(hw, ATL2_MIF_BOOT_REG_ADR), + ((rbl_status & ATL2_BOOT_STARTED) == 0) || + (rbl_status == 0xffffffff)); + if (!(rbl_status & ATL2_BOOT_STARTED)) + atl_dev_dbg("Boot code probably hung, reboot anyway"); + + + atl_write(hw, ATL2_MCP_HOST_REQ_INT_CLR, 0x01); + rbl_request = ATL2_FW_BOOT_REQ_REBOOT; + if (hw->mcp.ops) { + rbl_request |= ATL2_FW_BOOT_REQ_MAC_FAST_BOOT; + rbl_request |= ATL2_FW_BOOT_REQ_HOST_ITI; + } +#ifdef AQ_CFG_FAST_START + rbl_request |= ATL2_FW_BOOT_REQ_MAC_FAST_BOOT; +#endif + + atl_write(hw, ATL2_MIF_BOOT_REG_ADR, rbl_request); + + /* Wait for RBL boot */ + busy_wait(200, mdelay(1), rbl_status, + atl_read(hw, ATL2_MIF_BOOT_REG_ADR), + ((rbl_status & ATL2_BOOT_STARTED) == 0) || + (rbl_status == 0xffffffff)); + if (!(rbl_status & ATL2_BOOT_STARTED)) { + err = -ETIME; + atl_dev_err("Boot code hung, rbl_status %#x", rbl_status); + goto unlock; + } + +next_turn: + busy_wait(1000, mdelay(1), rbl_complete, atl2_mcp_boot_complete(hw), + !rbl_complete); + if (!rbl_complete) { + err = -ETIME; + atl_dev_err("FW Restart timed out, status %#x", + atl_read(hw, ATL2_MIF_BOOT_REG_ADR)); + goto unlock; + } + + rbl_status = atl_read(hw, ATL2_MIF_BOOT_REG_ADR); + + if (rbl_status & ATL2_FW_BOOT_FAILED_MASK) { + err = -EIO; + atl_dev_err("FW Restart failed, status = %#x", rbl_status); + goto unlock; + } + + if (atl_read(hw, ATL2_MCP_HOST_REQ_INT) & 0x1) { + uint32_t req_adr = atl_read(hw, ATL2_MIF_BOOT_READ_REQ_ADR); + uint32_t req_len = atl_read(hw, ATL2_MIF_BOOT_READ_REQ_LEN); + + atl_write(hw, ATL2_MCP_HOST_REQ_INT_CLR, 0x01); + + if (((req_adr & BIT(31)) != 0) || + (req_len >= ATL2_FW_HOSTLOAD_REQ_LEN_MAX)) { + err = -EIO; + atl_dev_err("FW read request invalid"); + goto unlock; + } + if (req_adr >= ATL2_ITI_ADDRESS_START) + if (hw->mcp.ops) { + err = hw->mcp.ops->restore_cfg(hw); + if (!err) + goto next_turn; + } + + err = -EIO; + atl_dev_err("No FW detected. Dynamic FW load not implemented"); + goto unlock; + } + + err = atl2_fw_init(hw); + +unlock: + atl_unlock_fw(hw); + + if (err) + set_bit(ATL_ST_RESET_NEEDED, &hw->state); + else + set_bit(ATL_ST_GLOBAL_CONF_NEEDED, &hw->state); + return err; +} + +static int atl1_hw_reset(struct atl_hw *hw) +{ + uint32_t reg; + uint32_t flb_stat; + int tries = 0; + /* bool host_load_done = false; */ + int ret; + + atl_lock_fw(hw); + + reg = atl_read(hw, ATL_MCP_SCRATCH(RBL_STS)); + flb_stat = atl_read(hw, ATL_GLOBAL_DAISY_CHAIN_STS1); + + while (!reg && flb_stat == 0x6000000 && tries++ < 1000) { + mdelay(1); + reg = atl_read(hw, ATL_MCP_SCRATCH(RBL_STS)); + flb_stat = atl_read(hw, ATL_GLOBAL_DAISY_CHAIN_STS1); + } + + atl_dev_dbg("0x388: %#x 0x704: %#x\n", reg, flb_stat); + if (tries >= 1000) { + atl_dev_err("Timeout waiting to choose RBL or FLB path\n"); + ret = -EIO; + goto unlock; + } + + if (!reg) + /* atl_hw_reset_nonrbl() releases the fw lock */ + return atl_hw_reset_nonrbl(hw); + + atl_write(hw, 0x404, 0x40e1); + atl_write(hw, 0x3a0, 1); + atl_write(hw, 0x32a8, 0); + + atl_write(hw, ATL_MCP_SCRATCH(RBL_STS), 0xdead); + + atl_glb_soft_reset_full(hw); + + if (hw->mcp.ops) + hw->mcp.ops->restore_cfg(hw); + + atl_write(hw, ATL_GLOBAL_CTRL2, 0x40e0); + + for (tries = 0; tries < 10000; mdelay(1)) { + tries++; + reg = atl_read(hw, ATL_MCP_SCRATCH(RBL_STS)) & 0xffff; + + if (reg && reg != 0xdead) + break; + } + + if (reg == 0xf1a7) { + atl_dev_err("MAC FW Host load not supported yet\n"); + ret = -EIO; + goto unlock; + } + if (!reg || reg == 0xdead) { + atl_dev_err("RBL restart timeout: %#x\n", reg); + ret = -EIO; + goto unlock; + } + atl_dev_dbg("RBL restart took %d ms result %#x\n", tries, reg); + + /* if (host_load_done) { */ + /* // Wait for MAC FW to decide whether it wants to reload the PHY FW */ + /* busy_wait(10, mdelay(1), reg, atl_read(hw, 0x340), !(reg & (1 << 9 | 1 << 1 | 1 << 0))); */ + + /* if (reg & 1 << 9) { */ + /* ret = atl_load_phy_fw(hw); */ + /* if (ret) { */ + /* atl_dev_err("PHY FW host load failed\n"); */ + /* return ret; */ + /* } */ + /* } */ + /* } */ + + ret = atl_fw_init(hw); + +unlock: + atl_unlock_fw(hw); + + if (ret) + set_bit(ATL_ST_RESET_NEEDED, &hw->state); + else + set_bit(ATL_ST_GLOBAL_CONF_NEEDED, &hw->state); + + return ret; +} + +/* Must be called either during early init when netdev isn't yet + * registered, or with RTNL lock held + */ +int atl_hw_reset(struct atl_hw *hw) +{ + int ret; + + if (hw->chip_id == ATL_ANTIGUA) + ret = atl2_hw_reset(hw); + else + ret = atl1_hw_reset(hw); + if (ret) + return ret; + + ret = atl_fw_configure(hw); + + return ret; +} + +static int atl_get_mac_addr(struct atl_hw *hw, uint8_t *buf) +{ + int ret; + + ret = hw->mcp.ops->get_mac_addr(hw, buf); + + return ret; +} + +static unsigned int atl_newrpf = 1; +atl_module_param(newrpf, uint, 0644); + +int atl_hwinit(struct atl_hw *hw, enum atl_chip chip_id) +{ + int ret; + + hw->chip_id = chip_id; + hw->chip_rev = ((atl_read(hw, ATL_GLOBAL_MIF_ID) & 0xf) == 0xa) ? + 0xb1 : 0xb0; + + if (chip_id == ATL_ANTIGUA && atl_newrpf) + hw->new_rpf = 1; + + hw->thermal = atl_def_thermal; + + ret = atl_hw_reset(hw); + + atl_dev_info("rev 0x%x chip 0x%x FW img 0x%x\n", + atl_read(hw, ATL_GLOBAL_CHIP_REV) & 0xffff, + atl_read(hw, ATL_GLOBAL_CHIP_ID) & 0xffff, + hw->mcp.fw_rev); + + if (ret) + return ret; + + ret = atl_get_mac_addr(hw, hw->mac_addr); + if (ret) + atl_dev_err("couldn't read MAC address\n"); + + return ret; +} + +static void atl_rx_xoff_set(struct atl_hw *hw, bool fc) +{ + atl_write_bit(hw, ATL_RX_PBUF_REG2(0), 31, fc); +} + +void atl_refresh_link(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + struct atl_link_type *link, *prev_link = hw->link_state.link; + + if (test_bit(ATL_ST_RESETTING, &hw->state) || + !test_bit(ATL_ST_ENABLED, &hw->state) || + !test_and_clear_bit(ATL_ST_UPDATE_LINK, &hw->state)) + return; + + link = hw->mcp.ops->check_link(hw); + + if (link) { + if (link != prev_link) { + atl_nic_info("Link up: %s\n", link->name); + netif_carrier_on(nic->ndev); + pm_runtime_get_sync(&nic->hw.pdev->dev); +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + { + int ret; + + ret = atl_init_macsec(hw); + + if (ret) + atl_dev_err("atl_init_macsec failed with %d", ret); + } +#endif + } + } else { + if (link != prev_link) { + atl_nic_info("Link down\n"); + netif_carrier_off(nic->ndev); + pm_runtime_put_sync(&nic->hw.pdev->dev); + } + } + if (nic->ptp) { + atl_ptp_clock_init(nic); + atl_ptp_tm_offset_set(nic, link ? link->speed : 0); + atl_ptp_link_change(nic); + } + + atl_rx_xoff_set(hw, !!(hw->link_state.fc.cur & atl_fc_rx)); + + atl_intr_enable_non_ring(nic); +} + +static irqreturn_t atl_link_irq(int irq, void *priv) +{ + struct atl_nic *nic = (struct atl_nic *)priv; + + set_bit(ATL_ST_UPDATE_LINK, &nic->hw.state); + atl_schedule_work(nic); + if (nic->hw.chip_id == ATL_ANTIGUA) + atl_write(&nic->hw, ATL2_MCP_HOST_REQ_INT_CLR, 0xffff); + + return IRQ_HANDLED; +} + +static irqreturn_t atl_legacy_irq(int irq, void *priv) +{ + struct atl_nic *nic = priv; + struct atl_hw *hw = &nic->hw; + uint32_t mask = hw->non_ring_intr_mask; + uint32_t stat; + int cpu; + int i; + + for (i = 0; i != nic->nvecs; i++) + mask |= BIT(atl_qvec_intr(&nic->qvecs[i])); + + stat = atl_read(hw, ATL_INTR_STS); + + /* Mask asserted intr sources */ + atl_intr_disable(hw, stat); + + if (!(stat & mask)) + /* Interrupt from another device on a shared int + * line. As no status bits were set, nothing was + * masked above, so no need to unmask anything. */ + return IRQ_NONE; + + for (i = 0; i != nic->nvecs; i++) { + if (likely(stat & BIT(atl_qvec_intr(&nic->qvecs[i])))) { + if (nic->nvecs == 1 || !atl_wq_non_msi) { + atl_ring_irq(irq, &nic->qvecs[i].napi); + continue; + } + + cpu = cpumask_any(&nic->qvecs[i].affinity_hint); + WARN_ON_ONCE(cpu >= nr_cpu_ids); + if (cpu >= nr_cpu_ids) + cpu = 0; + schedule_work_on(cpu, nic->qvecs[i].work); + } + } + + if (unlikely(stat & hw->non_ring_intr_mask)) + atl_link_irq(irq, nic); + return IRQ_HANDLED; +} + +int atl_alloc_link_intr(struct atl_nic *nic) +{ + struct pci_dev *pdev = nic->hw.pdev; + int ret; + + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) { + ret = request_irq(pci_irq_vector(pdev, 0), atl_link_irq, + IRQF_NO_SUSPEND, nic->ndev->name, nic); + if (ret) + atl_nic_err("request MSI link vector failed: %d\n", + -ret); + return ret; + } + + ret = request_irq(pci_irq_vector(pdev, 0), atl_legacy_irq, IRQF_SHARED, + nic->ndev->name, nic); + if (ret) + atl_nic_err("request legacy irq failed: %d\n", -ret); + + return ret; +} + +void atl_free_link_intr(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + atl_intr_disable(hw, BIT(0)); + free_irq(pci_irq_vector(hw->pdev, 0), nic); +} + +void atl_set_uc_flt(struct atl_hw *hw, int idx, uint8_t mac_addr[ETH_ALEN]) +{ + atl_write(hw, ATL_RX_UC_FLT_REG1(idx), + be32_to_cpu(*(uint32_t *)&mac_addr[2])); + atl_write(hw, ATL_RX_UC_FLT_REG2(idx), + (uint32_t)be16_to_cpu(*(uint16_t *)mac_addr) | + 1 << 16 | 1 << 31); + atl_write_bits(hw, ATL_RX_UC_FLT_REG2(idx), 22, 6, ATL2_RPF_TAG_BASE_UC); +} + +static void atl_disable_uc_flt(struct atl_hw *hw, int idx) +{ + atl_write(hw, ATL_RX_UC_FLT_REG2(idx), 0); +} + +void atl_set_rss_key(struct atl_hw *hw) +{ + int i; + uint32_t val; + + for (i = 0; i < ATL_RSS_KEY_SIZE / 4; i++) { + val = swab32(((uint32_t *)hw->rss_key)[i]); + atl_write(hw, ATL_RX_RSS_KEY_WR_DATA, val); + atl_write(hw, ATL_RX_RSS_KEY_ADDR, i | BIT(5)); + busy_wait(100, udelay(1), val, + atl_read(hw, ATL_RX_RSS_KEY_ADDR), + val & BIT(5)); + if (val & BIT(5)) { + atl_dev_err("Timeout writing RSS key[%d]: %#x\n", + i, val); + return; + } + } +} + +int atl_set_rss_tbl(struct atl_hw *hw) +{ + int i, shift = 0, addr = 0; + uint32_t val = 0, stat; + uint32_t tc = 0; + + if (hw->new_rpf) + for (i = ATL_RSS_TBL_SIZE; i--;) { + atl_write_bits(hw, ATL2_RPF_RSS_REDIR(tc, i), + 5 * ((tc) % 4), 5, hw->rss_tbl[i]); + } + + for (i = 0; i < ATL_RSS_TBL_SIZE; i++) { + val |= (uint32_t)(hw->rss_tbl[i]) << shift; + shift += 3; + + if (shift < 16) + continue; + + atl_write(hw, ATL_RX_RSS_TBL_WR_DATA, val & 0xffff); + atl_write(hw, ATL_RX_RSS_TBL_ADDR, addr | BIT(4)); + + busy_wait(100, udelay(1), stat, + atl_read(hw, ATL_RX_RSS_TBL_ADDR), stat & BIT(4)); + if (stat & BIT(4)) { + atl_dev_err("Timeout writing RSS redir table[%d] (addr %d): %#x\n", + i, addr, stat); + return -ETIME; + } + + shift -= 16; + val >>= 16; + addr++; + } + return 0; +} + +static unsigned int atl_ptp_rx_buf_reserve = 16; +static unsigned int atl_ptp_tx_buf_reserve = 8; + +unsigned int atl_fwd_rx_buf_reserve = +#ifdef CONFIG_ATLFWD_FWD_RXBUF + CONFIG_ATLFWD_FWD_RXBUF; +#else + 0; +#endif + +unsigned int atl_fwd_tx_buf_reserve = +#ifdef CONFIG_ATLFWD_FWD_TXBUF + CONFIG_ATLFWD_FWD_TXBUF; +#else + 0; +#endif + +module_param_named(fwd_tx_buf_reserve, atl_fwd_tx_buf_reserve, uint, 0444); +module_param_named(fwd_rx_buf_reserve, atl_fwd_rx_buf_reserve, uint, 0444); + +/* Must be called either during early init when netdev isn't yet + * registered, or with RTNL lock held */ +void atl_start_hw_global(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + int rpb_size = 320, tpb_size = 160; + + if (!test_and_clear_bit(ATL_ST_GLOBAL_CONF_NEEDED, &hw->state)) + return; + + if (nic->priv_flags & ATL_PF_BIT(LPB_NET_DMA)) + atl_enable_dma_net_lpb_mode(nic); + + /* Enable TPO2 */ + atl_write(hw, 0x7040, 0x10000); + /* Enable RPF2, filter logic 3 */ + atl_write(hw, 0x5040, BIT(16) | (3 << 17)); + + if (hw->chip_id == ATL_ANTIGUA) { + rpb_size = 192; + tpb_size = 128; + } + /* Alloc TPB */ + /* TC2: space for PTP */ + tpb_size -= atl_ptp_tx_buf_reserve; + atl_write(hw, ATL_TX_PBUF_REG1(2), atl_ptp_tx_buf_reserve); + /* TC1: space for offload engine iface */ + tpb_size -= atl_fwd_tx_buf_reserve; + atl_write(hw, ATL_TX_PBUF_REG1(1), atl_fwd_tx_buf_reserve); + atl_write(hw, ATL_TX_PBUF_REG2(1), + (atl_fwd_tx_buf_reserve * 32 * 66 / 100) << 16 | + (atl_fwd_tx_buf_reserve * 32 * 50 / 100)); + /* TC0: 160k minus TC1 size */ + atl_write(hw, ATL_TX_PBUF_REG1(0), tpb_size); + atl_write(hw, ATL_TX_PBUF_REG2(0), + (tpb_size * 32 * 66 / 100) << 16 | + (tpb_size * 32 * 50 / 100)); + /* 4-TC | Enable TPB */ + atl_set_bits(hw, ATL_TX_PBUF_CTRL1, BIT(8) | BIT(0)); + /* TX Buffer clk gate off */ + if (hw->chip_id == ATL_ANTIGUA) + atl_clear_bits(hw, ATL_TX_PBUF_CTRL1, BIT(5)); + + /* Alloc RPB */ + /* TC2: space for PTP */ + rpb_size -= atl_ptp_rx_buf_reserve; + atl_write(hw, ATL_RX_PBUF_REG1(2), atl_ptp_rx_buf_reserve); + /* No flow control for PTP */ + atl_write_bit(hw, ATL_RX_PBUF_REG2(2), 31, 0); + /* TC1: space for offload engine iface */ + rpb_size -= atl_fwd_rx_buf_reserve; + atl_write(hw, ATL_RX_PBUF_REG1(1), atl_fwd_rx_buf_reserve); + atl_write(hw, ATL_RX_PBUF_REG2(1), BIT(31) | + (atl_fwd_rx_buf_reserve * 32 * 66 / 100) << 16 | + (atl_fwd_rx_buf_reserve * 32 * 50 / 100)); + /* TC1: 320k minus TC1 size */ + atl_write(hw, ATL_RX_PBUF_REG1(0), rpb_size); + atl_write(hw, ATL_RX_PBUF_REG2(0), BIT(31) | + (rpb_size * 32 * 66 / 100) << 16 | + (rpb_size * 32 * 50 / 100)); + /* 4-TC | Enable RPB */ + atl_set_bits(hw, ATL_RX_PBUF_CTRL1, BIT(8) | BIT(4) | BIT(0)); + + /* TPO */ + /* Enable L3 | L4 chksum */ + atl_set_bits(hw, ATL_TX_PO_CTRL1, 3); + /* TSO TCP flags bitmask first / middle */ + atl_write(hw, ATL_TX_LSO_TCP_CTRL1, 0x0ff60ff6); + /* TSO TCP flags bitmask last */ + atl_write(hw, ATL_TX_LSO_TCP_CTRL2, 0xf7f); + + /* RPO */ + /* Enable L3 | L4 chksum */ + atl_set_bits(hw, ATL_RX_PO_CTRL1, 3); + atl_write_bits(hw, ATL_RX_LRO_CTRL2, 12, 2, 0); + atl_write_bits(hw, ATL_RX_LRO_CTRL2, 5, 2, 0); + /* 10uS base, 20uS inactive timeout, 60 uS max coalescing + * interval + */ + atl_write(hw, ATL_RX_LRO_TMRS, 0xc35 << 20 | 2 << 10 | 6); + atl_write(hw, ATL_INTR_RSC_DELAY, (atl_min_intr_delay / 2) - 1); + + /* RPF */ + /* Default RPF2 parser options */ + atl_write(hw, ATL_RX_FLT_CTRL2, 0x0); + atl_set_uc_flt(hw, nic->rxf_mac.base_index, hw->mac_addr); + /* BC action host */ + atl_write_bits(hw, ATL_RX_FLT_CTRL1, 12, 1, 1); + /* Enable BC */ + atl_write_bit(hw, ATL_RX_FLT_CTRL1, 0, 1); + /* BC thresh */ + atl_write_bits(hw, ATL_RX_FLT_CTRL1, 16, 16, 0x1000); + + /* Enable untagged packets */ + atl_write(hw, ATL_RX_VLAN_FLT_CTRL1, 1 << 2 | 1 << 3); + + if (hw->chip_id == ATL_ANTIGUA) { + /* RSS hash type */ + atl_set_bits(hw, ATL2_RX_RSS_HASH_TYPE_ADR, BIT(9) - 1); + + /* ATL2 Apply legacy ring to TC mapping */ + atl_write(hw, ATL2_RX_Q_TO_TC_MAP(0), 0x00000000); + atl_write(hw, ATL2_RX_Q_TO_TC_MAP(1), 0x11111111); + atl_write(hw, ATL2_RX_Q_TO_TC_MAP(2), 0x22222222); + atl_write(hw, ATL2_RX_Q_TO_TC_MAP(3), 0x33333333); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(0), 0x00000000); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(1), 0x00000000); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(2), 0x01010101); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(3), 0x01010101); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(4), 0x02020202); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(5), 0x02020202); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(6), 0x03030303); + atl_write(hw, ATL2_TX_Q_TO_TC_MAP(7), 0x03030303); + + } + /* Turn new filters on*/ + if (hw->new_rpf) { + atl_set_bits(hw, ATL_RX_FLT_CTRL2, BIT(0xB)); + atl2_hw_init_new_rx_filters(hw); + } + + /* Reprogram ethtool Rx filters */ + atl_refresh_rxfs(nic); + + atl_set_rss_key(hw); + /* Enable RSS | 8 queues per TC */ + atl_write(hw, ATL_RX_RSS_CTRL, BIT(31) | 3); + + /* Global interrupt block init */ + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) { + /* MSI or MSI-X mode interrupt mode */ + uint32_t ctrl = hw->pdev->msix_enabled ? 2 : 1; + + /* Enable multi-vector mode and mask autoclear + * register */ + ctrl |= BIT(2) | BIT(5); + + atl_write(hw, ATL_INTR_CTRL, ctrl); + + /* Enable auto-masking of link interrupt on intr generation */ + atl_set_bits(hw, ATL_INTR_AUTO_MASK, BIT(0)); + /* Enable status auto-clear on link intr generation */ + atl_set_bits(hw, ATL_INTR_AUTO_CLEAR, BIT(0)); + } else { + /* Enable legacy INTx mode and status clear-on-read */ + atl_write(hw, ATL_INTR_CTRL, BIT(7)); + /* Clear the registers, which might have been set on previous + * driver load and might interfere with legacy IRQ handling + */ + atl_write(hw, ATL_INTR_AUTO_MASK, 0); + atl_write(hw, ATL_INTR_AUTO_CLEAR, 0); + } + + /* Map MCP 4 interrupt to cause 0 */ + atl_write(hw, ATL_INTR_GEN_INTR_MAP4, BIT(7) | (0 << 0)); + + if (hw->chip_id == ATL_ANTIGUA) { + atl_set_bits(hw, ATL_GLOBAL_CTRL2, BIT(3) << 0xA); + atl_write(hw, ATL2_MCP_HOST_REQ_INT_MASK(3), + ATL2_FW_HOST_INTERRUPT_LINK_UP | + ATL2_FW_HOST_INTERRUPT_LINK_DOWN); + } + + atl_write(hw, ATL_TX_INTR_CTRL, BIT(4)); + atl_write(hw, ATL_RX_INTR_CTRL, BIT(3)); + + /* Reset Rx/Tx on unexpected PERST# */ + atl_write_bit(hw, 0x1000, 29, 0); + atl_write(hw, 0x448, 3); +} + +#define atl_vlan_flt_val(vid) ((uint32_t)(vid) | 1 << 16 | 1 << 31) + +void atl_set_rx_mode(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + bool is_multicast_enabled = !!(ndev->flags & IFF_MULTICAST); + bool all_multi_needed = !!(ndev->flags & IFF_ALLMULTI); + bool promisc_needed = !!(ndev->flags & IFF_PROMISC); + int i = nic->rxf_mac.base_index + 1; /* 1 reserved for MAC address */ + int uc_count = netdev_uc_count(ndev); + int mc_count = 0; + struct netdev_hw_addr *hwaddr; + + if (!pm_runtime_active(&nic->hw.pdev->dev)) + return; + + if (is_multicast_enabled) + mc_count = netdev_mc_count(ndev); + + if (uc_count > nic->rxf_mac.available - 1) + promisc_needed = true; + else if (uc_count + mc_count > nic->rxf_mac.available - 1) + all_multi_needed = true; + + /* Enable promisc VLAN mode if IFF_PROMISC explicitly + * requested or too many VIDs registered + */ + atl_set_vlan_promisc(hw, + ndev->flags & IFF_PROMISC || nic->rxf_vlan.promisc_count || + !nic->rxf_vlan.vlans_active); + + atl_set_promisc(hw, promisc_needed, + is_multicast_enabled && all_multi_needed); + if (hw->new_rpf) + atl2_fw_set_filter_policy(hw, promisc_needed, + is_multicast_enabled && all_multi_needed); + + if (promisc_needed) + return; + + netdev_for_each_uc_addr(hwaddr, ndev) + atl_set_uc_flt(hw, i++, hwaddr->addr); + + if (is_multicast_enabled && !all_multi_needed) + netdev_for_each_mc_addr(hwaddr, ndev) + atl_set_uc_flt(hw, i++, hwaddr->addr); + + while (i < ATL_UC_FLT_NUM) + atl_disable_uc_flt(hw, i++); +} + +int atl_alloc_descs(struct atl_nic *nic, struct atl_hw_ring *ring, size_t extra) +{ + struct device *dev = &nic->hw.pdev->dev; + + ring->descs = dma_alloc_coherent(dev, + ring->size * sizeof(*ring->descs) + extra, + &ring->daddr, GFP_KERNEL); + + if (!ring->descs) + return -ENOMEM; + + return 0; +} + +void atl_free_descs(struct atl_nic *nic, struct atl_hw_ring *ring, size_t extra) +{ + struct device *dev = &nic->hw.pdev->dev; + + if (!ring->descs) + return; + + dma_free_coherent(dev, ring->size * sizeof(*ring->descs) + extra, + ring->descs, ring->daddr); + ring->descs = 0; +} + +void atl_set_intr_bits(struct atl_hw *hw, int idx, int rxbit, int txbit) +{ + int shift = idx & 1 ? 0 : 8; + uint32_t clear_mask = 0; + uint32_t set_mask = 0; + uint32_t val; + + if (rxbit >= 0) { + clear_mask |= BIT(7) | (BIT(5) - 1); + if (rxbit < ATL_NUM_MSI_VECS) + set_mask |= BIT(7) | rxbit; + } + if (txbit >= 0) { + clear_mask |= (BIT(7) | (BIT(5) - 1)) << 0x10; + if (txbit < ATL_NUM_MSI_VECS) + set_mask |= (BIT(7) | txbit) << 0x10; + } + + val = atl_read(hw, ATL_INTR_RING_INTR_MAP(idx)); + val &= ~(clear_mask << shift); + val |= set_mask << shift; + atl_write(hw, ATL_INTR_RING_INTR_MAP(idx), val); +} + +void atl_set_loopback(struct atl_nic *nic, int idx, bool on) +{ + struct atl_hw *hw = &nic->hw; + + switch (idx) { + case ATL_PF_LPB_SYS_DMA: + atl_write_bit(hw, ATL_TX_CTRL1, 6, on); + atl_write_bit(hw, ATL_RX_CTRL1, 6, on); + break; + case ATL_PF_LPB_SYS_PB: + atl_write_bit(hw, ATL_TX_CTRL1, 7, on); + atl_write_bit(hw, ATL_RX_CTRL1, 8, on); + break; + case ATL_PF_LPB_INT_PHY: + case ATL_PF_LPB_EXT_PHY: + hw->mcp.ops->set_phy_loopback(nic, idx); + break; + case ATL_PF_LPB_NET_DMA: + /* To switch DMANetworkLoopback mode + * you need a reset datapath + */ + set_bit(ATL_ST_GLOBAL_CONF_NEEDED, &hw->state); + atl_reconfigure(nic); + break; + } +} + +int atl_hwsem_get(struct atl_hw *hw, int idx) +{ + uint32_t val; + + busy_wait(10000, udelay(1), val, atl_read(hw, ATL_MCP_SEM(idx)), !val); + + if (!val) + return -ETIME; + + return 0; +} + +void atl_hwsem_put(struct atl_hw *hw, int idx) +{ + atl_write(hw, ATL_MCP_SEM(idx), 1); +} + +static int atl_msm_wait(struct atl_hw *hw) +{ + uint32_t val; + + busy_wait(10, udelay(10), val, atl_read(hw, ATL_MPI_MSM_ADDR), + val & BIT(12)); + if (val & BIT(12)) { + /* If MSM CLKL off, return ENODATA */ + if ((atl_read(hw, ATL_MPI_MSM_CTRL) & BIT(0xB)) == 0) + return -ENODATA; + + return -ETIME; + } + + return 0; +} + +int __atl_msm_read(struct atl_hw *hw, uint32_t addr, uint32_t *val) +{ + int ret; + + ret = atl_msm_wait(hw); + if (ret) + return ret; + + atl_write(hw, ATL_MPI_MSM_ADDR, (addr >> 2) | BIT(9)); + ret = atl_msm_wait(hw); + if (ret) + return ret; + + *val = atl_read(hw, ATL_MPI_MSM_RD); + return 0; +} + +int atl_msm_read(struct atl_hw *hw, uint32_t addr, uint32_t *val) +{ + int ret; + + /* If MSM CLKL off, fail here */ + if ((atl_read(hw, ATL_MPI_MSM_CTRL) & BIT(0xB)) == 0) + return -ENODATA; + + ret = atl_hwsem_get(hw, ATL_MCP_SEM_MSM); + if (ret) + return -ENODATA; + + ret = __atl_msm_read(hw, addr, val); + atl_hwsem_put(hw, ATL_MCP_SEM_MSM); + + return ret; +} + +int __atl_msm_write(struct atl_hw *hw, uint32_t addr, uint32_t val) +{ + int ret; + + ret = atl_msm_wait(hw); + if (ret) + return ret; + + atl_write(hw, ATL_MPI_MSM_WR, val); + atl_write(hw, ATL_MPI_MSM_ADDR, (addr >> 2) | BIT(8)); + ret = atl_msm_wait(hw); + if (ret) + return ret; + + return 0; +} + +int atl_msm_write(struct atl_hw *hw, uint32_t addr, uint32_t val) +{ + int ret; + + ret = atl_hwsem_get(hw, ATL_MCP_SEM_MSM); + if (ret) + return ret; + + ret = __atl_msm_write(hw, addr, val); + atl_hwsem_put(hw, ATL_MCP_SEM_MSM); + + return ret; +} + +static int atl_mdio_wait(struct atl_hw *hw) +{ + uint32_t val; + + busy_wait(20, udelay(1), val, atl_read(hw, ATL_GLOBAL_MDIO_CMD), + val & BIT(31)); + if (val & BIT(31)) + return -ETIME; + + return 0; +} + +int atl_mdio_hwsem_get(struct atl_hw *hw) +{ + int ret; + + ret = atl_hwsem_get(hw, ATL_MCP_SEM_MDIO); + if (ret) + return ret; + + /* Enable MDIO Clock (active low) in case MBU have disabled + * it. */ + atl_write_bit(hw, ATL_GLOBAL_MDIO_CTL, 14, 0); + return 0; +} + +void atl_mdio_hwsem_put(struct atl_hw *hw) +{ + /* It's ok to leave MDIO Clock running according to FW + * guys. In fact that's what FW does. */ + atl_hwsem_put(hw, ATL_MCP_SEM_MDIO); +} + +static void atl_mdio_set_addr(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr) +{ + /* Set address */ + atl_write(hw, ATL_GLOBAL_MDIO_ADDR, addr & (BIT(16) - 1)); + /* Address operation | execute | prtad + mmd */ + atl_write(hw, ATL_GLOBAL_MDIO_CMD, BIT(15) | 3 << 12 | + prtad << 5 | mmd); +} + +int __atl_mdio_read(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t *val) +{ + int ret; + + ret = atl_mdio_wait(hw); + if (ret) + return ret; + + atl_mdio_set_addr(hw, prtad, mmd, addr); + ret = atl_mdio_wait(hw); + if (ret) + return ret; + + /* Read operation | execute | prtad + mmd */ + atl_write(hw, ATL_GLOBAL_MDIO_CMD, BIT(15) | 1 << 12 | + prtad << 5 | mmd); + + ret = atl_mdio_wait(hw); + if (ret) + return ret; + + *val = atl_read(hw, ATL_GLOBAL_MDIO_RDATA); + return 0; +} + +int atl_mdio_read(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t *val) +{ + int ret; + + ret = atl_mdio_hwsem_get(hw); + if (ret) + return ret; + + ret = __atl_mdio_read(hw, prtad, mmd, addr, val); + atl_mdio_hwsem_put(hw); + + return ret; +} + +int __atl_mdio_write(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t val) +{ + int ret; + + ret = atl_mdio_wait(hw); + if (ret) + return ret; + + atl_mdio_set_addr(hw, prtad, mmd, addr); + ret = atl_mdio_wait(hw); + if (ret) + return ret; + + atl_write(hw, ATL_GLOBAL_MDIO_WDATA, val); + /* Write operation | execute | prtad + mmd */ + atl_write(hw, ATL_GLOBAL_MDIO_CMD, BIT(15) | 2 << 12 | + prtad << 5 | mmd); + ret = atl_mdio_wait(hw); + if (ret) + return ret; + + return 0; +} + +int atl_mdio_write(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t val) +{ + int ret; + + ret = atl_mdio_hwsem_get(hw); + if (ret) + return ret; + + ret = __atl_mdio_write(hw, prtad, mmd, addr, val); + atl_mdio_hwsem_put(hw); + + return 0; +} + +#define __READ_MSM_OR_GOTO(RET, HW, REGISTER, PVARIABLE, label) \ + RET = __atl_msm_read(HW, REGISTER, PVARIABLE); \ + if (RET) \ + goto label; + +void atl_adjust_eth_stats(struct atl_ether_stats *stats, + struct atl_ether_stats *base, bool add) +{ + int i; + uint64_t *_stats = (uint64_t *)stats; + uint64_t *_base = (uint64_t *)base; + + for (i = 0; i < sizeof(*stats) / sizeof(uint64_t); i++) + _stats[i] += add ? _base[i] : - _base[i]; +} + +int atl_update_eth_stats(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + struct atl_ether_stats stats; + uint32_t reg = 0, reg2 = 0; + int ret; + + if (!test_bit(ATL_ST_ENABLED, &nic->hw.state) || + test_bit(ATL_ST_RESETTING, &nic->hw.state)) + return 0; + + memset(&stats, 0, sizeof(stats)); + + atl_lock_fw(hw); + + ret = atl_hwsem_get(hw, ATL_MCP_SEM_MSM); + if (ret) + goto unlock_fw; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_TX_PAUSE, ®, hwsem_put); + stats.tx_pause = reg; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_PAUSE, ®, hwsem_put); + stats.rx_pause = reg; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_OCTETS_LO, ®, hwsem_put); + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_OCTETS_HI, ®2, hwsem_put); + stats.rx_ether_octets = ((uint64_t)reg2 << 32) | reg; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_PKTS_GOOD, ®, hwsem_put); + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_ERRS, ®2, hwsem_put); + stats.rx_ether_pkts = reg + reg2;; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_BROADCAST, ®, hwsem_put); + stats.rx_ether_broacasts = reg; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_MULTICAST, ®, hwsem_put); + stats.rx_ether_multicasts = reg; + + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_FCS_ERRS, ®, hwsem_put); + __READ_MSM_OR_GOTO(ret, hw, ATL_MSM_CTR_RX_ALIGN_ERRS, ®2, hwsem_put); + stats.rx_ether_crc_align_errs = reg + reg2; + + stats.rx_ether_drops = atl_read(hw, ATL_RX_DMA_STATS_CNT7); + + /* capture debug counters*/ + atl_write_bit(hw, ATL_RX_RPF_DBG_CNT_CTRL, 0x1f, 1); + + reg = atl_read(hw, ATL_RX_RPF_HOST_CNT_LO); + reg2 = atl_read(hw, ATL_RX_RPF_HOST_CNT_HI); + stats.rx_filter_host = ((uint64_t)reg2 << 32) | reg; + + reg = atl_read(hw, ATL_RX_RPF_LOST_CNT_LO); + reg2 = atl_read(hw, ATL_RX_RPF_LOST_CNT_HI); + stats.rx_filter_lost = ((uint64_t)reg2 << 32) | reg; + + spin_lock(&nic->stats_lock); + + atl_adjust_eth_stats(&stats, &nic->stats.eth_base, false); + nic->stats.eth = stats; + + spin_unlock(&nic->stats_lock); + + ret = 0; + +hwsem_put: + atl_hwsem_put(hw, ATL_MCP_SEM_MSM); +unlock_fw: + atl_unlock_fw(hw); + return ret; +} +#undef __READ_MSM_OR_GOTO + +int atl_get_lpi_timer(struct atl_nic *nic, uint32_t *lpi_delay) +{ + struct atl_hw *hw = &nic->hw; + uint32_t lpi; + int ret = 0; + + + ret = atl_msm_read(hw, ATL_MSM_TX_LPI_DELAY, &lpi); + if (ret) + return ret; + *lpi_delay = ATL_HW_CLOCK_TO_US(lpi); + + return ret; +} + +static uint32_t atl_mcp_mbox_wait(struct atl_hw *hw, enum mcp_area area, int loops) +{ + uint32_t stat; + + busy_wait(loops, cpu_relax(), stat, + (atl_read(hw, ATL_MCP_SCRATCH(FW2_MBOX_CMD)) & (0xf << 28)), + stat == area); + + return stat; +} + +int atl_write_mcp_mem_b1(struct atl_hw *hw, uint32_t offt, void *host_addr, + size_t size, enum mcp_area area) +{ + uint32_t *addr = (uint32_t *)host_addr; + + if (offt > 0xffff) + return -EINVAL; + + while (size) { + uint32_t stat; + + atl_write(hw, ATL_MCP_SCRATCH(FW2_MBOX_DATA), *addr++); + atl_write(hw, ATL_MCP_SCRATCH(FW2_MBOX_CMD), area | offt); + ndelay(750); + stat = atl_mcp_mbox_wait(hw, area, 5); + + if (stat == area) { + /* Send MCP mbox interrupt */ + atl_set_bits(hw, ATL_GLOBAL_CTRL2, BIT(1)); + ndelay(1200); + stat = atl_mcp_mbox_wait(hw, area, 10000); + } + + if (stat == area) { + atl_dev_err("FW mbox timeout offt %x, remaining %zx\n", + offt, size); + return -ETIME; + } else if (stat != BIT(0x1E)) { + atl_dev_err("FW mbox error status 0x%x, offt 0x%x, remaining %zx\n", + stat, offt, size); + return -EIO; + } + + offt += 4; + size -= 4; + } + + return 0; +} + +int atl_write_mcp_mem_b0(struct atl_hw *hw, uint32_t offt, void *host_addr, + size_t size, enum mcp_area area) +{ + uint32_t *addr = (uint32_t *)host_addr; + + if (offt > 0xffff) + return -EINVAL; + + if (area == MCP_AREA_CONFIG) + offt += hw->mcp.rpc_addr; + else + offt += hw->mcp.fw_settings_addr; + + atl_write(hw, ATL_GLOBAL_MBOX_ADDR, offt); + + while (size) { + uint32_t stat; + + atl_write(hw, ATL_GLOBAL_MBOX_DATA, *addr++); + atl_write(hw, ATL_GLOBAL_MBOX_CTRL, 0xc000); + + busy_wait(100, udelay(10), stat, + atl_read(hw, ATL_GLOBAL_MBOX_CTRL), + stat & BIT(8)); + if (stat & BIT(8)) + return -ETIME; + + size -= 4; + } + + return 0; +} + +int atl_write_mcp_mem(struct atl_hw *hw, uint32_t offt, void *host_addr, + size_t size, enum mcp_area area) +{ + int ret; + + ret = atl_hwsem_get(hw, ATL_MCP_SEM_MEM); + if (ret) + return ret; + + if (hw->chip_rev == 0xb1) + ret = atl_write_mcp_mem_b1(hw, offt, host_addr, size, area); + else + ret = atl_write_mcp_mem_b0(hw, offt, host_addr, size, area); + + atl_hwsem_put(hw, ATL_MCP_SEM_MEM); + return ret; +} + +void atl_thermal_check(struct atl_hw *hw, bool alarm) +{ + int temp, ret; + struct atl_link_state *lstate = &hw->link_state; + struct atl_link_type *link = lstate->link; + int lowest; + + if (link) { + /* ffs() / fls() number bits starting at 1 */ + lowest = ffs(lstate->lp_advertized) - 1; + if (lowest < lstate->lp_lowest) { + lstate->lp_lowest = lowest; + if (lowest < lstate->throttled_to && + lstate->thermal_throttled && alarm) + /* We're still thermal-throttled, and + * just found out we can lower the + * speed even more, so renegotiate. + */ + goto relink; + } + } else + lstate->lp_lowest = fls(lstate->supported) - 1; + + if (alarm == lstate->thermal_throttled) + return; + + lstate->thermal_throttled = alarm; + + ret = hw->mcp.ops->get_phy_temperature(hw, &temp); + if (ret) + temp = 0; + else + /* Temperature is in millidegrees C */ + temp = (temp + 50) / 100; + + if (alarm) { + if (temp) + atl_dev_warn("PHY temperature above threshold: %d.%d\n", + temp / 10, temp % 10); + else + atl_dev_warn("PHY temperature above threshold\n"); + } else { + if (temp) + atl_dev_warn("PHY temperature back in range: %d.%d\n", + temp / 10, temp % 10); + else + atl_dev_warn("PHY temperature back in range\n"); + } + +relink: + if (hw->thermal.flags & atl_thermal_throttle) { + /* If throttling is enabled, renegotiate link */ + lstate->link = 0; + lstate->throttled_to = lstate->lp_lowest; + hw->mcp.ops->set_link(hw, true); + } +} + +/* Atlanic2 new filters implementation */ + +/* set action resolver record */ +static void atl2_rpf_act_rslvr_record_set(struct atl_hw *hw, u8 location, + u32 tag, u32 mask, u32 action) +{ + atl_write(hw, ATL2_RPF_ACT_RSLVR_REQ_TAG(location), tag); + atl_write(hw, ATL2_RPF_ACT_RSLVR_TAG_MASK(location), mask); + atl_write(hw, ATL2_RPF_ACT_RSLVR_ACTN(location), action); +} + +int atl2_act_rslvr_table_set(struct atl_hw *hw, u8 location, + u32 tag, u32 mask, u32 action) +{ + static char action_str[][32] = {"Drop", "Host", "Management", + "Host & Management"}; + static char valid_str[][32] = {"Not Valid", "Valid"}; + static char rss_str[][32] = {"Queue", "TC"}; + int err = 0; + + dev_dbg(&hw->pdev->dev, "ACTRSLVR[%d] TAG %#x MASK %#x ACTION %#x (%s, %s %d, %s)", + location, tag, mask, action, + action_str[(action >> 8) & 3], rss_str[!!(action & BIT(7))], + (action >> 2) & 0x1f, + valid_str[action & 1]); + + err = atl_hwsem_get(hw, ATL2_MCP_SEM_ACT_RSLVR); + if (err) + return err; + + atl2_rpf_act_rslvr_record_set(hw, location, tag, mask, action); + + atl_hwsem_put(hw, ATL2_MCP_SEM_ACT_RSLVR); + + return err; +} + +/** Initialise new rx filters + * L2 promisc OFF + * VLAN promisc OFF + * user custom filtlers + * RSS TC0 + */ +static void atl2_hw_init_new_rx_filters(struct atl_hw *hw) +{ + struct atl_nic *nic = container_of(hw, struct atl_nic, hw); + uint32_t art_last_sec, art_first_sec, art_mask; + int index; + + atl_write_bits(hw, ATL_RX_UC_FLT_REG2(nic->rxf_mac.base_index), + 22, 6, ATL2_RPF_TAG_BASE_UC); + atl_write_bits(hw, ATL2_RX_FLT_L2_BC_TAG, 0, 6, ATL2_RPF_TAG_BASE_BC); + atl_set_bits(hw, ATL2_RPF_L3_FLT(0), BIT(0x17)); + atl_write_bit(hw, ATL_RX_MC_FLT_MSK, 14, 1); + + art_last_sec = hw->art_base_index / 8 + hw->art_available / 8; + art_first_sec = hw->art_base_index / 8; + art_mask = (BIT(art_last_sec) - 1) - (BIT(art_first_sec) - 1); + atl_set_bits(hw, ATL2_RPF_REC_TAB_EN, art_mask); + + index = hw->art_base_index + ATL2_RPF_L2_PROMISC_OFF_INDEX; + atl2_act_rslvr_table_set(hw, + index, + 0, + ATL2_RPF_TAG_UC_MASK | ATL2_RPF_TAG_ALLMC_MASK, + ATL2_ACTION_DROP); + + index = hw->art_base_index + ATL2_RPF_VLAN_PROMISC_OFF_INDEX; + atl2_act_rslvr_table_set(hw, + index, + 0, + ATL2_RPF_TAG_VLAN_MASK | ATL2_RPF_TAG_UNTAG_MASK, + ATL2_ACTION_DROP); + + index = hw->art_base_index + ATL2_RPF_DEFAULT_RULE_INDEX; + atl2_act_rslvr_table_set(hw, + index, + 0, + 0, + ATL2_ACTION_ASSIGN_TC(0)); +} + +static void atl2_hw_new_rx_filter_vlan_promisc(struct atl_hw *hw, bool promisc) +{ + u16 off_action = !promisc ? ATL2_ACTION_DROP : ATL2_ACTION_DISABLE; + int index = hw->art_base_index + ATL2_RPF_VLAN_PROMISC_OFF_INDEX; + + atl2_act_rslvr_table_set(hw, + index, + 0, + ATL2_RPF_TAG_VLAN_MASK | ATL2_RPF_TAG_UNTAG_MASK, + off_action); +} + +static void atl2_hw_new_rx_filter_promisc(struct atl_hw *hw, bool promisc, + bool allmulti) +{ + u16 off_action = promisc ? ATL2_ACTION_DISABLE : ATL2_ACTION_DROP; + u32 mask = allmulti ? (ATL2_RPF_TAG_UC_MASK | ATL2_RPF_TAG_ALLMC_MASK) : + ATL2_RPF_TAG_UC_MASK; + int index = hw->art_base_index + ATL2_RPF_L2_PROMISC_OFF_INDEX; + + atl2_act_rslvr_table_set(hw, + index, + 0, + mask, + off_action); + +} + diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.h new file mode 100644 index 000000000000..b8aa51c53a1a --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw.h @@ -0,0 +1,382 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_HW_H_ +#define _ATL_HW_H_ + +#include +#include +#include +#include + +#include "atl_fw.h" +#include "atl_macsec.h" +#include "atl_regs.h" + +union atl_desc; +struct atl_nic; + +#define PCI_VENDOR_ID_AQUANTIA 0x1d6a + +/* clock is 3.2 ns*/ +#define ATL_HW_CLOCK_TO_US(clk) (clk * 32 / 10000) + +#define ATL2_ACTION(ACTION, RSS, INDEX, VALID, TS_VALID) \ + ((((ACTION & 0x3U) << 8) | \ + ((RSS & 0x1U) << 7) | \ + ((INDEX & 0x3FU) << 2) | \ + ((TS_VALID & 0x1U) << 1)) | \ + ((VALID & 0x1U) << 0)) + +#define ATL2_ACTION_DROP ATL2_ACTION(0, 0, 0, 1, 0) +#define ATL2_ACTION_DISABLE ATL2_ACTION(0, 0, 0, 0, 0) +#define ATL2_ACTION_ASSIGN_QUEUE(QUEUE) ATL2_ACTION(1, 0, (QUEUE), 1, 0) +#define ATL2_ACTION_ASSIGN_TC(TC) ATL2_ACTION(1, 1, (TC), 1, 0) + +enum { + ATL2_RPF_L2_PROMISC_OFF_INDEX = 0, + ATL2_RPF_VLAN_PROMISC_OFF_INDEX, + ATL2_RPF_L3L4_USER_INDEX, + ATL2_RPF_ET_PCP_USER_INDEX = ATL2_RPF_L3L4_USER_INDEX + 16, + ATL2_RPF_VLAN_USER_INDEX = ATL2_RPF_ET_PCP_USER_INDEX + 16, + ATL2_RPF_FLEX_USER_INDEX = ATL2_RPF_VLAN_USER_INDEX + 16, + ATL2_RPF_DEFAULT_RULE_INDEX = ATL2_RPF_FLEX_USER_INDEX + 1, +}; + +#define ATL2_RPF_TAG_UC_OFFSET 0x0 +#define ATL2_RPF_TAG_ALLMC_OFFSET 0x6 +#define ATL2_RPF_TAG_ET_OFFSET 0x7 +#define ATL2_RPF_TAG_VLAN_OFFSET 0xA +#define ATL2_RPF_TAG_UNTAG_OFFSET 0xE +#define ATL2_RPF_TAG_L3_V4_OFFSET 0xF +#define ATL2_RPF_TAG_L3_V6_OFFSET 0x12 +#define ATL2_RPF_TAG_L4_OFFSET 0x15 +#define ATL2_RPF_TAG_L4_FLEX_OFFSET 0x18 +#define ATL2_RPF_TAG_FLEX_OFFSET 0x1B +#define ATL2_RPF_TAG_PCP_OFFSET 0x1D + +#define ATL2_RPF_TAG_UC_MASK (0x0000003F << ATL2_RPF_TAG_UC_OFFSET) +#define ATL2_RPF_TAG_ALLMC_MASK (0x00000001 << ATL2_RPF_TAG_ALLMC_OFFSET) +#define ATL2_RPF_TAG_UNTAG_MASK (0x00000001 << ATL2_RPF_TAG_UNTAG_OFFSET) +#define ATL2_RPF_TAG_VLAN_MASK (0x0000000F << ATL2_RPF_TAG_VLAN_OFFSET) +#define ATL2_RPF_TAG_ET_MASK (0x00000007 << ATL2_RPF_TAG_ET_OFFSET) +#define ATL2_RPF_TAG_L3_V4_MASK (0x00000007 << ATL2_RPF_TAG_L3_V4_OFFSET) +#define ATL2_RPF_TAG_L3_V6_MASK (0x00000007 << ATL2_RPF_TAG_L3_V6_OFFSET) +#define ATL2_RPF_TAG_L4_MASK (0x00000007 << ATL2_RPF_TAG_L4_OFFSET) +#define ATL2_RPF_TAG_FLEX_MASK (0x00000003 << ATL2_RPF_TAG_FLEX_OFFSET) +#define ATL2_RPF_TAG_PCP_MASK (0x00000007 << ATL2_RPF_TAG_PCP_OFFSET) + +#define ATL2_RPF_TAG_BASE_BC (1 << ATL2_RPF_TAG_UC_OFFSET) +#define ATL2_RPF_TAG_BASE_UC (2 << ATL2_RPF_TAG_UC_OFFSET) + +#define ATL2_FW_HOSTLOAD_REQ_LEN_MAX 0x1000 + +#define busy_wait(tries, wait, lvalue, fetch, cond) \ +({ \ + uint32_t _dummy = 0; \ + int i = (tries); \ + int orig = i; \ + (void)_dummy; \ + do { \ + wait; \ + (lvalue) = (fetch); \ + } while ((cond) && --i); \ + (orig - i); \ +}) + +enum atl_chip { + ATL_UNKNOWN, + ATL_ATLANTIC, + ATL_ANTIGUA, +}; + +struct atl_thermal { + unsigned flags; + uint8_t crit; + uint8_t high; + uint8_t low; +}; + +extern struct atl_thermal atl_def_thermal; + +enum atl_nic_state { + ATL_ST_ENABLED, + ATL_ST_CONFIGURED, + ATL_ST_RINGS_RUNNING, + /* ATL_ST_FWD_RINGS_RUNNING, */ + ATL_ST_UP, + ATL_ST_WORK_SCHED, + ATL_ST_UPDATE_LINK, + ATL_ST_RESETTING, + ATL_ST_RESET_NEEDED, + ATL_ST_GLOBAL_CONF_NEEDED, + ATL_ST_START_NEEDED, + ATL_ST_DETACHED, +}; + +#define ATL_WAKE_SUPPORTED (WAKE_MAGIC | WAKE_PHY) +struct atl_hw { + atomic_t flags; + uint8_t __iomem *regs; + struct pci_dev *pdev; + unsigned long state; + enum atl_chip chip_id; + u32 chip_rev; + bool new_rpf; + struct atl_link_state link_state; + uint32_t lpi_timer; + unsigned wol_mode; + struct atl_mcp mcp; + uint32_t non_ring_intr_mask; + uint8_t mac_addr[ETH_ALEN]; +#define ATL_RSS_KEY_SIZE 40 + uint8_t rss_key[ATL_RSS_KEY_SIZE]; +#define ATL_RSS_TBL_SIZE (1 << 6) + uint8_t rss_tbl[ATL_RSS_TBL_SIZE]; + struct atl_thermal thermal; +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + struct atl_macsec_cfg macsec_cfg; +#endif + s64 ptp_clk_offset; + int art_base_index; + int art_available; +}; + +struct atl_hw_ring { + union atl_desc *descs; + uint32_t size; + uint32_t reg_base; + dma_addr_t daddr; +}; + +enum mcp_area { + MCP_AREA_CONFIG = 0x80000000, + MCP_AREA_SETTINGS = 0x20000000, +}; + +#define offset_ptr(ptr, ring, amount) \ + ({ \ + struct atl_hw_ring *hw_ring = (ring); \ + uint32_t size = hw_ring->size; \ + \ + uint32_t res = (ptr) + (amount); \ + if ((int32_t)res < 0) \ + res += size; \ + else if (res >= size) \ + res -= size; \ + res; \ + }) + +void atl_check_unplug(struct atl_hw *hw, uint32_t addr); + +static inline uint32_t atl_read(struct atl_hw *hw, uint32_t addr) +{ + uint8_t __iomem *base = READ_ONCE(hw->regs); + uint32_t val = 0xffffffff; + + if (unlikely(!base)) + return val; + + val = readl(base + addr); + if (unlikely(val == 0xffffffff)) + atl_check_unplug(hw, addr); + return val; +} + +static inline void atl_write(struct atl_hw *hw, uint32_t addr, uint32_t val) +{ + uint8_t __iomem *base = READ_ONCE(hw->regs); + + if (unlikely(!base)) + return; + + writel(val, base + addr); +} + + +static inline void atl_write_mask_bits(struct atl_hw *hw, uint32_t addr, + uint32_t mask, uint32_t val) +{ + atl_write(hw, addr, + (atl_read(hw, addr) & ~mask) | (val & mask)); +} + +static inline void atl_write_bits(struct atl_hw *hw, uint32_t addr, + uint32_t shift, uint32_t width, uint32_t val) +{ + uint32_t mask = ((1u << width) - 1) << shift; + + atl_write(hw, addr, + (atl_read(hw, addr) & ~mask) | ((val << shift) & mask)); +} + +static inline void atl_write_bit(struct atl_hw *hw, uint32_t addr, + uint32_t shift, uint32_t val) +{ + atl_write_bits(hw, addr, shift, 1, val); +} + +static inline void atl_set_bits(struct atl_hw *hw, uint32_t addr, + uint32_t bits) +{ + atl_write(hw, addr, atl_read(hw, addr) | bits); +} + +static inline void atl_clear_bits(struct atl_hw *hw, uint32_t addr, + uint32_t bits) +{ + atl_write(hw, addr, atl_read(hw, addr) & ~bits); +} + +static inline void atl_intr_enable(struct atl_hw *hw, uint32_t mask) +{ + atl_write(hw, ATL_INTR_MSK_SET, mask); +} + +static inline void atl_intr_disable(struct atl_hw *hw, uint32_t mask) +{ + atl_write(hw, ATL_INTR_MSK_CLEAR, mask); +} + +static inline void atl_intr_disable_all(struct atl_hw *hw) +{ + atl_intr_disable(hw, 0xffffffff); +} + +static inline unsigned atl_fw_major(struct atl_hw *hw) +{ + return (hw->mcp.fw_rev >> 24) & 0xff; +} + +static inline void atl_init_rss_table(struct atl_hw *hw, int nvecs) +{ + int i; + + for (i = 0; i < ATL_RSS_TBL_SIZE; i++) + hw->rss_tbl[i] = i % nvecs; +} + +int atl2_act_rslvr_table_set(struct atl_hw *hw, u8 location, + u32 tag, u32 mask, u32 action); +static inline void atl2_rpf_vlan_flr_tag_set(struct atl_hw *hw, u32 tag, + u32 filter) +{ + atl_write_bits(hw, ATL_RX_VLAN_FLT(filter), 12, 4, tag); +} + +static inline void atl2_rpf_etht_flr_tag_set(struct atl_hw *hw, u32 tag, + u32 filter) +{ + atl_write_bits(hw, ATL2_RX_ETYPE_TAG(filter), 0, 3, tag); +} + +static inline void atl2_rpf_l3_v4_da_set(struct atl_hw *hw, u32 filter, u32 val) +{ + u32 dword = filter % 4; + u32 addr_set = 6 + ((filter < 4) ? (0) : (1)); + + atl_write(hw, ATL2_RPF_L3_DA(addr_set) + 4 * dword, swab32(val)); +} + +static inline void atl2_rpf_l3_v4_sa_set(struct atl_hw *hw, u32 filter, u32 val) +{ + u32 dword = filter % 4; + u32 addr_set = 6 + ((filter < 4) ? (0) : (1)); + + atl_write(hw, ATL2_RPF_L3_SA(addr_set) + 4 * dword, swab32(val)); +} + +static inline void atl2_rpf_l3_v6_sa_set(struct atl_hw *hw, u32 filter, + u32 val[4]) +{ + int i; + + for (i = 0; i < 4; i++) + atl_write(hw, ATL2_RPF_L3_SA(filter) + 4 * i, swab32(val[i])); +} + +static inline void atl2_rpf_l3_v6_da_set(struct atl_hw *hw, u32 filter, + u32 val[4]) +{ + int i; + + for (i = 0; i < 4; i++) + atl_write(hw, ATL2_RPF_L3_DA(filter) + 4 * i, swab32(val[i])); +} + +static inline void atl2_rpf_flex_flr_tag_set(struct atl_hw *hw, u32 tag, + u32 filter) +{ + atl_write_bits(hw, ATL_RX_FLEX_FLT_CTRL(filter), 0x19, 2, tag); +} + +int atl_read_mcp_mem(struct atl_hw *hw, uint32_t mcp_addr, void *host_addr, + unsigned size); +int atl_hwinit(struct atl_hw *hw, enum atl_chip chip_id); +void atl_refresh_link(struct atl_nic *nic); +void atl_set_rss_key(struct atl_hw *hw); +int atl_set_rss_tbl(struct atl_hw *hw); +void atl_set_uc_flt(struct atl_hw *hw, int idx, uint8_t mac_addr[ETH_ALEN]); + +int atl_alloc_descs(struct atl_nic *nic, struct atl_hw_ring *ring, size_t extra); +void atl_free_descs(struct atl_nic *nic, struct atl_hw_ring *ring, size_t extra); +void atl_set_intr_bits(struct atl_hw *hw, int idx, int rxbit, int txbit); +int atl_alloc_link_intr(struct atl_nic *nic); +void atl_free_link_intr(struct atl_nic *nic); +int atl_write_mcp_mem(struct atl_hw *hw, uint32_t offt, void *addr, + size_t size, enum mcp_area area); + +static inline int atl_write_fwcfg_word(struct atl_hw *hw, uint32_t offt, + uint32_t val) +{ + return atl_write_mcp_mem(hw, offt, &val, sizeof(val), MCP_AREA_CONFIG); +} + +static inline int atl_write_fwsettings_word(struct atl_hw *hw, uint32_t offt, + uint32_t val) +{ + return atl_write_mcp_mem(hw, offt, &val, sizeof(val), MCP_AREA_SETTINGS); +} + +static inline int atl_read_fwstat_word(struct atl_hw *hw, uint32_t offt, + uint32_t *val) +{ + return atl_read_mcp_word(hw, offt + hw->mcp.fw_stat_addr, val); +} + +static inline int atl_read_rpc_mem(struct atl_hw *hw, uint32_t offt, + uint32_t *val, size_t length) +{ + return atl_read_mcp_mem(hw, offt + hw->mcp.rpc_addr, val, length); +} + +static inline int atl_read_fwsettings_word(struct atl_hw *hw, uint32_t offt, + uint32_t *val) +{ + if (offt >= hw->mcp.fw_settings_len) + return -EINVAL; + + return atl_read_mcp_word(hw, offt + hw->mcp.fw_settings_addr, val); +} + +static inline void atl_lock_fw(struct atl_hw *hw) +{ + mutex_lock(&hw->mcp.lock); +} + +static inline void atl_unlock_fw(struct atl_hw *hw) +{ + mutex_unlock(&hw->mcp.lock); +} + +void atl_fw_watchdog(struct atl_hw *hw); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.c new file mode 100644 index 000000000000..fd2f0bd549b7 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.c @@ -0,0 +1,247 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include + +#include "atl_hw_ptp.h" +#include "atl_desc.h" +#include "atl_mdio.h" + +#define FRAC_PER_NS 0x100000000LL + +#define ATL_HW_MAC_COUNTER_HZ 312500000ll +#define ATL_HW_PHY_COUNTER_HZ 160000000ll + +/* register address for bitfield PTP Digital Clock Read Enable */ +#define HW_ATL_PCS_PTP_CLOCK_READ_ENABLE_ADR 0x00004628 +/* lower bit position of bitfield PTP Digital Clock Read Enable */ +#define HW_ATL_PCS_PTP_CLOCK_READ_ENABLE_SHIFT 4 +/* width of bitfield PTP Digital Clock Read Enable */ +#define HW_ATL_PCS_PTP_CLOCK_READ_ENABLE_WIDTH 1 + +/* register address for ptp counter reading */ +#define HW_ATL_PCS_PTP_TS_VAL_ADDR(index) (0x00004900 + (index) * 0x4) + +static void hw_atl_pcs_ptp_clock_read_enable(struct atl_hw *hw, + u32 ptp_clock_read_enable) +{ + atl_write_bits(hw, HW_ATL_PCS_PTP_CLOCK_READ_ENABLE_ADR, + HW_ATL_PCS_PTP_CLOCK_READ_ENABLE_SHIFT, + HW_ATL_PCS_PTP_CLOCK_READ_ENABLE_WIDTH, + ptp_clock_read_enable); +} + +static u32 hw_atl_pcs_ptp_clock_get(struct atl_hw *hw, u32 index) +{ + return atl_read(hw, HW_ATL_PCS_PTP_TS_VAL_ADDR(index)); +} + +#define get_ptp_ts_val_u64(self, indx) \ + ((u64)(hw_atl_pcs_ptp_clock_get(self, indx) & 0xffff)) + +void hw_atl_get_ptp_ts(struct atl_hw *hw, u64 *stamp) +{ + u64 ns; + + hw_atl_pcs_ptp_clock_read_enable(hw, 1); + hw_atl_pcs_ptp_clock_read_enable(hw, 0); + ns = (get_ptp_ts_val_u64(hw, 0) + + (get_ptp_ts_val_u64(hw, 1) << 16)) * NSEC_PER_SEC + + (get_ptp_ts_val_u64(hw, 3) + + (get_ptp_ts_val_u64(hw, 4) << 16)); + + *stamp = ns + hw->ptp_clk_offset; +} + +static void hw_atl_adj_params_get(u64 freq, s64 adj, u32 *ns, u32 *fns) +{ + /* For accuracy, the digit is extended */ + s64 base_ns = ((adj + NSEC_PER_SEC) * NSEC_PER_SEC); + u64 nsi_frac = 0; + u64 nsi; + + base_ns = div64_s64(base_ns, freq); + nsi = div64_u64(base_ns, NSEC_PER_SEC); + + if (base_ns != nsi * NSEC_PER_SEC) { + s64 divisor = div64_s64((s64)NSEC_PER_SEC * NSEC_PER_SEC, + base_ns - nsi * NSEC_PER_SEC); + nsi_frac = div64_s64(FRAC_PER_NS * NSEC_PER_SEC, divisor); + } + + *ns = (u32)nsi; + *fns = (u32)nsi_frac; +} + +static void +hw_atl_mac_adj_param_calc(struct ptp_adj_freq *ptp_adj_freq, u64 phyfreq, + u64 macfreq) +{ + s64 adj_fns_val; + s64 fns_in_sec_phy = phyfreq * (ptp_adj_freq->fns_phy + + FRAC_PER_NS * ptp_adj_freq->ns_phy); + s64 fns_in_sec_mac = macfreq * (ptp_adj_freq->fns_mac + + FRAC_PER_NS * ptp_adj_freq->ns_mac); + s64 fault_in_sec_phy = FRAC_PER_NS * NSEC_PER_SEC - fns_in_sec_phy; + s64 fault_in_sec_mac = FRAC_PER_NS * NSEC_PER_SEC - fns_in_sec_mac; + /* MAC MCP counter freq is macfreq / 4 */ + s64 diff_in_mcp_overflow = (fault_in_sec_mac - fault_in_sec_phy) * + 4 * FRAC_PER_NS; + + diff_in_mcp_overflow = div64_s64(diff_in_mcp_overflow, + ATL_HW_MAC_COUNTER_HZ); + adj_fns_val = (ptp_adj_freq->fns_mac + FRAC_PER_NS * + ptp_adj_freq->ns_mac) + diff_in_mcp_overflow; + + ptp_adj_freq->mac_ns_adj = div64_s64(adj_fns_val, FRAC_PER_NS); + ptp_adj_freq->mac_fns_adj = adj_fns_val - ptp_adj_freq->mac_ns_adj * + FRAC_PER_NS; +} + +int hw_atl_adj_sys_clock(struct atl_hw *hw, s64 delta) +{ + hw->ptp_clk_offset += delta; + + atl_write(hw, ATL_RX_SPARE_CTRL0, lower_32_bits(hw->ptp_clk_offset)); + atl_write(hw, ATL_RX_SPARE_CTRL1, upper_32_bits(hw->ptp_clk_offset)); + + return 0; +} + +int hw_atl_ts_to_sys_clock(struct atl_hw *hw, u64 ts, u64 *time) +{ + *time = hw->ptp_clk_offset + ts; + return 0; +} + +int hw_atl_adj_clock_freq(struct atl_hw *hw, s32 ppb) +{ + struct ptp_msg_fw_request fwreq; + struct atl_mcp *mcp = &hw->mcp; + + memset(&fwreq, 0, sizeof(fwreq)); + + fwreq.msg_id = ptp_adj_freq_msg; + hw_atl_adj_params_get(ATL_HW_MAC_COUNTER_HZ, ppb, + &fwreq.adj_freq.ns_mac, + &fwreq.adj_freq.fns_mac); + hw_atl_adj_params_get(ATL_HW_PHY_COUNTER_HZ, ppb, + &fwreq.adj_freq.ns_phy, + &fwreq.adj_freq.fns_phy); + hw_atl_mac_adj_param_calc(&fwreq.adj_freq, + ATL_HW_PHY_COUNTER_HZ, + ATL_HW_MAC_COUNTER_HZ); + + return mcp->ops->send_ptp_req(hw, &fwreq); +} + +int hw_atl_gpio_pulse(struct atl_hw *hw, u32 index, u64 start, u32 period) +{ + struct ptp_msg_fw_request fwreq; + struct atl_mcp *mcp = &hw->mcp; + + memset(&fwreq, 0, sizeof(fwreq)); + + fwreq.msg_id = ptp_gpio_ctrl_msg; + fwreq.gpio_ctrl.index = index; + fwreq.gpio_ctrl.period = period; + /* Apply time offset */ + fwreq.gpio_ctrl.start = start; + + return mcp->ops->send_ptp_req(hw, &fwreq); +} + +int hw_atl_extts_gpio_enable(struct atl_hw *hw, u32 index, u32 enable) +{ + /* Enable/disable Sync1588 GPIO Timestamping */ + return atl_mdio_write(hw, 0, MDIO_MMD_PCS, 0xc611, enable ? 0x71 : 0); +} + +int hw_atl_get_sync_ts(struct atl_hw *hw, u64 *ts) +{ + u16 nsec_l = 0; + u16 nsec_h = 0; + u16 sec_l = 0; + u16 sec_h = 0; + int ret; + + if (!ts) + return -1; + + /* PTP external GPIO clock seconds count 15:0 */ + ret = atl_mdio_read(hw, 0, MDIO_MMD_PCS, 0xc914, &sec_l); + /* PTP external GPIO clock seconds count 31:16 */ + if (!ret) + ret = atl_mdio_read(hw, 0, MDIO_MMD_PCS, 0xc915, &sec_h); + /* PTP external GPIO clock nanoseconds count 15:0 */ + if (!ret) + ret = atl_mdio_read(hw, 0, MDIO_MMD_PCS, 0xc916, &nsec_l); + /* PTP external GPIO clock nanoseconds count 31:16 */ + if (!ret) + ret = atl_mdio_read(hw, 0, MDIO_MMD_PCS, 0xc917, &nsec_h); + + *ts = ((u64)nsec_h << 16) + nsec_l + (((u64)sec_h << 16) + sec_l) * NSEC_PER_SEC; + + return ret; +} + +u16 hw_atl_rx_extract_ts(struct atl_hw *hw, u8 *p, unsigned int len, + u64 *timestamp) +{ + unsigned int offset = 14; + struct ethhdr *eth; + u64 sec; + u8 *ptr; + u32 ns; + + if (len <= offset || !timestamp) + return 0; + + /* The TIMESTAMP in the end of package has following format: + * (big-endian) + * struct { + * uint64_t sec; + * uint32_t ns; + * uint16_t stream_id; + * }; + */ + ptr = p + (len - offset); + memcpy(&sec, ptr, sizeof(sec)); + ptr += sizeof(sec); + memcpy(&ns, ptr, sizeof(ns)); + + sec = be64_to_cpu(sec) & 0xffffffffffffllu; + ns = be32_to_cpu(ns); + *timestamp = sec * NSEC_PER_SEC + ns + hw->ptp_clk_offset; + + eth = (struct ethhdr *)p; + + return (eth->h_proto == htons(ETH_P_1588)) ? 12 : 14; +} + +void hw_atl_extract_hwts(struct atl_hw *hw, struct atl_rx_desc_hwts_wb *hwts_wb, + u64 *timestamp) +{ + u64 tmp, sec, ns; + + sec = 0; + tmp = hwts_wb->sec_lw0 & 0x3ff; + sec += tmp; + tmp = (u64)((hwts_wb->sec_lw1 >> 16) & 0xffff) << 10; + sec += tmp; + tmp = (u64)(hwts_wb->sec_hw & 0xfff) << 26; + sec += tmp; + tmp = (u64)((hwts_wb->sec_hw >> 22) & 0x3ff) << 38; + sec += tmp; + ns = sec * NSEC_PER_SEC + hwts_wb->ns; + if (timestamp) + *timestamp = ns + hw->ptp_clk_offset; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.h new file mode 100644 index 000000000000..04e31370bc13 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hw_ptp.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_HW_PTP_H_ +#define _ATL_HW_PTP_H_ + +#include "atl_hw.h" + +struct atl_rx_desc_hwts_wb; + +void hw_atl_get_ptp_ts(struct atl_hw *hw, u64 *stamp); +int hw_atl_adj_sys_clock(struct atl_hw *hw, s64 delta); +int hw_atl_ts_to_sys_clock(struct atl_hw *hw, u64 ts, u64 *time); +int hw_atl_adj_clock_freq(struct atl_hw *hw, s32 ppb); +int hw_atl_gpio_pulse(struct atl_hw *hw, u32 index, u64 start, u32 period); +int hw_atl_extts_gpio_enable(struct atl_hw *hw, u32 index, u32 enable); +int hw_atl_get_sync_ts(struct atl_hw *hw, u64 *ts); +u16 hw_atl_rx_extract_ts(struct atl_hw *hw, u8 *p, unsigned int len, + u64 *timestamp); +void hw_atl_extract_hwts(struct atl_hw *hw, struct atl_rx_desc_hwts_wb *hwts_wb, + u64 *timestamp); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hwmon.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hwmon.c new file mode 100644 index 000000000000..103d2552de22 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_hwmon.c @@ -0,0 +1,214 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2018 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_common.h" +#include +#include + +#if LINUX_VERSION_CODE > KERNEL_VERSION(4,10,0) + +static ssize_t atl_hwmon_set_flag(struct device *dev, + struct device_attribute *attr, const char *buf, size_t size) +{ + struct sensor_device_attribute *sattr = to_sensor_dev_attr(attr); + struct atl_hw *hw = dev_get_drvdata(dev); + bool val; + int ret; + + if (strtobool(buf, &val) < 0) + return -EINVAL; + + ret = atl_update_thermal_flag(hw, sattr->index, val); + + return ret ?: size; +} + +static ssize_t atl_hwmon_show_flag(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct sensor_device_attribute *sattr = to_sensor_dev_attr(attr); + struct atl_hw *hw = dev_get_drvdata(dev); + + return snprintf(buf, PAGE_SIZE, "%d\n", + !!(hw->thermal.flags & BIT(sattr->index))); +} + +#define ATL_HWMON_BIT_ATTR(_name, _bit) \ + SENSOR_DEVICE_ATTR(_name, S_IRUGO | S_IWUSR, atl_hwmon_show_flag, \ + atl_hwmon_set_flag, _bit) + +static ATL_HWMON_BIT_ATTR(monitor, atl_thermal_monitor_shift); +static ATL_HWMON_BIT_ATTR(throttle, atl_thermal_throttle_shift); +static ATL_HWMON_BIT_ATTR(ignore_lims, atl_thermal_ignore_lims_shift); + +static struct attribute *atl_hwmon_attrs[] = { + &sensor_dev_attr_monitor.dev_attr.attr, + &sensor_dev_attr_throttle.dev_attr.attr, + &sensor_dev_attr_ignore_lims.dev_attr.attr, + NULL, +}; +ATTRIBUTE_GROUPS(atl_hwmon); + +static char *atl_hwmon_labels[] = { + "PHY Temperature", +}; + +static const uint32_t atl_hwmon_temp_config[] = { + HWMON_T_INPUT | HWMON_T_LABEL | HWMON_T_CRIT | HWMON_T_MAX | + HWMON_T_MAX_HYST | HWMON_T_MAX_ALARM, + 0, +}; + +static const struct hwmon_channel_info atl_hwmon_temp = { + .type = hwmon_temp, + .config = atl_hwmon_temp_config, +}; + +static const struct hwmon_channel_info *atl_hwmon_info[] = { + &atl_hwmon_temp, + NULL, +}; + +static umode_t atl_hwmon_is_visible(const void *p, + enum hwmon_sensor_types type, uint32_t attr, int channel) +{ + if (type != hwmon_temp) + return 0; + + switch (attr) { + case hwmon_temp_input: case hwmon_temp_max_alarm: case hwmon_temp_label: + return S_IRUGO; + + case hwmon_temp_crit: case hwmon_temp_max: case hwmon_temp_max_hyst: + return S_IRUGO | S_IWUSR; + } + + return 0; +} + +static int atl_hwmon_read(struct device *dev, enum hwmon_sensor_types type, + uint32_t attr, int channel, long *val) +{ + struct atl_hw *hw = dev_get_drvdata(dev); + int temp, ret; + + if (type != hwmon_temp) + return -EINVAL; + + switch (attr) { + case hwmon_temp_input: + ret = hw->mcp.ops->get_phy_temperature(hw, &temp); + if (ret) + return ret; + + *val = temp; + break; + + case hwmon_temp_crit: + *val = hw->thermal.crit * 1000; + break; + + case hwmon_temp_max: + *val = hw->thermal.high * 1000; + break; + + case hwmon_temp_max_hyst: + *val = hw->thermal.low * 1000; + break; + + case hwmon_temp_max_alarm: + *val = hw->link_state.thermal_throttled; + break; + + default: + return -EINVAL; + } + return 0; +} + +static int atl_hwmon_write(struct device *dev, enum hwmon_sensor_types type, + uint32_t attr, int channel, long val) +{ + struct atl_hw *hw = dev_get_drvdata(dev); + uint8_t *ptr, old; + int ret; + + if (type != hwmon_temp) + return -EINVAL; + + switch (attr) { + case hwmon_temp_crit: + ptr = &hw->thermal.crit; + break; + + case hwmon_temp_max: + ptr = &hw->thermal.high; + break; + + case hwmon_temp_max_hyst: + ptr = &hw->thermal.low; + break; + + default: + return -EINVAL; + } + + old = *ptr; + *ptr = val / 1000; + + ret = atl_update_thermal(hw); + if (ret) + *ptr = old; + + return ret; +} + +static int atl_hwmon_read_string(struct device *dev, + enum hwmon_sensor_types type, u32 attr, int channel, const char **str) +{ + if (type != hwmon_temp || attr != hwmon_temp_label) + return -EINVAL; + + *str = atl_hwmon_labels[channel]; + return 0; +} + +static const struct hwmon_ops atl_hwmon_ops = { + .is_visible = atl_hwmon_is_visible, + .read = atl_hwmon_read, + .read_string = atl_hwmon_read_string, + .write = atl_hwmon_write, +}; + +static const struct hwmon_chip_info atl_hwmon = { + .ops = &atl_hwmon_ops, + .info = atl_hwmon_info, +}; + +int atl_hwmon_init(struct atl_nic *nic) +{ + struct device *hwmon_dev; + struct atl_hw *hw = &nic->hw; + + hwmon_dev = devm_hwmon_device_register_with_info(&hw->pdev->dev, + nic->ndev->name, hw, &atl_hwmon, atl_hwmon_groups); + + return PTR_ERR_OR_ZERO(hwmon_dev); +} + +#else + +int atl_hwmon_init(struct atl_nic *nic) +{ + return 0; +} + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_log.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_log.h new file mode 100644 index 000000000000..3f158f5a746b --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_log.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_LOG_H_ +#define _ATL_LOG_H_ + +#ifdef __KERNEL__ + +#include + +/* Logging conveniency macros. + * + * atl_dev_xxx are for low-level contexts and implicitly reference + * struct atl_hw *hw; + * + * atl_nic_xxx are for high-level contexts and implicitly reference + * struct atl_nic *nic; + */ +#define atl_dev_dbg(fmt, args...) \ + dev_dbg(&hw->pdev->dev, fmt, ## args) +#define atl_dev_info(fmt, args...) \ + dev_info(&hw->pdev->dev, fmt, ## args) +#define atl_dev_warn(fmt, args...) \ + dev_warn(&hw->pdev->dev, fmt, ## args) +#define atl_dev_err(fmt, args...) \ + dev_err(&hw->pdev->dev, fmt, ## args) + +#define atl_nic_dbg(fmt, args...) \ + dev_dbg(&nic->hw.pdev->dev, fmt, ## args) +#define atl_nic_info(fmt, args...) \ + dev_info(&nic->hw.pdev->dev, fmt, ## args) +#define atl_nic_warn(fmt, args...) \ + dev_warn(&nic->hw.pdev->dev, fmt, ## args) +#define atl_nic_err(fmt, args...) \ + dev_err(&nic->hw.pdev->dev, fmt, ## args) + +#define atl_dev_init_warn(fmt, args...) \ +do { \ + if (hw) \ + atl_dev_warn(fmt, ## args); \ + else \ + printk(KERN_WARNING "%s: " fmt, atl_driver_name, ##args); \ +} while (0) + +#define atl_dev_init_err(fmt, args...) \ +do { \ + if (hw) \ + atl_dev_err(fmt, ## args); \ + else \ + printk(KERN_ERR "%s: " fmt, atl_driver_name, ##args); \ +} while (0) + +#else /* __KERNEL__ */ + +/* Logging is disabled in case of unit tests */ +#define atl_dev_dbg(fmt, args...) +#define atl_dev_info(fmt, args...) +#define atl_dev_warn(fmt, args...) +#define atl_dev_err(fmt, args...) +#define atl_nic_dbg(fmt, args...) +#define atl_nic_info(fmt, args...) +#define atl_nic_warn(fmt, args...) +#define atl_nic_err(fmt, args...) +#define atl_dev_init_warn(fmt, args...) +#define atl_dev_init_err(fmt, args...) + +#endif /* __KERNEL__ */ + +#endif /* _ATL_LOG_H_ */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.c new file mode 100644 index 000000000000..448b868ff1e2 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.c @@ -0,0 +1,1738 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_macsec.h" +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) +#include "atl_common.h" +#include + +#include "macsec/macsec_api.h" +#define ATL_MACSEC_KEY_LEN_128_BIT 16 +#define ATL_MACSEC_KEY_LEN_192_BIT 24 +#define ATL_MACSEC_KEY_LEN_256_BIT 32 + +static unsigned int atl_macsec_bridge = 1; +module_param_named(macsec_bridge, atl_macsec_bridge, uint, 0644); + +enum atl_clear_type { + /* update HW configuration */ + ATL_CLEAR_HW = BIT(0), + /* update SW configuration (busy bits, pointers) */ + ATL_CLEAR_SW = BIT(1), + /* update both HW and SW configuration */ + ATL_CLEAR_ALL = ATL_CLEAR_HW | ATL_CLEAR_SW, +}; + +static int atl_clear_txsc(struct atl_nic *nic, const int txsc_idx, + enum atl_clear_type clear_type); +static int atl_clear_txsa(struct atl_nic *nic, struct atl_macsec_txsc *atl_txsc, + const int sa_num, enum atl_clear_type clear_type); +static int atl_clear_rxsc(struct atl_nic *nic, const int rxsc_idx, + enum atl_clear_type clear_type); +static int atl_clear_rxsa(struct atl_nic *nic, struct atl_macsec_rxsc *atl_rxsc, + const int sa_num, enum atl_clear_type clear_type); +static int atl_clear_secy(struct atl_nic *nic, const struct macsec_secy *secy, + enum atl_clear_type clear_type); +static int atl_apply_macsec_cfg(struct atl_hw *hw); +static int atl_apply_secy_cfg(struct atl_hw *hw, + const struct macsec_secy *secy); + +static void atl_ether_addr_to_mac(u32 mac[2], unsigned char *emac) +{ + u32 tmp[2]; + + memset(&tmp, 0, sizeof(tmp)); + memcpy(((u8 *)tmp) + 2, emac, ETH_ALEN); + + mac[0] = swab32(tmp[1]); + mac[1] = swab32(tmp[0]); +} + +/* There's a 1:1 mapping between SecY and TX SC */ +static int atl_get_txsc_idx_from_secy(struct atl_hw *hw, + const struct macsec_secy *secy) +{ + int i; + + if (unlikely(!secy)) + return -1; + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (hw->macsec_cfg.atl_txsc[i].sw_secy == secy) + return i; + } + return -1; +} + +static int atl_get_rxsc_idx_from_rxsc(struct atl_hw *hw, + const struct macsec_rx_sc *rxsc) +{ + int i; + + if (unlikely(!rxsc)) + return -1; + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (hw->macsec_cfg.atl_rxsc[i].sw_rxsc == rxsc) + return i; + } + + return -1; +} + +static int atl_get_txsc_idx_from_sc_idx(const enum atl_macsec_sc_sa sc_sa, + const unsigned int sc_idx) +{ + switch (sc_sa) { + case atl_macsec_sa_sc_4sa_8sc: + return sc_idx >> 2; + case atl_macsec_sa_sc_2sa_16sc: + return sc_idx >> 1; + case atl_macsec_sa_sc_1sa_32sc: + return sc_idx; + default: + WARN_ONCE(1, "Invalid sc_sa"); + } + return -1; +} + +/* Rotate keys u32[8] */ +static void atl_rotate_keys(u32 (*key)[8], int key_len) +{ + u32 tmp[8]; + + memcpy(&tmp, key, sizeof(tmp)); + memset(*key, 0, sizeof(*key)); + + if (key_len == ATL_MACSEC_KEY_LEN_128_BIT) { + (*key)[0] = swab32(tmp[3]); + (*key)[1] = swab32(tmp[2]); + (*key)[2] = swab32(tmp[1]); + (*key)[3] = swab32(tmp[0]); + } else if (key_len == ATL_MACSEC_KEY_LEN_192_BIT) { + (*key)[0] = swab32(tmp[5]); + (*key)[1] = swab32(tmp[4]); + (*key)[2] = swab32(tmp[3]); + (*key)[3] = swab32(tmp[2]); + (*key)[4] = swab32(tmp[1]); + (*key)[5] = swab32(tmp[0]); + } else if (key_len == ATL_MACSEC_KEY_LEN_256_BIT) { + (*key)[0] = swab32(tmp[7]); + (*key)[1] = swab32(tmp[6]); + (*key)[2] = swab32(tmp[5]); + (*key)[3] = swab32(tmp[4]); + (*key)[4] = swab32(tmp[3]); + (*key)[5] = swab32(tmp[2]); + (*key)[6] = swab32(tmp[1]); + (*key)[7] = swab32(tmp[0]); + } else { + pr_warn("Rotate_keys: invalid key_len\n"); + } +} + +#define STATS_2x32_TO_64(stat_field) \ + (((u64)stat_field[1] << 32) | stat_field[0]) + +static int atl_get_macsec_common_stats(struct atl_hw *hw, + struct atl_macsec_common_stats *stats) +{ + struct aq_mss_ingress_common_counters ingress_counters; + struct aq_mss_egress_common_counters egress_counters; + int ret; + + /* MACSEC counters */ + ret = aq_mss_get_ingress_common_counters(hw, &ingress_counters); + if (unlikely(ret)) + return ret; + + stats->in.ctl_pkts = STATS_2x32_TO_64(ingress_counters.ctl_pkts); + stats->in.tagged_miss_pkts = + STATS_2x32_TO_64(ingress_counters.tagged_miss_pkts); + stats->in.untagged_miss_pkts = + STATS_2x32_TO_64(ingress_counters.untagged_miss_pkts); + stats->in.notag_pkts = STATS_2x32_TO_64(ingress_counters.notag_pkts); + stats->in.untagged_pkts = + STATS_2x32_TO_64(ingress_counters.untagged_pkts); + stats->in.bad_tag_pkts = + STATS_2x32_TO_64(ingress_counters.bad_tag_pkts); + stats->in.no_sci_pkts = STATS_2x32_TO_64(ingress_counters.no_sci_pkts); + stats->in.unknown_sci_pkts = + STATS_2x32_TO_64(ingress_counters.unknown_sci_pkts); + stats->in.ctrl_prt_pass_pkts = + STATS_2x32_TO_64(ingress_counters.ctrl_prt_pass_pkts); + stats->in.unctrl_prt_pass_pkts = + STATS_2x32_TO_64(ingress_counters.unctrl_prt_pass_pkts); + stats->in.ctrl_prt_fail_pkts = + STATS_2x32_TO_64(ingress_counters.ctrl_prt_fail_pkts); + stats->in.unctrl_prt_fail_pkts = + STATS_2x32_TO_64(ingress_counters.unctrl_prt_fail_pkts); + stats->in.too_long_pkts = + STATS_2x32_TO_64(ingress_counters.too_long_pkts); + stats->in.igpoc_ctl_pkts = + STATS_2x32_TO_64(ingress_counters.igpoc_ctl_pkts); + stats->in.ecc_error_pkts = + STATS_2x32_TO_64(ingress_counters.ecc_error_pkts); + stats->in.unctrl_hit_drop_redir = + STATS_2x32_TO_64(ingress_counters.unctrl_hit_drop_redir); + + ret = aq_mss_get_egress_common_counters(hw, &egress_counters); + if (unlikely(ret)) + return ret; + stats->out.ctl_pkts = STATS_2x32_TO_64(egress_counters.ctl_pkt); + stats->out.unknown_sa_pkts = + STATS_2x32_TO_64(egress_counters.unknown_sa_pkts); + stats->out.untagged_pkts = + STATS_2x32_TO_64(egress_counters.untagged_pkts); + stats->out.too_long = STATS_2x32_TO_64(egress_counters.too_long); + stats->out.ecc_error_pkts = + STATS_2x32_TO_64(egress_counters.ecc_error_pkts); + stats->out.unctrl_hit_drop_redir = + STATS_2x32_TO_64(egress_counters.unctrl_hit_drop_redir); + + return 0; +} + +static int atl_get_rxsa_stats(struct atl_hw *hw, int sa_idx, + struct atl_macsec_rx_sa_stats *stats) +{ + struct aq_mss_ingress_sa_counters i_sa_counters; + int ret; + + ret = aq_mss_get_ingress_sa_counters(hw, &i_sa_counters, sa_idx); + if (unlikely(ret)) + return ret; + + stats->untagged_hit_pkts = + STATS_2x32_TO_64(i_sa_counters.untagged_hit_pkts); + stats->ctrl_hit_drop_redir_pkts = + STATS_2x32_TO_64(i_sa_counters.ctrl_hit_drop_redir_pkts); + stats->not_using_sa = STATS_2x32_TO_64(i_sa_counters.not_using_sa); + stats->unused_sa = STATS_2x32_TO_64(i_sa_counters.unused_sa); + stats->not_valid_pkts = STATS_2x32_TO_64(i_sa_counters.not_valid_pkts); + stats->invalid_pkts = STATS_2x32_TO_64(i_sa_counters.invalid_pkts); + stats->ok_pkts = STATS_2x32_TO_64(i_sa_counters.ok_pkts); + stats->late_pkts = STATS_2x32_TO_64(i_sa_counters.late_pkts); + stats->delayed_pkts = STATS_2x32_TO_64(i_sa_counters.delayed_pkts); + stats->unchecked_pkts = STATS_2x32_TO_64(i_sa_counters.unchecked_pkts); + stats->validated_octets = + STATS_2x32_TO_64(i_sa_counters.validated_octets); + stats->decrypted_octets = + STATS_2x32_TO_64(i_sa_counters.decrypted_octets); + + return 0; +} + +static int atl_get_txsa_stats(struct atl_hw *hw, int sa_idx, + struct atl_macsec_tx_sa_stats *stats) +{ + struct aq_mss_egress_sa_counters e_sa_counters; + int ret; + + ret = aq_mss_get_egress_sa_counters(hw, &e_sa_counters, sa_idx); + if (unlikely(ret)) + return ret; + + stats->sa_hit_drop_redirect = + STATS_2x32_TO_64(e_sa_counters.sa_hit_drop_redirect); + stats->sa_protected2_pkts = + STATS_2x32_TO_64(e_sa_counters.sa_protected2_pkts); + stats->sa_protected_pkts = + STATS_2x32_TO_64(e_sa_counters.sa_protected_pkts); + stats->sa_encrypted_pkts = + STATS_2x32_TO_64(e_sa_counters.sa_encrypted_pkts); + + return 0; +} + +static int atl_get_txsa_next_pn(struct atl_hw *hw, int sa_idx, u32 *pn) +{ + struct aq_mss_egress_sa_record sa_rec; + int ret; + + ret = aq_mss_get_egress_sa_record(hw, &sa_rec, sa_idx); + if (likely(!ret)) + *pn = sa_rec.next_pn; + + return ret; +} + +static int atl_get_rxsa_next_pn(struct atl_hw *hw, int sa_idx, u32 *pn) +{ + struct aq_mss_ingress_sa_record sa_rec; + int ret; + + ret = aq_mss_get_ingress_sa_record(hw, &sa_rec, sa_idx); + if (likely(!ret)) + *pn = (!sa_rec.sat_nextpn) ? sa_rec.next_pn : 0; + + return ret; +} + +static int atl_get_txsc_stats(struct atl_hw *hw, int sc_idx, + struct atl_macsec_tx_sc_stats *stats) +{ + struct aq_mss_egress_sc_counters e_sc_counters; + int ret; + + ret = aq_mss_get_egress_sc_counters(hw, &e_sc_counters, sc_idx); + if (unlikely(ret)) + return ret; + + stats->sc_protected_pkts = + STATS_2x32_TO_64(e_sc_counters.sc_protected_pkts); + stats->sc_encrypted_pkts = + STATS_2x32_TO_64(e_sc_counters.sc_encrypted_pkts); + stats->sc_protected_octets = + STATS_2x32_TO_64(e_sc_counters.sc_protected_octets); + stats->sc_encrypted_octets = + STATS_2x32_TO_64(e_sc_counters.sc_encrypted_octets); + + return 0; +} + +int atl_macsec_rx_sa_cnt(struct atl_hw *hw) +{ + int i, cnt = 0; + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (!test_bit(i, &hw->macsec_cfg.rxsc_idx_busy)) + continue; + cnt += hweight_long(hw->macsec_cfg.atl_rxsc[i].rx_sa_idx_busy); + } + + return cnt; +} + +int atl_macsec_tx_sc_cnt(struct atl_hw *hw) +{ + return hweight_long(hw->macsec_cfg.txsc_idx_busy); +} + +int atl_macsec_tx_sa_cnt(struct atl_hw *hw) +{ + int i, cnt = 0; + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (!test_bit(i, &hw->macsec_cfg.txsc_idx_busy)) + continue; + cnt += hweight_long(hw->macsec_cfg.atl_txsc[i].tx_sa_idx_busy); + } + + return cnt; +} + +int atl_macsec_update_stats(struct atl_hw *hw) +{ + struct atl_macsec_txsc *atl_txsc; + struct atl_macsec_rxsc *atl_rxsc; + int i, sa_idx, assoc_num; + int ret = 0; + + atl_get_macsec_common_stats(hw, &hw->macsec_cfg.stats); + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (!(hw->macsec_cfg.txsc_idx_busy & BIT(i))) + continue; + atl_txsc = &hw->macsec_cfg.atl_txsc[i]; + + ret = atl_get_txsc_stats(hw, atl_txsc->hw_sc_idx, + &atl_txsc->stats); + if (ret) + return ret; + + for (assoc_num = 0; assoc_num < MACSEC_NUM_AN; assoc_num++) { + if (!test_bit(assoc_num, &atl_txsc->tx_sa_idx_busy)) + continue; + sa_idx = atl_txsc->hw_sc_idx | assoc_num; + ret = atl_get_txsa_stats( + hw, sa_idx, &atl_txsc->tx_sa_stats[assoc_num]); + if (ret) + return ret; + } + } + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (!(test_bit(i, &hw->macsec_cfg.rxsc_idx_busy))) + continue; + atl_rxsc = &hw->macsec_cfg.atl_rxsc[i]; + + for (assoc_num = 0; assoc_num < MACSEC_NUM_AN; assoc_num++) { + if (!test_bit(assoc_num, &atl_rxsc->rx_sa_idx_busy)) + continue; + sa_idx = atl_rxsc->hw_sc_idx | assoc_num; + + ret = atl_get_rxsa_stats( + hw, sa_idx, &atl_rxsc->rx_sa_stats[assoc_num]); + if (ret) + return ret; + } + } + + return ret; +} + +int atl_init_macsec(struct atl_hw *hw) +{ + struct aq_mss_ingress_prectlf_record rx_prectlf_rec; + u32 ctl_ether_types[2] = { ETH_P_PAE, ETH_P_PAUSE}; + struct macsec_msg_fw_response resp; + struct macsec_msg_fw_request msg; + int num_ctl_ether_types = 0; + int index = 0, tbl_idx; + int ret; + + rtnl_lock(); + memset(&msg, 0, sizeof(msg)); + memset(&resp, 0, sizeof(resp)); + + if (hw->mcp.ops->send_macsec_req) { + struct macsec_cfg_request cfg; + + memset(&cfg, 0, sizeof(cfg)); + cfg.enabled = 1; + cfg.egress_threshold = 0xffffffff; + cfg.ingress_threshold = 0xffffffff; + cfg.interrupts_enabled = 1; + + msg.msg_type = macsec_cfg_msg; + msg.cfg = cfg; + + ret = hw->mcp.ops->send_macsec_req(hw, &msg, &resp); + if (ret) + goto unlock; + } + + /* Init Ethertype bypass filters */ + memset(&rx_prectlf_rec, 0, sizeof(rx_prectlf_rec)); + rx_prectlf_rec.match_type = 4; /* Match eth_type only */ + rx_prectlf_rec.action = 0; /* Bypass MACSEC modules */ + aq_mss_set_ingress_prectlf_record(hw, &rx_prectlf_rec, 0); + aq_mss_set_packet_edit_control(hw, 0xb068); + for (index = 0; index < ARRAY_SIZE(ctl_ether_types); index++) { + struct aq_mss_egress_ctlf_record tx_ctlf_rec; + + memset(&rx_prectlf_rec, 0, sizeof(rx_prectlf_rec)); + memset(&tx_ctlf_rec, 0, sizeof(tx_ctlf_rec)); + + if (ctl_ether_types[index] == 0) + continue; + + tx_ctlf_rec.eth_type = ctl_ether_types[index]; + tx_ctlf_rec.match_type = 4; /* Match eth_type only */ + tx_ctlf_rec.match_mask = 0xf; /* match for eth_type */ + tx_ctlf_rec.action = 0; /* Bypass MACSEC modules */ + tbl_idx = NUMROWS_EGRESSCTLFRECORD - num_ctl_ether_types - 1; + aq_mss_set_egress_ctlf_record(hw, &tx_ctlf_rec, tbl_idx); + + rx_prectlf_rec.eth_type = ctl_ether_types[index]; + rx_prectlf_rec.match_type = 4; /* Match eth_type only */ + rx_prectlf_rec.match_mask = 0xf; /* match for eth_type */ + rx_prectlf_rec.action = 0; /* Bypass MACSEC modules */ + tbl_idx = + NUMROWS_INGRESSPRECTLFRECORD - num_ctl_ether_types - 1; + aq_mss_set_ingress_prectlf_record(hw, &rx_prectlf_rec, tbl_idx); + + num_ctl_ether_types++; + } + + ret = atl_apply_macsec_cfg(hw); + +unlock: + rtnl_unlock(); + return ret; +} + +static int atl_mdo_dev_open(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + int ret = 0; + + if (ctx->prepare) + return 0; + + if (netif_carrier_ok(nic->ndev)) + ret = atl_apply_secy_cfg(&nic->hw, ctx->secy); + + atl_fwd_notify(nic, ATL_FWD_NOTIFY_MACSEC_ON, ctx->secy->netdev); + + return ret; +} + +static int atl_mdo_dev_stop(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + + if (ctx->prepare) + return 0; + + atl_fwd_notify(nic, ATL_FWD_NOTIFY_MACSEC_OFF, ctx->secy->netdev); + + return atl_clear_secy(nic, ctx->secy, ATL_CLEAR_HW); +} + +static int atl_set_txsc(struct atl_hw *hw, int txsc_idx) +{ + struct atl_macsec_txsc *atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + const struct macsec_secy *secy = atl_txsc->sw_secy; + struct aq_mss_egress_class_record tx_class_rec; + unsigned int sc_idx = atl_txsc->hw_sc_idx; + struct aq_mss_egress_sc_record sc_rec; + __be64 nsci; + int ret = 0; + + memset(&tx_class_rec, 0, sizeof(tx_class_rec)); + memset(&sc_rec, 0, sizeof(sc_rec)); + + atl_ether_addr_to_mac(tx_class_rec.mac_sa, secy->netdev->dev_addr); + + atl_dev_dbg("set secy: sci %#llx, sc_idx=%d, protect=%d, curr_an=%d\n", + secy->sci, sc_idx, secy->protect_frames, + secy->tx_sc.encoding_sa); + + nsci = cpu_to_be64((__force u64)secy->sci); + memcpy(tx_class_rec.sci, &nsci, sizeof(nsci)); + tx_class_rec.sci_mask = 0; + + if (!atl_macsec_bridge) + tx_class_rec.sa_mask = 0x3f; + + tx_class_rec.action = 0; /* forward to SA/SC table */ + tx_class_rec.valid = 1; + + tx_class_rec.sc_idx = sc_idx; + + tx_class_rec.sc_sa = hw->macsec_cfg.sc_sa; + + ret = aq_mss_set_egress_class_record(hw, &tx_class_rec, txsc_idx); + if (ret) + return ret; + + sc_rec.protect = secy->protect_frames; + if (secy->tx_sc.encrypt) + sc_rec.tci |= BIT(1); + if (secy->tx_sc.scb) + sc_rec.tci |= BIT(2); + if (secy->tx_sc.send_sci) + sc_rec.tci |= BIT(3); + if (secy->tx_sc.end_station) + sc_rec.tci |= BIT(4); + /* The C bit is clear if and only if the Secure Data is + * exactly the same as the User Data and the ICV is 16 octets long. + */ + if (!(secy->icv_len == 16 && !secy->tx_sc.encrypt)) + sc_rec.tci |= BIT(0); + + sc_rec.an_roll = 0; + + switch (secy->key_len) { + case ATL_MACSEC_KEY_LEN_128_BIT: + sc_rec.sak_len = 0; + break; + case ATL_MACSEC_KEY_LEN_192_BIT: + sc_rec.sak_len = 1; + break; + case ATL_MACSEC_KEY_LEN_256_BIT: + sc_rec.sak_len = 2; + break; + default: + WARN_ONCE(1, "Invalid sc_sa"); + return -EINVAL; + } + + sc_rec.curr_an = secy->tx_sc.encoding_sa; + sc_rec.valid = 1; + sc_rec.fresh = 1; + + if (atl_macsec_bridge) { + struct aq_mss_ingress_prectlf_record rx_prectlf_rec; + + memset(&rx_prectlf_rec, 0, sizeof(rx_prectlf_rec)); + ret = aq_mss_set_ingress_prectlf_record(hw, &rx_prectlf_rec, 0); + if (ret) + return ret; + ret = aq_mss_set_packet_edit_control(hw, 0); + if (ret) + return ret; + ret = aq_mss_set_drop_igprc_miss_packets(hw, 1); + if (ret) + return ret; + } + + return aq_mss_set_egress_sc_record(hw, &sc_rec, sc_idx); +} + +static u32 sc_idx_max(const enum atl_macsec_sc_sa sc_sa) +{ + u32 result = 0; + + switch (sc_sa) { + case atl_macsec_sa_sc_4sa_8sc: + result = 8; + break; + case atl_macsec_sa_sc_2sa_16sc: + result = 16; + break; + case atl_macsec_sa_sc_1sa_32sc: + result = 32; + break; + default: + break; + }; + + return result; +} + +static u32 atl_to_hw_sc_idx(const u32 sc_idx, const enum atl_macsec_sc_sa sc_sa) +{ + switch (sc_sa) { + case atl_macsec_sa_sc_4sa_8sc: + return sc_idx << 2; + case atl_macsec_sa_sc_2sa_16sc: + return sc_idx << 1; + case atl_macsec_sa_sc_1sa_32sc: + return sc_idx; + default: + /* Should never happen */ + break; + }; + + WARN_ON(true); + return sc_idx; +} + +static enum atl_macsec_sc_sa sc_sa_from_num_an(const int num_an) +{ + enum atl_macsec_sc_sa sc_sa = atl_macsec_sa_sc_not_used; + + switch (num_an) { + case 4: + sc_sa = atl_macsec_sa_sc_4sa_8sc; + break; + case 2: + sc_sa = atl_macsec_sa_sc_2sa_16sc; + break; + case 1: + sc_sa = atl_macsec_sa_sc_1sa_32sc; + break; + default: + break; + } + + return sc_sa; +} + +static int atl_mdo_add_secy(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + const struct macsec_secy *secy = ctx->secy; + struct atl_hw *hw = &nic->hw; + enum atl_macsec_sc_sa sc_sa; + u32 txsc_idx; + int ret = 0; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 7, 0) + if (secy->xpn) + return -EOPNOTSUPP; +#endif + + sc_sa = sc_sa_from_num_an(MACSEC_NUM_AN); + if (sc_sa == atl_macsec_sa_sc_not_used) + return -EINVAL; + + if (atl_macsec_bridge && hweight32(hw->macsec_cfg.txsc_idx_busy)) + return -ENOSPC; + + if (hweight32(hw->macsec_cfg.txsc_idx_busy) >= sc_idx_max(sc_sa)) + return -ENOSPC; + + txsc_idx = ffz(hw->macsec_cfg.txsc_idx_busy); + if (txsc_idx == ATL_MACSEC_MAX_SC) + return -ENOSPC; + + if (ctx->prepare) + return 0; + + hw->macsec_cfg.sc_sa = sc_sa; + hw->macsec_cfg.atl_txsc[txsc_idx].hw_sc_idx = + atl_to_hw_sc_idx(txsc_idx, sc_sa); + hw->macsec_cfg.atl_txsc[txsc_idx].sw_secy = secy; + atl_dev_dbg("add secy: txsc_idx=%d, sc_idx=%d\n", txsc_idx, + hw->macsec_cfg.atl_txsc[txsc_idx].hw_sc_idx); + + if (netif_carrier_ok(nic->ndev) && netif_running(secy->netdev)) + ret = atl_set_txsc(hw, txsc_idx); + + set_bit(txsc_idx, &hw->macsec_cfg.txsc_idx_busy); + + return 0; +} + +static int atl_mdo_upd_secy(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + const struct macsec_secy *secy; + struct atl_hw *hw = &nic->hw; + int txsc_idx; + int ret = 0; + + txsc_idx = atl_get_txsc_idx_from_secy(&nic->hw, ctx->secy); + if (txsc_idx < 0) + return -ENOENT; + + if (ctx->prepare) + return 0; + + secy = hw->macsec_cfg.atl_txsc[txsc_idx].sw_secy; + + if (netif_carrier_ok(nic->ndev) && netif_running(secy->netdev)) + ret = atl_set_txsc(hw, txsc_idx); + + return ret; +} + +static int atl_clear_txsc(struct atl_nic *nic, const int txsc_idx, + enum atl_clear_type clear_type) +{ + struct atl_hw *hw = &nic->hw; + struct atl_macsec_txsc *tx_sc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + struct aq_mss_egress_class_record tx_class_rec; + struct aq_mss_egress_sc_record sc_rec; + int ret = 0; + int sa_num; + + memset(&tx_class_rec, 0, sizeof(tx_class_rec)); + memset(&sc_rec, 0, sizeof(sc_rec)); + + for_each_set_bit (sa_num, &tx_sc->tx_sa_idx_busy, ATL_MACSEC_MAX_SA) { + ret = atl_clear_txsa(nic, tx_sc, sa_num, clear_type); + if (ret) + return ret; + } + + if (clear_type & ATL_CLEAR_HW) { + ret = aq_mss_set_egress_class_record(hw, &tx_class_rec, + txsc_idx); + if (ret) + return ret; + + sc_rec.fresh = 1; + ret = aq_mss_set_egress_sc_record(hw, &sc_rec, + tx_sc->hw_sc_idx); + if (ret) + return ret; + + /* if last txsc is removed then pass macsec packets to host */ + if (atl_macsec_bridge) { + struct aq_mss_ingress_prectlf_record rx_prectlf; + + memset(&rx_prectlf, 0, sizeof(rx_prectlf)); + rx_prectlf.match_type = 4; /* Match eth_type only */ + rx_prectlf.action = 0; + ret = aq_mss_set_ingress_prectlf_record(hw, &rx_prectlf, 0); + if (ret) + return ret; + ret = aq_mss_set_packet_edit_control(hw, 0xb068); + if (ret) + return ret; + ret = aq_mss_set_drop_igprc_miss_packets(hw, 0); + if (ret) + return ret; + } + } + + if (clear_type & ATL_CLEAR_SW) { + clear_bit(txsc_idx, &hw->macsec_cfg.txsc_idx_busy); + hw->macsec_cfg.atl_txsc[txsc_idx].sw_secy = NULL; + } + + return ret; +} + +static int atl_mdo_del_secy(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + + if (ctx->prepare) + return 0; + + return atl_clear_secy(nic, ctx->secy, ATL_CLEAR_ALL); +} + +static int atl_update_txsa(struct atl_hw *hw, unsigned int sc_idx, + const struct macsec_secy *secy, + const struct macsec_tx_sa *tx_sa, + const unsigned char *key, unsigned char an) +{ + struct aq_mss_egress_sakey_record key_rec; + const unsigned int sa_idx = sc_idx | an; + struct aq_mss_egress_sa_record sa_rec; +#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 7, 0) + const u32 next_pn = tx_sa->next_pn; +#else + const u32 next_pn = tx_sa->next_pn_halves.lower; +#endif + int ret = 0; + + atl_dev_dbg("set tx_sa %d: active=%d, next_pn=%d\n", an, tx_sa->active, + next_pn); + + memset(&sa_rec, 0, sizeof(sa_rec)); + sa_rec.valid = tx_sa->active; + sa_rec.fresh = 1; + sa_rec.next_pn = next_pn; + + ret = aq_mss_set_egress_sa_record(hw, &sa_rec, sa_idx); + if (ret) { + atl_dev_err("aq_mss_set_egress_sa_record failed with %d\n", + ret); + return ret; + } + + if (!key) + return ret; + + memset(&key_rec, 0, sizeof(key_rec)); + memcpy(&key_rec.key, key, secy->key_len); + + atl_rotate_keys(&key_rec.key, secy->key_len); + + ret = aq_mss_set_egress_sakey_record(hw, &key_rec, sa_idx); + if (ret) + atl_dev_err("aq_mss_set_egress_sakey_record failed with %d\n", + ret); + + return ret; +} + +static int atl_mdo_add_txsa(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + const struct macsec_secy *secy = ctx->secy; + struct atl_macsec_txsc *atl_txsc; + struct atl_hw *hw = &nic->hw; + int txsc_idx; + + txsc_idx = atl_get_txsc_idx_from_secy(hw, secy); + if (txsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + set_bit(ctx->sa.assoc_num, &atl_txsc->tx_sa_idx_busy); + + memcpy(atl_txsc->tx_sa_key[ctx->sa.assoc_num], ctx->sa.key, + secy->key_len); + + if (netif_carrier_ok(nic->ndev) && netif_running(secy->netdev)) + return atl_update_txsa(&nic->hw, atl_txsc->hw_sc_idx, secy, + ctx->sa.tx_sa, ctx->sa.key, + ctx->sa.assoc_num); + + return 0; +} + +static int atl_mdo_upd_txsa(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + const struct macsec_secy *secy = ctx->secy; + struct atl_macsec_txsc *atl_txsc; + struct atl_hw *hw = &nic->hw; + int txsc_idx; + + txsc_idx = atl_get_txsc_idx_from_secy(hw, secy); + if (txsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + if (netif_carrier_ok(nic->ndev) && netif_running(secy->netdev)) + return atl_update_txsa(&nic->hw, atl_txsc->hw_sc_idx, secy, + ctx->sa.tx_sa, NULL, ctx->sa.assoc_num); + + return 0; +} + +static int atl_clear_txsa(struct atl_nic *nic, struct atl_macsec_txsc *atl_txsc, + const int sa_num, enum atl_clear_type clear_type) +{ + int sa_idx = atl_txsc->hw_sc_idx | sa_num; + struct atl_hw *hw = &nic->hw; + int ret = 0; + + if (clear_type & ATL_CLEAR_SW) + clear_bit(sa_num, &atl_txsc->tx_sa_idx_busy); + + if ((clear_type & ATL_CLEAR_HW) && netif_carrier_ok(nic->ndev)) { + struct aq_mss_egress_sakey_record key_rec; + struct aq_mss_egress_sa_record sa_rec; + + memset(&sa_rec, 0, sizeof(sa_rec)); + + sa_rec.fresh = 1; + + ret = aq_mss_set_egress_sa_record(hw, &sa_rec, sa_idx); + if (ret) + return ret; + + memset(&key_rec, 0, sizeof(key_rec)); + return aq_mss_set_egress_sakey_record(hw, &key_rec, sa_idx); + } + + return 0; +} + +static int atl_mdo_del_txsa(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + int txsc_idx; + + txsc_idx = atl_get_txsc_idx_from_secy(&nic->hw, ctx->secy); + if (txsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + return atl_clear_txsa(nic, &nic->hw.macsec_cfg.atl_txsc[txsc_idx], + ctx->sa.assoc_num, ATL_CLEAR_ALL); +} + +static int atl_rxsc_validate_frames(const enum macsec_validation_type validate) +{ + switch (validate) { + case MACSEC_VALIDATE_DISABLED: + return 2; + case MACSEC_VALIDATE_CHECK: + return 1; + case MACSEC_VALIDATE_STRICT: + return 0; + default: + break; + } + + /* should never be here */ + WARN_ON(true); + return 0; +} + +static int atl_set_rxsc(struct atl_hw *hw, const u32 rxsc_idx) +{ + const struct atl_macsec_rxsc *atl_rxsc = + &hw->macsec_cfg.atl_rxsc[rxsc_idx]; + struct aq_mss_ingress_preclass_record pre_class_record; + const struct macsec_rx_sc *rx_sc = atl_rxsc->sw_rxsc; + const struct macsec_secy *secy = atl_rxsc->sw_secy; + const u32 hw_sc_idx = atl_rxsc->hw_sc_idx; + struct aq_mss_ingress_sc_record sc_record; + __be64 nsci; + int ret = 0; + + atl_dev_dbg("set rx_sc: rxsc_idx=%d, sci %#llx, hw_sc_idx=%d\n", + rxsc_idx, rx_sc->sci, hw_sc_idx); + + memset(&pre_class_record, 0, sizeof(pre_class_record)); + nsci = cpu_to_be64((__force u64)rx_sc->sci); + memcpy(pre_class_record.sci, &nsci, sizeof(nsci)); + pre_class_record.sci_mask = 0xff; + /* match all MACSEC ethertype packets */ + pre_class_record.eth_type = ETH_P_MACSEC; + pre_class_record.eth_type_mask = 0x3; + + atl_ether_addr_to_mac(pre_class_record.mac_sa, (char *)&rx_sc->sci); + pre_class_record.sa_mask = 0x3f; + + pre_class_record.an_mask = hw->macsec_cfg.sc_sa; + pre_class_record.sc_idx = hw_sc_idx; + /* strip SecTAG & forward for decryption */ + pre_class_record.action = 0x0; + pre_class_record.valid = 1; + + ret = aq_mss_set_ingress_preclass_record(hw, &pre_class_record, + 2 * rxsc_idx + 1); + if (ret) { + atl_dev_err( + "aq_mss_set_ingress_preclass_record failed with %d\n", + ret); + return ret; + } + + /* If SCI is absent, then match by SA alone */ + pre_class_record.sci_mask = 0; + pre_class_record.sci_from_table = 1; + + ret = aq_mss_set_ingress_preclass_record(hw, &pre_class_record, + 2 * rxsc_idx); + if (ret) { + atl_dev_err( + "aq_mss_set_ingress_preclass_record failed with %d\n", + ret); + return ret; + } + + memset(&sc_record, 0, sizeof(sc_record)); + sc_record.validate_frames = + atl_rxsc_validate_frames(secy->validate_frames); + if (secy->replay_protect) { + sc_record.replay_protect = 1; + sc_record.anti_replay_window = secy->replay_window; + } else { + /* HW workaround to not drop Delayed frames */ + sc_record.anti_replay_window = ~0; + } + sc_record.valid = 1; + sc_record.fresh = 1; + + ret = aq_mss_set_ingress_sc_record(hw, &sc_record, hw_sc_idx); + if (ret) { + atl_dev_err("aq_mss_set_ingress_sc_record failed with %d\n", + ret); + return ret; + } + + return ret; +} + +static int atl_mdo_add_rxsc(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + struct atl_macsec_cfg *cfg = &nic->hw.macsec_cfg; + const u32 rxsc_idx_max = sc_idx_max(cfg->sc_sa); + u32 rxsc_idx; + int ret = 0; + + if (hweight32(cfg->rxsc_idx_busy) >= rxsc_idx_max) + return -ENOSPC; + + rxsc_idx = ffz(cfg->rxsc_idx_busy); + if (rxsc_idx >= rxsc_idx_max) + return -ENOSPC; + + if (ctx->prepare) + return 0; + + cfg->atl_rxsc[rxsc_idx].hw_sc_idx = + atl_to_hw_sc_idx(rxsc_idx, cfg->sc_sa); + cfg->atl_rxsc[rxsc_idx].sw_secy = ctx->secy; + cfg->atl_rxsc[rxsc_idx].sw_rxsc = ctx->rx_sc; + atl_nic_dbg("add rxsc: rxsc_idx=%u, hw_sc_idx=%u, rxsc=%p\n", rxsc_idx, + cfg->atl_rxsc[rxsc_idx].hw_sc_idx, + cfg->atl_rxsc[rxsc_idx].sw_rxsc); + + if (netif_carrier_ok(nic->ndev) && netif_running(ctx->secy->netdev)) + ret = atl_set_rxsc(&nic->hw, rxsc_idx); + + if (ret < 0) + return ret; + + set_bit(rxsc_idx, &cfg->rxsc_idx_busy); + + return 0; +} + +static int atl_mdo_upd_rxsc(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + int rxsc_idx; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(&nic->hw, ctx->rx_sc); + if (rxsc_idx < 0) + return -ENOENT; + + if (ctx->prepare) + return 0; + + if (netif_carrier_ok(nic->ndev) && netif_running(ctx->secy->netdev)) + return atl_set_rxsc(&nic->hw, rxsc_idx); + + return 0; +} + +static int atl_clear_rxsc(struct atl_nic *nic, const int rxsc_idx, + enum atl_clear_type clear_type) +{ + struct atl_hw *hw = &nic->hw; + struct atl_macsec_rxsc *rx_sc = &hw->macsec_cfg.atl_rxsc[rxsc_idx]; + int ret = 0; + int sa_num; + + for_each_set_bit (sa_num, &rx_sc->rx_sa_idx_busy, ATL_MACSEC_MAX_SA) { + ret = atl_clear_rxsa(nic, rx_sc, sa_num, clear_type); + if (ret) + return ret; + } + + if (clear_type & ATL_CLEAR_HW) { + struct aq_mss_ingress_preclass_record pre_class_record; + struct aq_mss_ingress_sc_record sc_record; + + memset(&pre_class_record, 0, sizeof(pre_class_record)); + memset(&sc_record, 0, sizeof(sc_record)); + + ret = aq_mss_set_ingress_preclass_record(hw, &pre_class_record, + 2 * rxsc_idx); + if (ret) { + atl_dev_err( + "aq_mss_set_ingress_preclass_record failed with %d\n", + ret); + return ret; + } + + ret = aq_mss_set_ingress_preclass_record(hw, &pre_class_record, + 2 * rxsc_idx + 1); + if (ret) { + atl_dev_err( + "aq_mss_set_ingress_preclass_record failed with %d\n", + ret); + return ret; + } + + sc_record.fresh = 1; + ret = aq_mss_set_ingress_sc_record(hw, &sc_record, + rx_sc->hw_sc_idx); + if (ret) + return ret; + } + + if (clear_type & ATL_CLEAR_SW) { + clear_bit(rxsc_idx, &hw->macsec_cfg.rxsc_idx_busy); + hw->macsec_cfg.atl_rxsc[rxsc_idx].sw_secy = NULL; + hw->macsec_cfg.atl_rxsc[rxsc_idx].sw_rxsc = NULL; + } + + return ret; +} + +static int atl_mdo_del_rxsc(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + enum atl_clear_type clear_type = ATL_CLEAR_SW; + int rxsc_idx; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(&nic->hw, ctx->rx_sc); + if (rxsc_idx < 0) + return -ENOENT; + + if (ctx->prepare) + return 0; + + if (netif_carrier_ok(nic->ndev)) + clear_type = ATL_CLEAR_ALL; + + return atl_clear_rxsc(nic, rxsc_idx, clear_type); +} + +static int atl_update_rxsa(struct atl_hw *hw, const unsigned int sc_idx, + const struct macsec_secy *secy, + const struct macsec_rx_sa *rx_sa, + const unsigned char *key, const unsigned char an) +{ + struct aq_mss_ingress_sakey_record sa_key_record; + struct aq_mss_ingress_sa_record sa_record; +#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 7, 0) + const u32 next_pn = rx_sa->next_pn; +#else + const u32 next_pn = rx_sa->next_pn_halves.lower; +#endif + const int sa_idx = sc_idx | an; + int ret = 0; + + atl_dev_dbg("set rx_sa %d: active=%d, next_pn=%d\n", an, rx_sa->active, + next_pn); + + memset(&sa_record, 0, sizeof(sa_record)); + sa_record.valid = rx_sa->active; + sa_record.fresh = 1; + sa_record.next_pn = next_pn; + + ret = aq_mss_set_ingress_sa_record(hw, &sa_record, sa_idx); + if (ret) { + atl_dev_err("aq_mss_set_ingress_sa_record failed with %d\n", + ret); + return ret; + } + + if (!key) + return ret; + + memset(&sa_key_record, 0, sizeof(sa_key_record)); + memcpy(&sa_key_record.key, key, secy->key_len); + + switch (secy->key_len) { + case ATL_MACSEC_KEY_LEN_128_BIT: + sa_key_record.key_len = 0; + break; + case ATL_MACSEC_KEY_LEN_192_BIT: + sa_key_record.key_len = 1; + break; + case ATL_MACSEC_KEY_LEN_256_BIT: + sa_key_record.key_len = 2; + break; + default: + return -1; + } + + atl_rotate_keys(&sa_key_record.key, secy->key_len); + + ret = aq_mss_set_ingress_sakey_record(hw, &sa_key_record, sa_idx); + if (ret) + atl_dev_err("aq_mss_set_ingress_sakey_record failed with %d\n", + ret); + + return ret; +} + +static int atl_mdo_add_rxsa(struct macsec_context *ctx) +{ + const struct macsec_rx_sc *rx_sc = ctx->sa.rx_sa->sc; + struct atl_nic *nic = netdev_priv(ctx->netdev); + const struct macsec_secy *secy = ctx->secy; + struct atl_hw *hw = &nic->hw; + int rxsc_idx; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(hw, rx_sc); + if (rxsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + set_bit(ctx->sa.assoc_num, + &hw->macsec_cfg.atl_rxsc[rxsc_idx].rx_sa_idx_busy); + + memcpy(hw->macsec_cfg.atl_rxsc[rxsc_idx].rx_sa_key[ctx->sa.assoc_num], + ctx->sa.key, secy->key_len); + + if (netif_carrier_ok(nic->ndev) && netif_running(secy->netdev)) + return atl_update_rxsa( + hw, hw->macsec_cfg.atl_rxsc[rxsc_idx].hw_sc_idx, secy, + ctx->sa.rx_sa, ctx->sa.key, ctx->sa.assoc_num); + + return 0; +} +static int atl_mdo_upd_rxsa(struct macsec_context *ctx) +{ + const struct macsec_rx_sc *rx_sc = ctx->sa.rx_sa->sc; + struct atl_nic *nic = netdev_priv(ctx->netdev); + const struct macsec_secy *secy = ctx->secy; + struct atl_hw *hw = &nic->hw; + int rxsc_idx; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(hw, rx_sc); + if (rxsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + if (netif_carrier_ok(nic->ndev) && netif_running(secy->netdev)) + return atl_update_rxsa( + hw, hw->macsec_cfg.atl_rxsc[rxsc_idx].hw_sc_idx, secy, + ctx->sa.rx_sa, NULL, ctx->sa.assoc_num); + + return 0; +} + +static int atl_clear_rxsa(struct atl_nic *nic, struct atl_macsec_rxsc *atl_rxsc, + const int sa_num, enum atl_clear_type clear_type) +{ + int sa_idx = atl_rxsc->hw_sc_idx | sa_num; + struct atl_hw *hw = &nic->hw; + int ret = 0; + + if (clear_type & ATL_CLEAR_SW) + clear_bit(sa_num, &atl_rxsc->rx_sa_idx_busy); + + if ((clear_type & ATL_CLEAR_HW) && netif_carrier_ok(nic->ndev)) { + struct aq_mss_ingress_sakey_record sa_key_record; + struct aq_mss_ingress_sa_record sa_record; + + memset(&sa_key_record, 0, sizeof(sa_key_record)); + memset(&sa_record, 0, sizeof(sa_record)); + + sa_record.fresh = 1; + ret = aq_mss_set_ingress_sa_record(hw, &sa_record, sa_idx); + if (ret) + return ret; + + return aq_mss_set_ingress_sakey_record(hw, &sa_key_record, + sa_idx); + } + + return ret; +} + +static int atl_mdo_del_rxsa(struct macsec_context *ctx) +{ + const struct macsec_rx_sc *rx_sc = ctx->sa.rx_sa->sc; + struct atl_nic *nic = netdev_priv(ctx->netdev); + int rxsc_idx; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(&nic->hw, rx_sc); + if (rxsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + return atl_clear_rxsa(nic, &nic->hw.macsec_cfg.atl_rxsc[rxsc_idx], + ctx->sa.assoc_num, ATL_CLEAR_ALL); +} + +static int atl_mdo_get_dev_stats(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + struct atl_hw *hw = &nic->hw; + struct atl_macsec_common_stats *stats = &hw->macsec_cfg.stats; + + if (ctx->prepare) + return 0; + + atl_get_macsec_common_stats(hw, stats); + + ctx->stats.dev_stats->OutPktsUntagged = stats->out.untagged_pkts; + ctx->stats.dev_stats->InPktsUntagged = stats->in.untagged_pkts; + ctx->stats.dev_stats->OutPktsTooLong = stats->out.too_long; + ctx->stats.dev_stats->InPktsNoTag = stats->in.notag_pkts; + ctx->stats.dev_stats->InPktsBadTag = stats->in.bad_tag_pkts; + ctx->stats.dev_stats->InPktsUnknownSCI = stats->in.unknown_sci_pkts; + ctx->stats.dev_stats->InPktsNoSCI = stats->in.no_sci_pkts; + ctx->stats.dev_stats->InPktsOverrun = 0; + + return 0; +} + +static int atl_mdo_get_tx_sc_stats(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + struct atl_macsec_tx_sc_stats *stats; + struct atl_macsec_txsc *atl_txsc; + struct atl_hw *hw = &nic->hw; + int txsc_idx; + + txsc_idx = atl_get_txsc_idx_from_secy(hw, ctx->secy); + if (txsc_idx < 0) + return -ENOENT; + + if (ctx->prepare) + return 0; + + atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + stats = &atl_txsc->stats; + atl_get_txsc_stats(hw, atl_txsc->hw_sc_idx, &atl_txsc->stats); + + ctx->stats.tx_sc_stats->OutPktsProtected = stats->sc_protected_pkts; + ctx->stats.tx_sc_stats->OutPktsEncrypted = stats->sc_encrypted_pkts; + ctx->stats.tx_sc_stats->OutOctetsProtected = stats->sc_protected_octets; + ctx->stats.tx_sc_stats->OutOctetsEncrypted = stats->sc_encrypted_octets; + + return 0; +} + +static int atl_mdo_get_tx_sa_stats(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + struct atl_macsec_tx_sa_stats *stats; + struct atl_macsec_txsc *atl_txsc; + const struct macsec_secy *secy; + struct atl_hw *hw = &nic->hw; + struct macsec_tx_sa *tx_sa; + unsigned int sa_idx; + int txsc_idx; + u32 next_pn; + int ret; + + txsc_idx = atl_get_txsc_idx_from_secy(hw, ctx->secy); + if (txsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + sa_idx = atl_txsc->hw_sc_idx | ctx->sa.assoc_num; + stats = &atl_txsc->tx_sa_stats[ctx->sa.assoc_num]; + ret = atl_get_txsa_stats(hw, sa_idx, stats); + if (ret) + return ret; + + ctx->stats.tx_sa_stats->OutPktsProtected = stats->sa_protected_pkts; + ctx->stats.tx_sa_stats->OutPktsEncrypted = stats->sa_encrypted_pkts; + + secy = atl_txsc->sw_secy; + tx_sa = rcu_dereference_bh(secy->tx_sc.sa[ctx->sa.assoc_num]); + ret = atl_get_txsa_next_pn(hw, sa_idx, &next_pn); + if (ret == 0) { + spin_lock_bh(&tx_sa->lock); + tx_sa->next_pn = next_pn; + spin_unlock_bh(&tx_sa->lock); + } + + return ret; +} + +static int atl_mdo_get_rx_sc_stats(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + struct atl_macsec_rx_sa_stats *stats; + struct atl_macsec_rxsc *atl_rxsc; + struct atl_hw *hw = &nic->hw; + unsigned int sa_idx; + int rxsc_idx; + int ret = 0; + int i; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(hw, ctx->rx_sc); + if (rxsc_idx < 0) + return -ENOENT; + + if (ctx->prepare) + return 0; + + atl_rxsc = &hw->macsec_cfg.atl_rxsc[rxsc_idx]; + for (i = 0; i < MACSEC_NUM_AN; i++) { + if (!test_bit(i, &atl_rxsc->rx_sa_idx_busy)) + continue; + + stats = &atl_rxsc->rx_sa_stats[i]; + sa_idx = atl_rxsc->hw_sc_idx | i; + ret = atl_get_rxsa_stats(hw, sa_idx, stats); + if (ret) + break; + + ctx->stats.rx_sc_stats->InOctetsValidated += + stats->validated_octets; + ctx->stats.rx_sc_stats->InOctetsDecrypted += + stats->decrypted_octets; + ctx->stats.rx_sc_stats->InPktsUnchecked += + stats->unchecked_pkts; + ctx->stats.rx_sc_stats->InPktsDelayed += stats->delayed_pkts; + ctx->stats.rx_sc_stats->InPktsOK += stats->ok_pkts; + ctx->stats.rx_sc_stats->InPktsInvalid += stats->invalid_pkts; + ctx->stats.rx_sc_stats->InPktsLate += stats->late_pkts; + ctx->stats.rx_sc_stats->InPktsNotValid += stats->not_valid_pkts; + ctx->stats.rx_sc_stats->InPktsNotUsingSA += stats->not_using_sa; + ctx->stats.rx_sc_stats->InPktsUnusedSA += stats->unused_sa; + } + + return ret; +} + +static int atl_mdo_get_rx_sa_stats(struct macsec_context *ctx) +{ + struct atl_nic *nic = netdev_priv(ctx->netdev); + struct atl_macsec_rx_sa_stats *stats; + struct atl_macsec_rxsc *atl_rxsc; + struct atl_hw *hw = &nic->hw; + struct macsec_rx_sa *rx_sa; + unsigned int sa_idx; + int rxsc_idx; + u32 next_pn; + int ret; + + rxsc_idx = atl_get_rxsc_idx_from_rxsc(hw, ctx->rx_sc); + if (rxsc_idx < 0) + return -EINVAL; + + if (ctx->prepare) + return 0; + + atl_rxsc = &hw->macsec_cfg.atl_rxsc[rxsc_idx]; + stats = &atl_rxsc->rx_sa_stats[ctx->sa.assoc_num]; + sa_idx = atl_rxsc->hw_sc_idx | ctx->sa.assoc_num; + ret = atl_get_rxsa_stats(hw, sa_idx, stats); + if (ret) + return ret; + + ctx->stats.rx_sa_stats->InPktsOK = stats->ok_pkts; + ctx->stats.rx_sa_stats->InPktsInvalid = stats->invalid_pkts; + ctx->stats.rx_sa_stats->InPktsNotValid = stats->not_valid_pkts; + ctx->stats.rx_sa_stats->InPktsNotUsingSA = stats->not_using_sa; + ctx->stats.rx_sa_stats->InPktsUnusedSA = stats->unused_sa; + + rx_sa = rcu_dereference_bh(atl_rxsc->sw_rxsc->sa[ctx->sa.assoc_num]); + ret = atl_get_rxsa_next_pn(hw, sa_idx, &next_pn); + if (ret == 0) { + spin_lock_bh(&rx_sa->lock); + rx_sa->next_pn = next_pn; + spin_unlock_bh(&rx_sa->lock); + } + + return ret; +} + +static int atl_apply_txsc_cfg(struct atl_hw *hw, const int txsc_idx) +{ + struct atl_macsec_txsc *atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + const struct macsec_secy *secy = atl_txsc->sw_secy; + struct macsec_tx_sa *tx_sa; + int ret = 0; + int i; + + if (!netif_running(secy->netdev)) + return ret; + + ret = atl_set_txsc(hw, txsc_idx); + if (ret) + return ret; + + for (i = 0; i < MACSEC_NUM_AN; i++) { + tx_sa = rcu_dereference_bh(secy->tx_sc.sa[i]); + if (tx_sa) { + ret = atl_update_txsa(hw, atl_txsc->hw_sc_idx, secy, + tx_sa, atl_txsc->tx_sa_key[i], i); + if (ret) + return ret; + } + } + + return ret; +} + +static int atl_apply_rxsc_cfg(struct atl_hw *hw, const int rxsc_idx) +{ + struct atl_macsec_rxsc *atl_rxsc = &hw->macsec_cfg.atl_rxsc[rxsc_idx]; + const struct macsec_secy *secy = atl_rxsc->sw_secy; + struct macsec_rx_sa *rx_sa; + int ret = 0; + int i; + + if (!netif_running(secy->netdev)) + return ret; + + ret = atl_set_rxsc(hw, rxsc_idx); + if (ret) + return ret; + + for (i = 0; i < MACSEC_NUM_AN; i++) { + rx_sa = rcu_dereference_bh(atl_rxsc->sw_rxsc->sa[i]); + if (rx_sa) { + ret = atl_update_rxsa(hw, atl_rxsc->hw_sc_idx, secy, + rx_sa, atl_rxsc->rx_sa_key[i], i); + if (ret) + return ret; + } + } + + return ret; +} + +static int atl_clear_secy(struct atl_nic *nic, const struct macsec_secy *secy, + enum atl_clear_type clear_type) +{ + struct atl_hw *hw = &nic->hw; + struct macsec_rx_sc *rx_sc; + int txsc_idx; + int rxsc_idx; + int ret = 0; + + txsc_idx = atl_get_txsc_idx_from_secy(hw, secy); + if (txsc_idx >= 0) { + ret = atl_clear_txsc(nic, txsc_idx, clear_type); + if (ret) + return ret; + } + + for (rx_sc = rcu_dereference_bh(secy->rx_sc); rx_sc; + rx_sc = rcu_dereference_bh(rx_sc->next)) { + rxsc_idx = atl_get_rxsc_idx_from_rxsc(hw, rx_sc); + if (rxsc_idx < 0) + continue; + + ret = atl_clear_rxsc(nic, rxsc_idx, clear_type); + if (ret) + return ret; + } + + return ret; +} + +static int atl_apply_secy_cfg(struct atl_hw *hw, const struct macsec_secy *secy) +{ + struct macsec_rx_sc *rx_sc; + int txsc_idx; + int rxsc_idx; + int ret = 0; + + txsc_idx = atl_get_txsc_idx_from_secy(hw, secy); + if (txsc_idx >= 0) + atl_apply_txsc_cfg(hw, txsc_idx); + + for (rx_sc = rcu_dereference_bh(secy->rx_sc); rx_sc && rx_sc->active; + rx_sc = rcu_dereference_bh(rx_sc->next)) { + rxsc_idx = atl_get_rxsc_idx_from_rxsc(hw, rx_sc); + if (unlikely(rxsc_idx < 0)) + continue; + + ret = atl_apply_rxsc_cfg(hw, rxsc_idx); + if (ret) + return ret; + } + + return ret; +} + +static int atl_apply_macsec_cfg(struct atl_hw *hw) +{ + int ret = 0; + int i; + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (hw->macsec_cfg.txsc_idx_busy & BIT(i)) { + ret = atl_apply_txsc_cfg(hw, i); + if (ret) + return ret; + } + } + + for (i = 0; i < ATL_MACSEC_MAX_SC; i++) { + if (hw->macsec_cfg.rxsc_idx_busy & BIT(i)) { + ret = atl_apply_rxsc_cfg(hw, i); + if (ret) + return ret; + } + } + + return ret; +} + +const struct macsec_ops atl_macsec_ops = { + .mdo_dev_open = atl_mdo_dev_open, + .mdo_dev_stop = atl_mdo_dev_stop, + .mdo_add_secy = atl_mdo_add_secy, + .mdo_upd_secy = atl_mdo_upd_secy, + .mdo_del_secy = atl_mdo_del_secy, + .mdo_add_rxsc = atl_mdo_add_rxsc, + .mdo_upd_rxsc = atl_mdo_upd_rxsc, + .mdo_del_rxsc = atl_mdo_del_rxsc, + .mdo_add_rxsa = atl_mdo_add_rxsa, + .mdo_upd_rxsa = atl_mdo_upd_rxsa, + .mdo_del_rxsa = atl_mdo_del_rxsa, + .mdo_add_txsa = atl_mdo_add_txsa, + .mdo_upd_txsa = atl_mdo_upd_txsa, + .mdo_del_txsa = atl_mdo_del_txsa, + .mdo_get_dev_stats = atl_mdo_get_dev_stats, + .mdo_get_tx_sc_stats = atl_mdo_get_tx_sc_stats, + .mdo_get_tx_sa_stats = atl_mdo_get_tx_sa_stats, + .mdo_get_rx_sc_stats = atl_mdo_get_rx_sc_stats, + .mdo_get_rx_sa_stats = atl_mdo_get_rx_sa_stats, +}; + +static int atl_sa_from_sa_idx(enum atl_macsec_sc_sa sc_sa, int sa_idx) +{ + switch (sc_sa) { + case atl_macsec_sa_sc_4sa_8sc: + return sa_idx & 3; + case atl_macsec_sa_sc_2sa_16sc: + return sa_idx & 1; + case atl_macsec_sa_sc_1sa_32sc: + return 0; + default: + WARN_ONCE(1, "Invalid sc_sa"); + } + return -EINVAL; +} + +static int atl_sc_idx_from_sa_idx(enum atl_macsec_sc_sa sc_sa, int sa_idx) +{ + switch (sc_sa) { + case atl_macsec_sa_sc_4sa_8sc: + return sa_idx & ~3; + case atl_macsec_sa_sc_2sa_16sc: + return sa_idx & ~1; + case atl_macsec_sa_sc_1sa_32sc: + return sa_idx; + default: + WARN_ONCE(1, "Invalid sc_sa"); + } + return -EINVAL; +} + +static void atl_check_txsa_expiration(struct atl_nic *nic) +{ + u32 egress_sa_expired, egress_sa_threshold_expired; + unsigned int sc_idx = 0, txsc_idx = 0; + struct atl_macsec_txsc *atl_txsc; + const struct macsec_secy *secy; + struct atl_hw *hw = &nic->hw; + enum atl_macsec_sc_sa sc_sa; + struct macsec_tx_sa *tx_sa; + unsigned char an = 0; + int ret; + int i; + + sc_sa = hw->macsec_cfg.sc_sa; + + ret = aq_mss_get_egress_sa_expired(hw, &egress_sa_expired); + if (unlikely(ret)) + return; + + ret = aq_mss_get_egress_sa_threshold_expired( + hw, &egress_sa_threshold_expired); + + for (i = 0; i < ATL_MACSEC_MAX_SA; i++) { + if (egress_sa_expired & BIT(i)) { + an = atl_sa_from_sa_idx(sc_sa, i); + sc_idx = atl_sc_idx_from_sa_idx(sc_sa, i); + txsc_idx = atl_get_txsc_idx_from_sc_idx(sc_sa, sc_idx); + if (txsc_idx < 0) + continue; + + atl_txsc = &hw->macsec_cfg.atl_txsc[txsc_idx]; + if (!(hw->macsec_cfg.txsc_idx_busy & BIT(txsc_idx))) { + netdev_warn( + nic->ndev, + "PN threshold expired on invalid TX SC"); + continue; + } + + secy = atl_txsc->sw_secy; + if (!netif_running(secy->netdev)) { + netdev_warn( + nic->ndev, + "PN threshold expired on down TX SC"); + continue; + } + + if (unlikely(!(atl_txsc->tx_sa_idx_busy & BIT(an)))) { + netdev_warn( + nic->ndev, + "PN threshold expired on invalid TX SA"); + continue; + } + + tx_sa = rcu_dereference_bh(secy->tx_sc.sa[an]); + macsec_pn_wrapped((struct macsec_secy *)secy, tx_sa); + } + } + + aq_mss_set_egress_sa_expired(hw, egress_sa_expired); + if (likely(!ret)) + aq_mss_set_egress_sa_threshold_expired( + hw, egress_sa_threshold_expired); +} + +void atl_macsec_work(struct atl_nic *nic) +{ + if ((nic->hw.mcp.caps_low & atl_fw2_macsec) == 0) + return; + + if (!netif_carrier_ok(nic->ndev)) + return; + + rtnl_lock(); + atl_check_txsa_expiration(nic); + rtnl_unlock(); +} +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.h new file mode 100644 index 000000000000..47e9768876e8 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_macsec.h @@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_MACSEC_H_ +#define _ATL_MACSEC_H_ + +#include +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + +#include "net/macsec.h" + +struct atl_hw; +struct atl_nic; + +#define ATL_MACSEC_MAX_SC 32 +#define ATL_MACSEC_MAX_SA 32 + +enum atl_macsec_sc_sa { + atl_macsec_sa_sc_4sa_8sc, + atl_macsec_sa_sc_not_used, + atl_macsec_sa_sc_2sa_16sc, + atl_macsec_sa_sc_1sa_32sc, +}; + +struct atl_macsec_common_stats { + /* Ingress Common Counters */ + struct { + u64 ctl_pkts; + u64 tagged_miss_pkts; + u64 untagged_miss_pkts; + u64 notag_pkts; + u64 untagged_pkts; + u64 bad_tag_pkts; + u64 no_sci_pkts; + u64 unknown_sci_pkts; + u64 ctrl_prt_pass_pkts; + u64 unctrl_prt_pass_pkts; + u64 ctrl_prt_fail_pkts; + u64 unctrl_prt_fail_pkts; + u64 too_long_pkts; + u64 igpoc_ctl_pkts; + u64 ecc_error_pkts; + u64 unctrl_hit_drop_redir; + } in; + + /* Egress Common Counters */ + struct { + u64 ctl_pkts; + u64 unknown_sa_pkts; + u64 untagged_pkts; + u64 too_long; + u64 ecc_error_pkts; + u64 unctrl_hit_drop_redir; + } out; +}; + +/* Ingress SA Counters */ +struct atl_macsec_rx_sa_stats { + u64 untagged_hit_pkts; + u64 ctrl_hit_drop_redir_pkts; + u64 not_using_sa; + u64 unused_sa; + u64 not_valid_pkts; + u64 invalid_pkts; + u64 ok_pkts; + u64 late_pkts; + u64 delayed_pkts; + u64 unchecked_pkts; + u64 validated_octets; + u64 decrypted_octets; +}; + +/* Egress SA Counters */ +struct atl_macsec_tx_sa_stats { + u64 sa_hit_drop_redirect; + u64 sa_protected2_pkts; + u64 sa_protected_pkts; + u64 sa_encrypted_pkts; +}; + +/* Egress SC Counters */ +struct atl_macsec_tx_sc_stats { + u64 sc_protected_pkts; + u64 sc_encrypted_pkts; + u64 sc_protected_octets; + u64 sc_encrypted_octets; +}; + +struct atl_macsec_txsc { + u32 hw_sc_idx; + unsigned long tx_sa_idx_busy; + const struct macsec_secy *sw_secy; + /* It is not OK to store key in driver but it is until ... */ + u8 tx_sa_key[MACSEC_NUM_AN][MACSEC_KEYID_LEN]; + struct atl_macsec_tx_sc_stats stats; + struct atl_macsec_tx_sa_stats tx_sa_stats[MACSEC_NUM_AN]; +}; + +struct atl_macsec_rxsc { + u32 hw_sc_idx; + unsigned long rx_sa_idx_busy; + const struct macsec_secy *sw_secy; + const struct macsec_rx_sc *sw_rxsc; + /* TODO: we shouldn't store keys in the driver */ + u8 rx_sa_key[MACSEC_NUM_AN][MACSEC_KEYID_LEN]; + struct atl_macsec_rx_sa_stats rx_sa_stats[MACSEC_NUM_AN]; +}; + +struct atl_macsec_cfg { + enum atl_macsec_sc_sa sc_sa; + /* Egress channel configuration */ + unsigned long txsc_idx_busy; + struct atl_macsec_txsc atl_txsc[ATL_MACSEC_MAX_SC]; + /* Ingress channel configuration */ + unsigned long rxsc_idx_busy; + struct atl_macsec_rxsc atl_rxsc[ATL_MACSEC_MAX_SC]; + struct atl_macsec_common_stats stats; +}; + +extern const struct macsec_ops atl_macsec_ops; + +int atl_init_macsec(struct atl_hw *hw); +void atl_macsec_work(struct atl_nic *nic); +int atl_macsec_rx_sa_cnt(struct atl_hw *hw); +int atl_macsec_tx_sc_cnt(struct atl_hw *hw); +int atl_macsec_tx_sa_cnt(struct atl_hw *hw); +int atl_macsec_update_stats(struct atl_hw *hw); + +#endif + +#endif /* _ATL_MACSEC_H_ */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_main.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_main.c new file mode 100644 index 000000000000..d2216e6825e5 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_main.c @@ -0,0 +1,1073 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_common.h" +#include +#include +#include +#include +#include +#include "atl_macsec.h" +#include "atl_ptp.h" + +const char atl_driver_name[] = "atlantic-fwd"; + +unsigned int atl_max_queues = ATL_MAX_QUEUES; +module_param_named(max_queues, atl_max_queues, uint, 0444); +unsigned int atl_max_queues_non_msi = 1; +module_param_named(max_queues_non_msi, atl_max_queues_non_msi, uint, 0444); + +static unsigned int atl_rx_mod = 15, atl_tx_mod = 15; +module_param_named(rx_mod, atl_rx_mod, uint, 0444); +module_param_named(tx_mod, atl_tx_mod, uint, 0444); + +static unsigned int atl_keep_link = 0; +module_param_named(keep_link, atl_keep_link, uint, 0644); + +static unsigned int atl_sleep_delay = 10000; +module_param_named(sleep_delay, atl_sleep_delay, uint, 0644); + +static unsigned int atl_rx_ring_size = ATL_RING_SIZE; +static unsigned int atl_tx_ring_size = ATL_RING_SIZE; +module_param_named(rx_ring_size, atl_rx_ring_size, uint, 0444); +module_param_named(tx_ring_size, atl_tx_ring_size, uint, 0444); + +static void atl_start_link(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + hw->link_state.force_off = 0; + hw->mcp.ops->set_link(hw, true); + set_bit(ATL_ST_UPDATE_LINK, &hw->state); + atl_schedule_work(nic); +} + +static void atl_stop_link(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + bool was_up = netif_carrier_ok(nic->ndev); + + hw->link_state.force_off = 1; + hw->mcp.ops->set_link(hw, true); + hw->link_state.link = 0; + netif_carrier_off(nic->ndev); + if (was_up) + pm_runtime_put(&nic->hw.pdev->dev); +} + +static int atl_start(struct atl_nic *nic) +{ + int ret = 0; + + atl_start_hw_global(nic); + atl_set_rx_mode(nic->ndev); + + if (atl_keep_link || netif_running(nic->ndev)) + atl_start_link(nic); + + if (netif_running(nic->ndev)) + ret = atl_start_rings(nic); + + if (ret && !atl_keep_link) + atl_stop_link(nic); + + /* if (ret) */ + /* goto out; */ + /* ret = atl_fwd_resume_rings(nic); */ + +/* out: */ + return ret; +} + +static void atl_stop(struct atl_nic *nic, bool drop_link) +{ + atl_stop_rings(nic); + + /* if (drop_link) { */ + /* atl_stop_fwd_rings(nic); */ + /* } */ + + if (drop_link) + atl_stop_link(nic); +} + +static int atl_open(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + int ret; + + pm_runtime_get_sync(&nic->hw.pdev->dev); + + if (!test_bit(ATL_ST_CONFIGURED, &nic->hw.state)) { + /* A previous atl_reconfigure() had failed. Try once more. */ + ret = atl_setup_datapath(nic); + if (ret) + goto out; + } + + ret = atl_alloc_rings(nic); + if (ret) + goto out; + + ret = netif_set_real_num_tx_queues(ndev, nic->nvecs); + if (ret) + goto free_rings; + ret = netif_set_real_num_rx_queues(ndev, nic->nvecs); + if (ret) + goto free_rings; + + ret = atl_start(nic); + if (ret) + goto free_rings; + + set_bit(ATL_ST_UP, &nic->hw.state); + + pm_runtime_put_sync(&nic->hw.pdev->dev); + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + atlfwd_nl_on_open(nic->ndev); +#endif + + return 0; + +free_rings: + atl_free_rings(nic); +out: + pm_runtime_put_noidle(&nic->hw.pdev->dev); + return ret; +} + +static int atl_close(struct atl_nic *nic, bool drop_link) +{ + /* atl_close() can be called a second time if + * atl_reconfigure() fails. Just return + */ + if (!test_and_clear_bit(ATL_ST_UP, &nic->hw.state)) + return 0; + + pm_runtime_get_sync(&nic->hw.pdev->dev); + + atl_stop(nic, drop_link); + atl_free_rings(nic); + if (drop_link) + atl_reconfigure(nic); + + pm_runtime_put_sync(&nic->hw.pdev->dev); + + return 0; +} + +static int atl_ndo_close(struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + + return atl_close(nic, !atl_keep_link); +} + +#ifndef ATL_HAVE_MINMAX_MTU + +static int atl_change_mtu(struct net_device *ndev, int mtu) +{ + struct atl_nic *nic = netdev_priv(ndev); + + if (mtu < 64 || mtu > nic->max_mtu) + return -EINVAL; + + ndev->mtu = mtu; + return 0; +} + +#endif + +static int atl_hwtstamp_config(struct atl_nic *nic, struct hwtstamp_config *config) +{ + if (config->flags) + return -EINVAL; + + switch (config->tx_type) { + case HWTSTAMP_TX_OFF: + case HWTSTAMP_TX_ON: + break; + default: + return -ERANGE; + } + + switch (config->rx_filter) { + case HWTSTAMP_FILTER_PTP_V2_L4_EVENT: + case HWTSTAMP_FILTER_PTP_V2_L4_SYNC: + case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ: + case HWTSTAMP_FILTER_PTP_V2_L2_EVENT: + case HWTSTAMP_FILTER_PTP_V2_L2_SYNC: + case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ: + case HWTSTAMP_FILTER_PTP_V2_SYNC: + case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ: + config->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT; + break; + case HWTSTAMP_FILTER_PTP_V2_EVENT: + case HWTSTAMP_FILTER_NONE: + break; + default: + return -ERANGE; + } + + return atl_ptp_hwtstamp_config_set(nic, config); +} + +static int atl_hwtstamp_set(struct atl_nic *nic, struct ifreq *ifr) +{ + struct hwtstamp_config config; + int err; + + if (!nic->ptp) + return -EOPNOTSUPP; + + if (copy_from_user(&config, ifr->ifr_data, sizeof(config))) + return -EFAULT; + + err = atl_hwtstamp_config(nic, &config); + if (err) + return err; + + return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ? + -EFAULT : 0; +} + +static int atl_hwtstamp_get(struct atl_nic *nic, struct ifreq *ifr) +{ + struct hwtstamp_config config; + + if (!nic->ptp) + return -EOPNOTSUPP; + + atl_ptp_hwtstamp_config_get(nic, &config); + return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ? + -EFAULT : 0; +} + +static int atl_ndo_ioctl(struct net_device *ndev, struct ifreq *ifr, int cmd) +{ + struct atl_nic *nic = netdev_priv(ndev); + + pm_runtime_get_sync(&nic->hw.pdev->dev); + + switch (cmd) { + case SIOCSHWTSTAMP: + return atl_hwtstamp_set(nic, ifr); + + case SIOCGHWTSTAMP: + return atl_hwtstamp_get(nic, ifr); + } + + pm_runtime_put(&nic->hw.pdev->dev); + + return -EOPNOTSUPP; +} + +static int atl_set_mac_address(struct net_device *ndev, void *priv) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_hw *hw = &nic->hw; + struct sockaddr *addr = priv; + + if (!is_valid_ether_addr(addr->sa_data)) + return -EADDRNOTAVAIL; + + ether_addr_copy(hw->mac_addr, addr->sa_data); + ether_addr_copy(ndev->dev_addr, addr->sa_data); + + if (netif_running(ndev) && pm_runtime_active(&nic->hw.pdev->dev)) + atl_set_uc_flt(hw, nic->rxf_mac.base_index, hw->mac_addr); + + return 0; +} + +static const struct net_device_ops atl_ndev_ops = { + .ndo_open = atl_open, + .ndo_stop = atl_ndo_close, + .ndo_start_xmit = atl_start_xmit, +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + .ndo_select_queue = atlfwd_nl_select_queue, +#endif + .ndo_vlan_rx_add_vid = atl_vlan_rx_add_vid, + .ndo_vlan_rx_kill_vid = atl_vlan_rx_kill_vid, + .ndo_set_rx_mode = atl_set_rx_mode, +#ifndef ATL_HAVE_MINMAX_MTU + .ndo_change_mtu = atl_change_mtu, +#endif + .ndo_set_features = atl_set_features, + .ndo_do_ioctl = atl_ndo_ioctl, + .ndo_set_mac_address = atl_set_mac_address, +#ifdef ATL_COMPAT_CAST_NDO_GET_STATS64 + .ndo_get_stats64 = (void *)atl_get_stats64, +#else + .ndo_get_stats64 = atl_get_stats64, +#endif +}; + +/* RTNL lock must be held */ +int atl_reconfigure(struct atl_nic *nic) +{ + struct net_device *ndev = nic->ndev; + int was_up = netif_running(ndev); + int ret = 0; + + nic->hw.mcp.ops->dump_cfg(&nic->hw); + + if (was_up) + atl_close(nic, false); + + atl_clear_datapath(nic); + + atl_fwd_suspend_rings(nic); + + ret = atl_hw_reset(&nic->hw); + if (ret) { + atl_nic_err("HW reset failed, re-trying\n"); + goto err; + } + + ret = atl_setup_datapath(nic); + if (ret) + goto err; + + /* Re-enable link interrupts disabled in atl_clear_datapath() */ + atl_intr_enable_non_ring(nic); + + /* Number of rings might have changed, re-init RSS + * redirection table. + */ + atl_init_rss_table(&nic->hw, nic->nvecs); + + if (was_up) { + ret = atl_open(ndev); + if (ret) + goto err; + } + + ret = atl_fwd_resume_rings(nic); + if (ret) + goto err; + + return 0; + +err: + if (was_up) + dev_close(ndev); + return ret; +} + +static struct workqueue_struct *atl_wq; + +void atl_schedule_work(struct atl_nic *nic) +{ + if (!test_and_set_bit(ATL_ST_WORK_SCHED, &nic->hw.state)) + queue_work(atl_wq, &nic->work); +} + +int atl_do_reset(struct atl_nic *nic) +{ + bool was_up = netif_running(nic->ndev); + struct atl_hw *hw = &nic->hw; + int ret; + + set_bit(ATL_ST_RESETTING, &hw->state); + + rtnl_lock(); + + hw->mcp.ops->dump_cfg(hw); + + atl_stop(nic, true); + + atl_fwd_suspend_rings(nic); + + ret = atl_hw_reset(hw); + if (ret) { + atl_nic_err("HW reset failed, re-trying\n"); + if (!test_and_set_bit(ATL_ST_DETACHED, &hw->state)) + netif_device_detach(nic->ndev); + goto out; + } + clear_bit(ATL_ST_RESETTING, &hw->state); + + if (was_up) { + ret = atl_start(nic); + if (ret) + goto out; + } + + ret = atl_fwd_resume_rings(nic); + if (ret) + goto out; + + if (test_and_clear_bit(ATL_ST_DETACHED, &hw->state)) + netif_device_attach(nic->ndev); + +out: + rtnl_unlock(); + return ret; +} + +static int atl_check_reset(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + bool reset; + + if (!test_bit(ATL_ST_ENABLED, &hw->state)) + /* We're suspending, postpone resets till resume */ + return 0; + + reset = test_and_clear_bit(ATL_ST_RESET_NEEDED, &hw->state); + + if (!reset) + return 0; + + return atl_do_reset(nic); +} + +int atl_fw_configure(struct atl_hw *hw) +{ + struct atl_nic *nic = container_of(hw, struct atl_nic, hw); + int ret; + + ret = hw->mcp.ops->set_mediadetect(hw, + !!(nic->priv_flags & ATL_PF_BIT(MEDIA_DETECT))); + if (ret && ret != -EOPNOTSUPP) + return ret; + ret = hw->mcp.ops->set_pad_stripping(hw, + !!(nic->priv_flags & ATL_PF_BIT(STRIP_PAD))); + if (ret && ret != -EOPNOTSUPP) + return ret; + ret = hw->mcp.ops->update_thermal(hw); + if (ret == -EOPNOTSUPP) + ret = 0; + + return ret; +} + +static void atl_work(struct work_struct *work) +{ + struct atl_nic *nic = container_of(work, struct atl_nic, work); + struct atl_hw *hw = &nic->hw; + int ret; + + atl_ptp_work(nic); + + clear_bit(ATL_ST_WORK_SCHED, &hw->state); + + atl_fw_watchdog(hw); + ret = atl_check_reset(nic); + if (ret) + goto out; + atl_refresh_link(nic); +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + atl_macsec_work(nic); +#endif +out: + if (test_bit(ATL_ST_ENABLED, &hw->state)) + mod_timer(&nic->work_timer, jiffies + HZ); +} + +static void atl_work_timer(struct timer_list *timer) +{ + struct atl_nic *nic = + container_of(timer, struct atl_nic, work_timer); + + atl_schedule_work(nic); +} + +static const struct pci_device_id atl_pci_tbl[] = { + { PCI_VDEVICE(AQUANTIA, 0x0001), ATL_UNKNOWN}, + { PCI_VDEVICE(AQUANTIA, 0xd107), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x07b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x87b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0xd108), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x08b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x88b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0xd109), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x09b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x89b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0xd100), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x00b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x80b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x11b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x91b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x12b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x92b1), ATL_ATLANTIC}, + { PCI_VDEVICE(AQUANTIA, 0x00c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x04c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x12c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x14c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x93c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x94c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x34c0), ATL_ANTIGUA}, + { PCI_VDEVICE(AQUANTIA, 0x11c0), ATL_ANTIGUA}, + {} +}; + +static uint8_t atl_def_rss_key[ATL_RSS_KEY_SIZE] = { + 0x1e, 0xad, 0x71, 0x87, 0x65, 0xfc, 0x26, 0x7d, + 0x0d, 0x45, 0x67, 0x74, 0xcd, 0x06, 0x1a, 0x18, + 0xb6, 0xc1, 0xf0, 0xc7, 0xbb, 0x18, 0xbe, 0xf8, + 0x19, 0x13, 0x4b, 0xa9, 0xd0, 0x3e, 0xfe, 0x70, + 0x25, 0x03, 0xab, 0x50, 0x6a, 0x8b, 0x82, 0x0c +}; + +static void atl_setup_rss(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + memcpy(hw->rss_key, atl_def_rss_key, sizeof(hw->rss_key)); + + atl_init_rss_table(hw, nic->nvecs); +} + +static int atl_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + /* Number of queues: + * Extra TX queue is used for redirection to FWD ring. + */ +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + const unsigned int txqs = atl_max_queues + 1; +#else + const unsigned int txqs = atl_max_queues; +#endif + const unsigned int rxqs = atl_max_queues; + int ret, pci_64 = 0; + struct net_device *ndev; + struct atl_nic *nic = NULL; + struct atl_hw *hw; + int disable_needed = 0; + + pm_runtime_set_active(&pdev->dev); + pm_runtime_forbid(&pdev->dev); + + ret = pci_enable_device_mem(pdev); + if (ret) + return ret; + + if (!dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64))) + pci_64 = 1; + else { + ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); + if (ret) { + dev_err(&pdev->dev, "Set DMA mask failed: %d\n", ret); + goto err_dma; + } + } + + ret = pci_request_mem_regions(pdev, atl_driver_name); + if (ret) { + dev_err(&pdev->dev, "Request PCI regions failed: %d\n", ret); + goto err_pci_reg; + } + + ndev = alloc_etherdev_mqs(sizeof(struct atl_nic), txqs, rxqs); + if (!ndev) { + ret = -ENOMEM; + goto err_alloc_ndev; + } + + SET_NETDEV_DEV(ndev, &pdev->dev); + nic = netdev_priv(ndev); + nic->ndev = ndev; + nic->hw.pdev = pdev; + spin_lock_init(&nic->stats_lock); + INIT_WORK(&nic->work, atl_work); + mutex_init(&nic->hw.mcp.lock); + +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + BLOCKING_INIT_NOTIFIER_HEAD(&nic->fwd.nh_clients); +#endif + + hw = &nic->hw; + __set_bit(ATL_ST_ENABLED, &hw->state); + hw->regs = ioremap(pci_resource_start(pdev, 0), + pci_resource_len(pdev, 0)); + if (!hw->regs) { + ret = -EIO; + goto err_ioremap; + } + + ret = atl_hwinit(hw, id->driver_data); + if (ret) + goto err_hwinit; + + hw->mcp.ops->set_default_link(hw); + hw->link_state.force_off = 1; + + pci_set_master(pdev); + + eth_platform_get_mac_address(&hw->pdev->dev, hw->mac_addr); + if (!is_valid_ether_addr(hw->mac_addr)) { + atl_dev_err("invalid MAC address: %*phC\n", ETH_ALEN, + hw->mac_addr); + /* XXX Workaround for bad MAC addr in efuse. Maybe + * switch to some predefined one later. + */ + eth_random_addr(hw->mac_addr); + /* ret = -EIO; */ + /* goto err_hwinit; */ + } + + ether_addr_copy(ndev->dev_addr, hw->mac_addr); + atl_dev_dbg("got MAC address: %pM\n", hw->mac_addr); + + ret = atl_ptp_init(nic); + if (ret) + goto err_ptp_init; + + nic->requested_nvecs = atl_max_queues; + nic->requested_tx_size = (atl_tx_ring_size & ~7); + nic->requested_rx_size = (atl_rx_ring_size & ~7); + nic->rx_intr_delay = atl_rx_mod; + nic->tx_intr_delay = atl_tx_mod; + + ret = atl_ptp_ring_alloc(nic); + if (ret) + goto err_ptp_alloc; + + ret = atl_setup_datapath(nic); + if (ret) + goto err_datapath; + + atl_setup_rss(nic); + + ndev->features |= NETIF_F_SG | NETIF_F_TSO | NETIF_F_TSO6 | + NETIF_F_RXCSUM | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | + NETIF_F_RXHASH; + + ndev->vlan_features |= ndev->features; + ndev->features |= NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_HW_VLAN_CTAG_TX | + NETIF_F_HW_VLAN_CTAG_FILTER; + + ndev->hw_features |= ndev->features | NETIF_F_RXALL | NETIF_F_LRO; + + if (pci_64) + ndev->features |= NETIF_F_HIGHDMA; + +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + if (hw->mcp.caps_low & atl_fw2_macsec) + ndev->features |= NETIF_F_HW_MACSEC; +#endif + + ndev->features |= NETIF_F_NTUPLE; + + ndev->priv_flags |= IFF_UNICAST_FLT; + + timer_setup(&nic->work_timer, &atl_work_timer, 0); + + hw->non_ring_intr_mask = BIT(ATL_NUM_NON_RING_IRQS) - 1; + ndev->netdev_ops = &atl_ndev_ops; +#if IS_ENABLED(CONFIG_MACSEC) && defined(NETIF_F_HW_MACSEC) + if (hw->mcp.caps_low & atl_fw2_macsec) + ndev->macsec_ops = &atl_macsec_ops, +#endif + ndev->mtu = 1500; +#ifdef ATL_HAVE_MINMAX_MTU + ndev->max_mtu = nic->max_mtu; +#endif + ndev->ethtool_ops = &atl_ethtool_ops; + ret = register_netdev(ndev); + if (ret) + goto err_register; + + ret = atl_ptp_register(nic); + if (ret) + goto err_ptp_register; + + pci_set_drvdata(pdev, nic); + netif_carrier_off(ndev); + + /* Safe to ignore ret value here. atl_start() only returns + * errors when rings are started. We could race with someone + * doing ifup on newly created netdev, but either they will + * succeed with grabbing RTNL first and handle ring-related + * errors there, or we will be first and just bring the + * global HW up. */ + rtnl_lock(); + atl_start(nic); + rtnl_unlock(); + + ret = atl_hwmon_init(nic); + if (ret) + goto err_hwmon_init; + + if (hw->mcp.caps_low & atl_fw2_wake_on_link_force) + pm_runtime_put_noidle(&pdev->dev); + + atl_intr_enable_non_ring(nic); + mod_timer(&nic->work_timer, jiffies + HZ); + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + atlfwd_nl_on_probe(nic->ndev); +#endif + + return 0; + +err_hwmon_init: + atl_stop(nic, true); +err_ptp_register: + unregister_netdev(nic->ndev); +err_register: + atl_clear_datapath(nic); +err_datapath: + atl_ptp_ring_free(nic); +err_ptp_alloc: + atl_ptp_free(nic); +err_ptp_init: +err_hwinit: + iounmap(hw->regs); +err_ioremap: + disable_needed = test_and_clear_bit(ATL_ST_ENABLED, &hw->state); + free_netdev(ndev); +err_alloc_ndev: + pci_release_regions(pdev); +err_pci_reg: +err_dma: + + if (!nic || disable_needed) + pci_disable_device(pdev); + return ret; +} + +static void atl_remove(struct pci_dev *pdev) +{ + struct atl_nic *nic = pci_get_drvdata(pdev); + int disable_needed; + bool wol_force; + + if (!nic) + return; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + atlfwd_nl_on_remove(nic->ndev); +#endif + + atl_stop(nic, true); + disable_needed = test_and_clear_bit(ATL_ST_ENABLED, &nic->hw.state); + wol_force = !!(nic->hw.mcp.caps_low & atl_fw2_wake_on_link_force); + del_timer_sync(&nic->work_timer); + cancel_work_sync(&nic->work); + atl_intr_disable_all(&nic->hw); + atl_ptp_unregister(nic); + unregister_netdev(nic->ndev); + +#if IS_ENABLED(CONFIG_ATLFWD_FWD) + atl_fwd_release_rings(nic); +#endif + + atl_clear_datapath(nic); + + atl_ptp_ring_free(nic); + atl_ptp_free(nic); + + iounmap(nic->hw.regs); + free_netdev(nic->ndev); + pci_release_regions(pdev); + + if (wol_force) + pm_runtime_get_sync(&pdev->dev); + + if (disable_needed) + pci_disable_device(pdev); +} + +static int atl_suspend_common(struct device *dev, unsigned int wol_mode) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + struct atl_hw *hw = &nic->hw; + bool rtnlocked; + int ret; + + rtnlocked = rtnl_trylock(); + hw->mcp.ops->dump_cfg(hw); + + atl_stop(nic, true); + + atl_intr_disable_non_ring(nic); + + atl_clear_rdm_cache(nic); + atl_clear_tdm_cache(nic); + + if (wol_mode) { + ret = hw->mcp.ops->enable_wol(hw, wol_mode); + if (ret) + atl_dev_err("Enable WoL failed: %d\n", -ret); + } else { + hw->mcp.ops->deinit(hw); + } + + clear_bit(ATL_ST_ENABLED, &hw->state); + cancel_work_sync(&nic->work); + clear_bit(ATL_ST_WORK_SCHED, &hw->state); + + pci_disable_device(pdev); + pci_save_state(pdev); + pci_prepare_to_sleep(pdev); + + if (rtnlocked) + rtnl_unlock(); + + return 0; +} + +static int atl_suspend_poweroff(struct device *dev) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + struct atl_hw *hw = &nic->hw; + + if (!test_and_set_bit(ATL_ST_DETACHED, &hw->state)) + netif_device_detach(nic->ndev); + + return atl_suspend_common(dev, hw->wol_mode); +} + +static int atl_freeze(struct device *dev) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + struct atl_hw *hw = &nic->hw; + + if (!test_and_set_bit(ATL_ST_DETACHED, &hw->state)) + netif_device_detach(nic->ndev); + + return atl_suspend_common(dev, 0); +} + +static int atl_resume_common(struct device *dev, bool deep) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + bool rtnlocked; + int ret; + + rtnlocked = rtnl_trylock(); + + pci_set_power_state(pdev, PCI_D0); + pci_restore_state(pdev); + + ret = pci_enable_device_mem(pdev); + if (ret) + goto exit; + + if (deep) { + atl_fwd_suspend_rings(nic); + ret = atl_hw_reset(&nic->hw); + if (ret) + goto exit; + } + + set_bit(ATL_ST_ENABLED, &nic->hw.state); + pci_set_master(pdev); + + if (test_bit(ATL_ST_UP, &nic->hw.state)) { + ret = atl_start(nic); + if (ret) + goto exit; + } + + if (deep) { + ret = atl_fwd_resume_rings(nic); + if (ret) + goto exit; + } + +exit: + if (rtnlocked) + rtnl_unlock(); + + return ret; +} + +static int atl_resume_restore(struct device *dev) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + + if (test_and_clear_bit(ATL_ST_DETACHED, &nic->hw.state)) + netif_device_attach(nic->ndev); + + return atl_resume_common(dev, true); +} + +static int atl_thaw(struct device *dev) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + + if (test_and_clear_bit(ATL_ST_DETACHED, &nic->hw.state)) + netif_device_attach(nic->ndev); + + return atl_resume_common(dev, false); +} + +static void atl_shutdown(struct pci_dev *pdev) +{ + struct atl_nic *nic = pci_get_drvdata(pdev); + atl_suspend_common(&pdev->dev, nic->hw.wol_mode); +} + +static int atl_pm_runtime_resume(struct device *dev) +{ + return atl_resume_common(dev, true); +} + +static int atl_pm_runtime_suspend(struct device *dev) +{ + return atl_suspend_common(dev, atl_fw_wake_on_link_rtpm); +} + +static int atl_pm_runtime_idle(struct device *dev) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct atl_nic *nic = pci_get_drvdata(pdev); + + /* pm_runtime_idle may be called during probe */ + if (!nic || nic->hw.pdev != pdev) + return -EBUSY; + + if (!netif_carrier_ok(nic->ndev)) { + pm_schedule_suspend(dev, atl_sleep_delay); + } + + return -EBUSY; +} + +const struct dev_pm_ops atl_pm_ops = { + .suspend = atl_suspend_poweroff, + .poweroff = atl_suspend_poweroff, + .freeze = atl_freeze, + .resume = atl_resume_restore, + .restore = atl_resume_restore, + .thaw = atl_thaw, + SET_RUNTIME_PM_OPS(atl_pm_runtime_suspend, atl_pm_runtime_resume, + atl_pm_runtime_idle) +}; + +static struct pci_driver atl_pci_ops = { + .name = atl_driver_name, + .id_table = atl_pci_tbl, + .probe = atl_probe, + .remove = atl_remove, + .shutdown = atl_shutdown, +#ifdef CONFIG_PM + .driver.pm = &atl_pm_ops, +#endif +}; + +struct atl_thermal atl_def_thermal; + +static bool atl_def_thermal_monitor = true, atl_def_thermal_throttle = false, + atl_def_thermal_ignore_lims = false; +module_param_named(thermal_monitor, atl_def_thermal_monitor, bool, 0444); +module_param_named(thermal_throttle, atl_def_thermal_throttle, bool, 0444); +module_param_named(thermal_ignore_limits, atl_def_thermal_ignore_lims, bool, 0444); + +static uint8_t atl_def_thermal_crit = 108, atl_def_thermal_high = 100, + atl_def_thermal_low = 80; +module_param_named(thermal_crit, atl_def_thermal_crit, byte, 0444); +module_param_named(thermal_high, atl_def_thermal_high, byte, 0444); +module_param_named(thermal_low, atl_def_thermal_low, byte, 0444); + +static int __init atl_module_init(void) +{ + struct atl_hw *hw = NULL; + int ret; + + if ((atl_tx_ring_size < 8) || (atl_tx_ring_size > ATL_MAX_RING_SIZE)) { + atl_dev_init_err( + "Bad atl_tx_ring_size value %d, must be between 8 and %d inclusive\n", + atl_tx_ring_size, ATL_MAX_RING_SIZE); + return -EINVAL; + } + if ((atl_rx_ring_size < 8) || (atl_rx_ring_size > ATL_MAX_RING_SIZE)) { + atl_dev_init_err( + "Bad atl_rx_ring_size value %d, must be between 8 and %d inclusive\n", + atl_rx_ring_size, ATL_MAX_RING_SIZE); + return -EINVAL; + } + + atl_def_thermal.flags = + atl_def_thermal_monitor << atl_thermal_monitor_shift | + atl_def_thermal_throttle << atl_thermal_throttle_shift | + atl_def_thermal_ignore_lims << atl_thermal_ignore_lims_shift; + atl_def_thermal.crit = atl_def_thermal_crit; + atl_def_thermal.high = atl_def_thermal_high; + atl_def_thermal.low = atl_def_thermal_low; + + ret = atl_verify_thermal_limits(hw, &atl_def_thermal); + if (ret) + return ret; + + if (atl_max_queues < 1 || atl_max_queues > ATL_MAX_QUEUES) { + atl_dev_init_err( + "Bad atl_max_queues value %d, must be between 1 and %d inclusive\n", + atl_max_queues, ATL_MAX_QUEUES); + return -EINVAL; + } + + if (atl_max_queues_non_msi < 1 || + atl_max_queues_non_msi > atl_max_queues) { + atl_dev_init_err( + "Bad atl_max_queues_non_msi value %d, must be between 1 and %d inclusive\n", + atl_max_queues_non_msi, atl_max_queues); + return -EINVAL; + } + + atl_wq = create_singlethread_workqueue(atl_driver_name); + if (!atl_wq) { + pr_err("%s: Couldn't create workqueue\n", atl_driver_name); + return -ENOMEM; + } + + ret = pci_register_driver(&atl_pci_ops); + if (ret) + goto err_pci_reg; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + ret = atlfwd_nl_init(); + if (ret) + goto err_fwd_netlink; +#endif + + return 0; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) +err_fwd_netlink: +#endif + pci_unregister_driver(&atl_pci_ops); +err_pci_reg: + destroy_workqueue(atl_wq); + return ret; +} +module_init(atl_module_init); + +static void __exit atl_module_exit(void) +{ +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + atlfwd_nl_exit(); +#endif + + pci_unregister_driver(&atl_pci_ops); + + if (atl_wq) { + destroy_workqueue(atl_wq); + atl_wq = NULL; + } +} +module_exit(atl_module_exit); + +MODULE_DEVICE_TABLE(pci, atl_pci_tbl); +MODULE_LICENSE("GPL v2"); +MODULE_VERSION(ATL_VERSION); +MODULE_AUTHOR("Aquantia Corp."); diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_mdio.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_mdio.h new file mode 100644 index 000000000000..c7d2179feb90 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_mdio.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_MDIO_H_ +#define _ATL_MDIO_H_ + +#include "atl_hw.h" + +int atl_mdio_hwsem_get(struct atl_hw *hw); +void atl_mdio_hwsem_put(struct atl_hw *hw); +int __atl_mdio_read(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t *val); +int atl_mdio_read(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t *val); +int __atl_mdio_write(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t val); +int atl_mdio_write(struct atl_hw *hw, uint8_t prtad, uint8_t mmd, + uint16_t addr, uint16_t val); + +#endif /* _ATL_MDIO_H_ */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.c new file mode 100644 index 000000000000..1348fbae126b --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.c @@ -0,0 +1,1328 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2014-2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_ptp.h" + +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) +#include +#include +#include +#endif + +#include "atl_ring.h" +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) +#include "atl_common.h" +#include "atl_ethtool.h" +#include "atl_hw_ptp.h" +#include "atl_ring_desc.h" + +#define ATL_PTP_TX_TIMEOUT (HZ * 10) + +#define POLL_SYNC_TIMER_MS 15 + +#define MAX_PTP_GPIO_COUNT 4 + +#define PTP_8TC_RING_IDX 8 +#define PTP_4TC_RING_IDX 16 +#define PTP_HWTS_RING_IDX 31 + +enum ptp_speed_offsets { + ptp_offset_idx_10 = 0, + ptp_offset_idx_100, + ptp_offset_idx_1000, + ptp_offset_idx_2500, + ptp_offset_idx_5000, + ptp_offset_idx_10000, +}; + +struct ptp_skb_ring { + struct sk_buff **buff; + spinlock_t lock; + unsigned int size; + unsigned int head; + unsigned int tail; +}; + +struct ptp_tx_timeout { + spinlock_t lock; + bool active; + unsigned long tx_start; +}; + +enum atl_ptp_queue { + ATL_PTPQ_PTP = 0, + ATL_PTPQ_HWTS = 1, + ATL_PTPQ_NUM, +}; + +struct atl_ptp { + struct atl_nic *nic; + struct hwtstamp_config hwtstamp_config; + spinlock_t ptp_lock; + spinlock_t ptp_ring_lock; + struct ptp_clock *ptp_clock; + struct ptp_clock_info ptp_info; + + atomic_t offset_egress; + atomic_t offset_ingress; + + struct ptp_tx_timeout ptp_tx_timeout; + + struct napi_struct *napi; + unsigned int idx_vector; + + struct atl_queue_vec qvec[ATL_PTPQ_NUM]; + + struct ptp_skb_ring skb_ring; + + s8 udp_filter_idx; + s8 eth_type_filter_idx; + + struct delayed_work poll_sync; + u32 poll_timeout_ms; + + bool extts_pin_enabled; + u64 last_sync1588_ts; +}; + +#define atl_for_each_ptp_qvec(ptp, qvec) \ + for (qvec = &ptp->qvec[0]; \ + qvec < &ptp->qvec[ATL_PTPQ_NUM]; qvec++) + +struct ptp_tm_offset { + unsigned int mbps; + int egress; + int ingress; +}; + +static struct ptp_tm_offset ptp_offset[6]; + +static int __atl_ptp_skb_put(struct ptp_skb_ring *ring, struct sk_buff *skb) +{ + unsigned int next_head = (ring->head + 1) % ring->size; + + if (next_head == ring->tail) + return -ENOMEM; + + ring->buff[ring->head] = skb_get(skb); + ring->head = next_head; + + return 0; +} + +static int atl_ptp_skb_put(struct ptp_skb_ring *ring, struct sk_buff *skb) +{ + unsigned long flags; + int ret; + + spin_lock_irqsave(&ring->lock, flags); + ret = __atl_ptp_skb_put(ring, skb); + spin_unlock_irqrestore(&ring->lock, flags); + + return ret; +} + +static struct sk_buff *__atl_ptp_skb_get(struct ptp_skb_ring *ring) +{ + struct sk_buff *skb; + + if (ring->tail == ring->head) + return NULL; + + skb = ring->buff[ring->tail]; + ring->tail = (ring->tail + 1) % ring->size; + + return skb; +} + +static struct sk_buff *atl_ptp_skb_get(struct ptp_skb_ring *ring) +{ + unsigned long flags; + struct sk_buff *skb; + + spin_lock_irqsave(&ring->lock, flags); + skb = __atl_ptp_skb_get(ring); + spin_unlock_irqrestore(&ring->lock, flags); + + return skb; +} + +static unsigned int atl_ptp_skb_buf_len(struct ptp_skb_ring *ring) +{ + unsigned long flags; + unsigned int len; + + spin_lock_irqsave(&ring->lock, flags); + len = (ring->head >= ring->tail) ? + ring->head - ring->tail : + ring->size - ring->tail + ring->head; + spin_unlock_irqrestore(&ring->lock, flags); + + return len; +} + +static int atl_ptp_skb_ring_init(struct ptp_skb_ring *ring, unsigned int size) +{ + struct sk_buff **buff = kcalloc(size, sizeof(*buff), GFP_KERNEL); + + if (!buff) + return -ENOMEM; + + spin_lock_init(&ring->lock); + + ring->buff = buff; + ring->size = size; + ring->head = 0; + ring->tail = 0; + + return 0; +} + +static void atl_ptp_skb_ring_clean(struct ptp_skb_ring *ring) +{ + struct sk_buff *skb; + + while ((skb = atl_ptp_skb_get(ring)) != NULL) + dev_kfree_skb_any(skb); +} + +static void atl_ptp_skb_ring_release(struct ptp_skb_ring *ring) +{ + if (ring->buff) { + atl_ptp_skb_ring_clean(ring); + kfree(ring->buff); + ring->buff = NULL; + } +} + +static void atl_ptp_tx_timeout_init(struct ptp_tx_timeout *timeout) +{ + spin_lock_init(&timeout->lock); + timeout->active = false; +} + +static void atl_ptp_tx_timeout_start(struct atl_ptp *ptp) +{ + struct ptp_tx_timeout *timeout = &ptp->ptp_tx_timeout; + unsigned long flags; + + spin_lock_irqsave(&timeout->lock, flags); + timeout->active = true; + timeout->tx_start = jiffies; + spin_unlock_irqrestore(&timeout->lock, flags); +} + +static void atl_ptp_tx_timeout_update(struct atl_ptp *ptp) +{ + if (!atl_ptp_skb_buf_len(&ptp->skb_ring)) { + struct ptp_tx_timeout *timeout = &ptp->ptp_tx_timeout; + unsigned long flags; + + spin_lock_irqsave(&timeout->lock, flags); + timeout->active = false; + spin_unlock_irqrestore(&timeout->lock, flags); + } +} + +static void atl_ptp_tx_timeout_check(struct atl_ptp *ptp) +{ + struct ptp_tx_timeout *timeout = &ptp->ptp_tx_timeout; + struct atl_nic *nic = ptp->nic; + unsigned long flags; + bool timeout_flag; + + timeout_flag = false; + + spin_lock_irqsave(&timeout->lock, flags); + if (timeout->active) { + timeout_flag = time_is_before_jiffies(timeout->tx_start + + ATL_PTP_TX_TIMEOUT); + /* reset active flag if timeout detected */ + if (timeout_flag) + timeout->active = false; + } + spin_unlock_irqrestore(&timeout->lock, flags); + + if (timeout_flag) { + atl_nic_err("PTP Timeout. Clearing Tx Timestamp SKBs\n"); + atl_ptp_skb_ring_clean(&ptp->skb_ring); + } +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 3, 0) +/* atl_ptp_adjfine + * @ptp_info: the ptp clock structure + * @ppb: parts per billion adjustment from base + * + * adjust the frequency of the ptp cycle counter by the + * indicated ppb from the base frequency. + */ +static int atl_ptp_adjfine(struct ptp_clock_info *ptp_info, long scaled_ppm) +{ + struct atl_ptp *ptp = container_of(ptp_info, struct atl_ptp, ptp_info); + struct atl_nic *nic = ptp->nic; + + hw_atl_adj_clock_freq(&nic->hw, scaled_ppm_to_ppb(scaled_ppm)); + + return 0; +} +#endif + +/* atl_ptp_adjfreq + * @ptp_info: the ptp clock structure + * @ppb: parts per billion adjustment from base + * + * adjust the frequency of the ptp cycle counter by the + * indicated ppb from the base frequency. + */ +static int atl_ptp_adjfreq(struct ptp_clock_info *ptp_info, s32 ppb) +{ + struct atl_ptp *ptp = container_of(ptp_info, struct atl_ptp, ptp_info); + struct atl_nic *nic = ptp->nic; + + hw_atl_adj_clock_freq(&nic->hw, ppb); + + return 0; +} + +/* atl_ptp_adjtime + * @ptp_info: the ptp clock structure + * @delta: offset to adjust the cycle counter by + * + * adjust the timer by resetting the timecounter structure. + */ +static int atl_ptp_adjtime(struct ptp_clock_info *ptp_info, s64 delta) +{ + struct atl_ptp *ptp = container_of(ptp_info, struct atl_ptp, ptp_info); + struct atl_nic *nic = ptp->nic; + unsigned long flags; + + spin_lock_irqsave(&ptp->ptp_lock, flags); + hw_atl_adj_sys_clock(&nic->hw, delta); + spin_unlock_irqrestore(&ptp->ptp_lock, flags); + + return 0; +} + +/* atl_ptp_gettime + * @ptp_info: the ptp clock structure + * @ts: timespec structure to hold the current time value + * + * read the timecounter and return the correct value on ns, + * after converting it into a struct timespec. + */ +static int atl_ptp_gettime(struct ptp_clock_info *ptp_info, struct timespec64 *ts) +{ + struct atl_ptp *ptp = container_of(ptp_info, struct atl_ptp, ptp_info); + struct atl_nic *nic = ptp->nic; + unsigned long flags; + u64 ns; + + spin_lock_irqsave(&ptp->ptp_lock, flags); + hw_atl_get_ptp_ts(&nic->hw, &ns); + spin_unlock_irqrestore(&ptp->ptp_lock, flags); + + *ts = ns_to_timespec64(ns); + + return 0; +} + +/* atl_ptp_settime + * @ptp_info: the ptp clock structure + * @ts: the timespec containing the new time for the cycle counter + * + * reset the timecounter to use a new base value instead of the kernel + * wall timer value. + */ +static int atl_ptp_settime(struct ptp_clock_info *ptp_info, + const struct timespec64 *ts) +{ + struct atl_ptp *ptp = container_of(ptp_info, struct atl_ptp, ptp_info); + struct atl_nic *nic = ptp->nic; + unsigned long flags; + u64 ns = timespec64_to_ns(ts); + u64 now; + + spin_lock_irqsave(&ptp->ptp_lock, flags); + hw_atl_get_ptp_ts(&nic->hw, &now); + hw_atl_adj_sys_clock(&nic->hw, (s64)ns - (s64)now); + + spin_unlock_irqrestore(&ptp->ptp_lock, flags); + + return 0; +} + +static void atl_ptp_convert_to_hwtstamp(struct skb_shared_hwtstamps *hwtstamp, + u64 timestamp) +{ + memset(hwtstamp, 0, sizeof(*hwtstamp)); + hwtstamp->hwtstamp = ns_to_ktime(timestamp); +} + +static int atl_ptp_hw_pin_conf(struct atl_nic *nic, u32 pin_index, u64 start, + u64 period) +{ + if (period) + atl_nic_dbg("Enable GPIO %d pulsing, start time %llu, period %u\n", + pin_index, start, (u32)period); + else + atl_nic_dbg("Disable GPIO %d pulsing, start time %llu, period %u\n", + pin_index, start, (u32)period); + + /* Notify hardware of request to being sending pulses. + * If period is ZERO then pulsen is disabled. + */ + hw_atl_gpio_pulse(&nic->hw, pin_index, start, (u32)period); + + return 0; +} + +static int atl_ptp_perout_pin_configure(struct ptp_clock_info *ptp_clock, + struct ptp_clock_request *rq, int on) +{ + struct atl_ptp *ptp = container_of(ptp_clock, struct atl_ptp, ptp_info); + struct ptp_clock_time *t = &rq->perout.period; + struct ptp_clock_time *s = &rq->perout.start; + u32 pin_index = rq->perout.index; + struct atl_nic *nic = ptp->nic; + u64 start, period; + + /* verify the request channel is there */ + if (pin_index >= ptp_clock->n_per_out) + return -EINVAL; + + /* we cannot support periods greater + * than 4 seconds due to reg limit + */ + if (t->sec > 4 || t->sec < 0) + return -ERANGE; + + /* convert to unsigned 64b ns, + * verify we can put it in a 32b register + */ + period = on ? t->sec * NSEC_PER_SEC + t->nsec : 0; + + /* verify the value is in range supported by hardware */ + if (period > U32_MAX) + return -ERANGE; + /* convert to unsigned 64b ns */ + /* TODO convert to AQ time */ + start = on ? s->sec * NSEC_PER_SEC + s->nsec : 0; + + atl_ptp_hw_pin_conf(nic, pin_index, start, period); + + return 0; +} + +static int atl_ptp_pps_pin_configure(struct ptp_clock_info *ptp_clock, + struct ptp_clock_request *rq, int on) +{ + struct atl_ptp *ptp = container_of(ptp_clock, struct atl_ptp, ptp_info); + struct atl_nic *nic = ptp->nic; + u64 start, period; + u32 pin_index = 0; + u32 rest = 0; + + /* verify the request channel is there */ + if (pin_index >= ptp_clock->n_per_out) + return -EINVAL; + + hw_atl_get_ptp_ts(&nic->hw, &start); + div_u64_rem(start, NSEC_PER_SEC, &rest); + period = on ? NSEC_PER_SEC : 0; /* PPS - pulse per second */ + start = on ? start - rest + NSEC_PER_SEC * + (rest > 990000000LL ? 2 : 1) : 0; + + atl_ptp_hw_pin_conf(nic, pin_index, start, period); + + return 0; +} + +static void atl_ptp_extts_pin_ctrl(struct atl_ptp *ptp) +{ + struct atl_nic *nic = ptp->nic; + u32 enable = ptp->extts_pin_enabled; + + hw_atl_extts_gpio_enable(&nic->hw, 0, enable); +} + +static int atl_ptp_extts_pin_configure(struct ptp_clock_info *ptp_clock, + struct ptp_clock_request *rq, int on) +{ + struct atl_ptp *ptp = container_of(ptp_clock, struct atl_ptp, ptp_info); + + u32 pin_index = rq->extts.index; + + if (pin_index >= ptp_clock->n_ext_ts) + return -EINVAL; + + ptp->extts_pin_enabled = !!on; + if (on) { + ptp->poll_timeout_ms = POLL_SYNC_TIMER_MS; + cancel_delayed_work_sync(&ptp->poll_sync); + schedule_delayed_work(&ptp->poll_sync, + msecs_to_jiffies(ptp->poll_timeout_ms)); + } + + atl_ptp_extts_pin_ctrl(ptp); + return 0; +} + +/* atl_ptp_gpio_feature_enable + * @ptp: the ptp clock structure + * @rq: the requested feature to change + * @on: whether to enable or disable the feature + */ +static int atl_ptp_gpio_feature_enable(struct ptp_clock_info *ptp, + struct ptp_clock_request *rq, int on) +{ + switch (rq->type) { + case PTP_CLK_REQ_EXTTS: + return atl_ptp_extts_pin_configure(ptp, rq, on); + case PTP_CLK_REQ_PEROUT: + return atl_ptp_perout_pin_configure(ptp, rq, on); + case PTP_CLK_REQ_PPS: + return atl_ptp_pps_pin_configure(ptp, rq, on); + default: + return -EOPNOTSUPP; + } + + return 0; +} + +/* atl_ptp_verify + * @ptp: the ptp clock structure + * @pin: index of the pin in question + * @func: the desired function to use + * @chan: the function channel index to use + */ +static int atl_ptp_verify(struct ptp_clock_info *ptp, unsigned int pin, + enum ptp_pin_function func, unsigned int chan) +{ + /* verify the requested pin is there */ + if (!ptp->pin_config || pin >= ptp->n_pins) + return -EINVAL; + + /* enforce locked channels, no changing them */ + if (chan != ptp->pin_config[pin].chan) + return -EINVAL; + + /* we want to keep the functions locked as well */ + if (func != ptp->pin_config[pin].func) + return -EINVAL; + + return 0; +} + +/* atl_ptp_rx_hwtstamp - utility function which checks for RX time stamp + * @skb: particular skb to send timestamp with + * + * if the timestamp is valid, we convert it into the timecounter ns + * value, then store that result into the hwtstamps structure which + * is passed up the network stack + */ +static void atl_ptp_rx_hwtstamp(struct atl_ptp *ptp, struct sk_buff *skb, + u64 timestamp) +{ + timestamp -= atomic_read(&ptp->offset_ingress); + atl_ptp_convert_to_hwtstamp(skb_hwtstamps(skb), timestamp); +} + +static int atl_ptp_ring_index(enum atl_ptp_queue ptp_queue) +{ + switch (ptp_queue) { + case ATL_PTPQ_PTP: + /* multi-TC is not supported in FWD driver, so tc mode is + * always set to 4 TCs (each with 8 queues) for now + */ + return PTP_4TC_RING_IDX; + case ATL_PTPQ_HWTS: + return PTP_HWTS_RING_IDX; + default: + break; + } + + WARN_ONCE(1, "Invalid ptp_queue"); + return 0; +} + +static int atl_ptp_poll(struct napi_struct *napi, int budget) +{ + struct atl_queue_vec *qvec = container_of(napi, struct atl_queue_vec, napi); + struct atl_ptp *ptp = qvec->nic->ptp; + int work_done = 0; + + /* Processing PTP TX and RX traffic */ + work_done = atl_poll_qvec(&ptp->qvec[ATL_PTPQ_PTP], budget); + + /* Processing HW_TIMESTAMP RX traffic */ + atl_clean_hwts_rx(&ptp->qvec[ATL_PTPQ_HWTS].rx, budget); + + if (work_done < budget) { + napi_complete_done(ptp->napi, work_done); + atl_intr_enable(&qvec->nic->hw, BIT(atl_qvec_intr(qvec))); + /* atl_set_intr_throttle(&nic->hw, qvec->idx); */ + } + + return work_done; +} + +static irqreturn_t atl_ptp_irq(int irq, void *private) +{ + struct atl_ptp *ptp = private; + int err = 0; + + if (!ptp) { + err = -EINVAL; + goto err_exit; + } + napi_schedule_irqoff(ptp->napi); + +err_exit: + return err >= 0 ? IRQ_HANDLED : IRQ_NONE; +} + +static struct ptp_clock_info atl_ptp_clock = { + .owner = THIS_MODULE, + .name = "atlantic ptp", + .max_adj = 999999999, + .n_ext_ts = 0, + .pps = 0, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 3, 0) + .adjfine = atl_ptp_adjfine, +#endif + .adjfreq = atl_ptp_adjfreq, + .adjtime = atl_ptp_adjtime, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 1, 0) + .gettime64 = atl_ptp_gettime, + .settime64 = atl_ptp_settime, +#else + .gettime = atl_ptp_gettime, + .settime = atl_ptp_settime, +#endif + .n_per_out = 0, + .enable = atl_ptp_gpio_feature_enable, + .n_pins = 0, + .verify = atl_ptp_verify, + .pin_config = NULL, +}; + +#define ptp_offset_init(__idx, __mbps, __egress, __ingress) do { \ + ptp_offset[__idx].mbps = (__mbps); \ + ptp_offset[__idx].egress = (__egress); \ + ptp_offset[__idx].ingress = (__ingress); } \ + while (0) + +static void atl_ptp_offset_init_from_fw(const struct atl_ptp_offset_info *offsets) +{ + int i; + + /* Load offsets for PTP */ + for (i = 0; i < ARRAY_SIZE(ptp_offset); i++) { + switch (i) { + /* 100M */ + case ptp_offset_idx_100: + ptp_offset_init(i, 100, + offsets->egress_100, + offsets->ingress_100); + break; + /* 1G */ + case ptp_offset_idx_1000: + ptp_offset_init(i, 1000, + offsets->egress_1000, + offsets->ingress_1000); + break; + /* 2.5G */ + case ptp_offset_idx_2500: + ptp_offset_init(i, 2500, + offsets->egress_2500, + offsets->ingress_2500); + break; + /* 5G */ + case ptp_offset_idx_5000: + ptp_offset_init(i, 5000, + offsets->egress_5000, + offsets->ingress_5000); + break; + /* 10G */ + case ptp_offset_idx_10000: + ptp_offset_init(i, 10000, + offsets->egress_10000, + offsets->ingress_10000); + break; + } + } +} + +static void atl_ptp_offset_init(const struct atl_ptp_offset_info *offsets) +{ + memset(ptp_offset, 0, sizeof(ptp_offset)); + + atl_ptp_offset_init_from_fw(offsets); +} + +static void atl_ptp_gpio_init(struct atl_nic *nic, + struct ptp_clock_info *info, + enum atl_gpio_pin_function *gpio_pin) +{ + struct ptp_pin_desc pin_desc[MAX_PTP_GPIO_COUNT]; + u32 extts_pin_cnt = 0; + u32 out_pin_cnt = 0; + u32 i; + + memset(pin_desc, 0, sizeof(pin_desc)); + + for (i = 0; i < MAX_PTP_GPIO_COUNT - 1; i++) { + if (gpio_pin[i] == + (GPIO_PIN_FUNCTION_PTP0 + out_pin_cnt)) { + snprintf(pin_desc[out_pin_cnt].name, + sizeof(pin_desc[out_pin_cnt].name), + "AQ_GPIO%d", i); + pin_desc[out_pin_cnt].index = out_pin_cnt; + pin_desc[out_pin_cnt].chan = out_pin_cnt; + pin_desc[out_pin_cnt++].func = PTP_PF_PEROUT; + } + } + + info->n_per_out = out_pin_cnt; + + if (nic->hw.mcp.caps_ex & atl_fw2_ex_caps_phy_ctrl_ts_pin) { + extts_pin_cnt += 1; + + snprintf(pin_desc[out_pin_cnt].name, + sizeof(pin_desc[out_pin_cnt].name), + "AQ_GPIO%d", out_pin_cnt); + pin_desc[out_pin_cnt].index = out_pin_cnt; + pin_desc[out_pin_cnt].chan = 0; + pin_desc[out_pin_cnt].func = PTP_PF_EXTTS; + } + + info->n_pins = out_pin_cnt + extts_pin_cnt; + info->n_ext_ts = extts_pin_cnt; + + if (!info->n_pins) + return; + + info->pin_config = kcalloc(info->n_pins, sizeof(struct ptp_pin_desc), + GFP_KERNEL); + + if (!info->pin_config) + return; + + memcpy(info->pin_config, &pin_desc, + sizeof(struct ptp_pin_desc) * info->n_pins); +} + +/* PTP external GPIO nanoseconds count */ +static uint64_t atl_ptp_get_sync1588_ts(struct atl_nic *nic) +{ + u64 ts = 0; + + hw_atl_get_sync_ts(&nic->hw, &ts); + + return ts; +} + +static void atl_ptp_start_work(struct atl_ptp *ptp) +{ + if (ptp->extts_pin_enabled) { + ptp->poll_timeout_ms = POLL_SYNC_TIMER_MS; + ptp->last_sync1588_ts = atl_ptp_get_sync1588_ts(ptp->nic); + schedule_delayed_work(&ptp->poll_sync, + msecs_to_jiffies(ptp->poll_timeout_ms)); + } +} + +static bool atl_ptp_sync_ts_updated(struct atl_ptp *ptp, u64 *new_ts) +{ + struct atl_nic *nic = ptp->nic; + u64 sync_ts2; + u64 sync_ts; + + sync_ts = atl_ptp_get_sync1588_ts(nic); + + if (sync_ts != ptp->last_sync1588_ts) { + sync_ts2 = atl_ptp_get_sync1588_ts(nic); + if (sync_ts != sync_ts2) { + sync_ts = sync_ts2; + sync_ts2 = atl_ptp_get_sync1588_ts(nic); + if (sync_ts != sync_ts2) { + atl_nic_err("%s: Unable to get correct GPIO TS", + __func__); + sync_ts = 0; + } + } + + *new_ts = sync_ts; + return true; + } + return false; +} + +static int atl_ptp_check_sync1588(struct atl_ptp *ptp) +{ + struct atl_nic *nic = ptp->nic; + u64 sync_ts; + + /* Sync1588 pin was triggered */ + if (atl_ptp_sync_ts_updated(ptp, &sync_ts)) { + if (ptp->extts_pin_enabled) { + struct ptp_clock_event ptp_event; + u64 time = 0; + + hw_atl_ts_to_sys_clock(&nic->hw, sync_ts, &time); + ptp_event.index = ptp->ptp_info.n_pins - 1; + ptp_event.timestamp = time; + + ptp_event.type = PTP_CLOCK_EXTTS; + ptp_clock_event(ptp->ptp_clock, &ptp_event); + } + + ptp->last_sync1588_ts = sync_ts; + } + + return 0; +} + +static void atl_ptp_poll_sync_work_cb(struct work_struct *w) +{ + struct delayed_work *dw = to_delayed_work(w); + struct atl_ptp *ptp = container_of(dw, struct atl_ptp, poll_sync); + + atl_ptp_check_sync1588(ptp); + + if (ptp->extts_pin_enabled) { + unsigned long timeout = msecs_to_jiffies(ptp->poll_timeout_ms); + + schedule_delayed_work(&ptp->poll_sync, timeout); + } +} + +#endif /* IS_REACHABLE(CONFIG_PTP_1588_CLOCK) */ + +void atl_ptp_tm_offset_set(struct atl_nic *nic, unsigned int mbps) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + int i, egress, ingress; + + if (!ptp) + return; + + egress = 0; + ingress = 0; + + for (i = 0; i < ARRAY_SIZE(ptp_offset); i++) { + if (mbps == ptp_offset[i].mbps) { + egress = ptp_offset[i].egress; + ingress = ptp_offset[i].ingress; + break; + } + } + + atomic_set(&ptp->offset_egress, egress); + atomic_set(&ptp->offset_ingress, ingress); +#endif +} + +/* atl_ptp_tx_hwtstamp - utility function which checks for TX time stamp + * + * if the timestamp is valid, we convert it into the timecounter ns + * value, then store that result into the hwtstamps structure which + * is passed up the network stack + */ +void atl_ptp_tx_hwtstamp(struct atl_nic *nic, u64 timestamp) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct sk_buff *skb = atl_ptp_skb_get(&ptp->skb_ring); + struct skb_shared_hwtstamps hwtstamp; + + if (!skb) { + atl_nic_err("have timestamp but tx_queues empty\n"); + return; + } + + timestamp += atomic_read(&ptp->offset_egress); + atl_ptp_convert_to_hwtstamp(&hwtstamp, timestamp); + do { + skb_tstamp_tx(skb, &hwtstamp); + dev_kfree_skb_any(skb); + skb = atl_ptp_skb_get(&ptp->skb_ring); + } while (skb); + + atl_ptp_tx_timeout_update(ptp); +#endif +} + +void atl_ptp_hwtstamp_config_get(struct atl_nic *nic, + struct hwtstamp_config *config) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + + *config = ptp->hwtstamp_config; +#endif +} + +int atl_ptp_hwtstamp_config_set(struct atl_nic *nic, + struct hwtstamp_config *config) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + static u32 ntuple_cmd = + ATL_NTC_PROTO | + ATL_NTC_L4_UDP | + ATL_NTC_DP | + ATL_RXF_ACT_TOHOST | + ATL_NTC_RXQ; + u32 ntuple_vec_idx = + ((ptp->qvec[ATL_PTPQ_PTP].idx << ATL_NTC_RXQ_SHIFT) & ATL_NTC_RXQ_MASK); + static u32 etype_cmd = + ETH_P_1588 | + ATL_RXF_ACT_TOHOST | + ATL_ETYPE_RXQ; + u32 etype_vec_idx = + ((ptp->qvec[ATL_PTPQ_PTP].idx << ATL_ETYPE_RXQ_SHIFT) & ATL_ETYPE_RXQ_MASK); + + if (config->tx_type == HWTSTAMP_TX_ON || + config->rx_filter == HWTSTAMP_FILTER_PTP_V2_EVENT) { + atl_write(&nic->hw, ATL_NTUPLE_DPORT(ptp->udp_filter_idx), + PTP_EV_PORT); + atl_write(&nic->hw, ATL_NTUPLE_CTRL(ptp->udp_filter_idx), + ATL_NTC_EN | ntuple_cmd | ntuple_vec_idx); + + atl_write(&nic->hw, ATL_RX_ETYPE_FLT(ptp->eth_type_filter_idx), + ATL_ETYPE_EN | etype_cmd | etype_vec_idx); + + nic->hw.link_state.ptp_datapath_up = true; + } else { + atl_write(&nic->hw, ATL_NTUPLE_CTRL(ptp->udp_filter_idx), ntuple_cmd); + + atl_write(&nic->hw, ATL_RX_ETYPE_FLT(ptp->eth_type_filter_idx), 0); + + nic->hw.link_state.ptp_datapath_up = false; + } + + ptp->hwtstamp_config = *config; +#endif + + return 0; +} + +int atl_ptp_qvec_intr(struct atl_queue_vec *qvec) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + int i; + + for (i = 0; i != ATL_PTPQ_NUM; i++) { + if (qvec->idx == atl_ptp_ring_index(i)) + return ATL_NUM_NON_RING_IRQS - 1; + } +#endif + + WARN_ONCE(1, "Not a PTP queue vector"); + return ATL_NUM_NON_RING_IRQS; +} + +bool atl_is_ptp_ring(struct atl_nic *nic, struct atl_desc_ring *ring) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return false; + + return &ptp->qvec[ATL_PTPQ_PTP].tx == ring || + &ptp->qvec[ATL_PTPQ_PTP].rx == ring || + &ptp->qvec[ATL_PTPQ_HWTS].rx == ring; +#else + return false; +#endif +} + +u16 atl_ptp_extract_ts(struct atl_nic *nic, struct sk_buff *skb, u8 *p, + unsigned int len) +{ + u16 ret = 0; +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + u64 timestamp = 0; + + ret = hw_atl_rx_extract_ts(&nic->hw, p, len, ×tamp); + if (ret > 0) + atl_ptp_rx_hwtstamp(ptp, skb, timestamp); +#endif + + return ret; +} + +netdev_tx_t atl_ptp_start_xmit(struct atl_nic *nic, struct sk_buff *skb) +{ + netdev_tx_t err = NETDEV_TX_OK; +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct atl_desc_ring *ring; + unsigned long irq_flags; + + ring = &ptp->qvec[ATL_PTPQ_PTP].tx; + + if (skb->len <= 0) { + dev_kfree_skb_any(skb); + goto err_exit; + } + + if (atl_tx_full(ring, skb_shinfo(skb)->nr_frags + 4)) { + /* Drop packet because it doesn't make sence to delay it */ + dev_kfree_skb_any(skb); + goto err_exit; + } + + err = atl_ptp_skb_put(&ptp->skb_ring, skb); + if (err) { + atl_nic_err("SKB Ring is overflow!\n"); + return NETDEV_TX_BUSY; + } + skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; + atl_ptp_tx_timeout_start(ptp); + skb_tx_timestamp(skb); + + spin_lock_irqsave(&ptp->ptp_ring_lock, irq_flags); + err = atl_map_skb(skb, ring); + spin_unlock_irqrestore(&ptp->ptp_ring_lock, irq_flags); + +err_exit: +#endif + return err; +} + +void atl_ptp_work(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return; + + atl_ptp_tx_timeout_check(ptp); +#endif +} + +int atl_ptp_irq_alloc(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct pci_dev *pdev = nic->hw.pdev; + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return 0; + + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) { + return request_irq(pci_irq_vector(pdev, ptp->idx_vector), + atl_ptp_irq, 0, nic->ndev->name, ptp); + } + + return -EINVAL; +#else + return 0; +#endif +} + +void atl_ptp_irq_free(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_hw *hw = &nic->hw; + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return; + + atl_intr_disable(hw, BIT(ptp->idx_vector)); + free_irq(pci_irq_vector(hw->pdev, ptp->idx_vector), ptp); +#endif +} + +int atl_ptp_ring_alloc(struct atl_nic *nic) +{ + int err = 0; +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct atl_queue_vec *qvec; + int i; + + if (!ptp) + return 0; + + for (i = 0; i != ATL_PTPQ_NUM; i++) { + qvec = &ptp->qvec[i]; + + atl_init_qvec(nic, qvec, atl_ptp_ring_index(i)); + } + + atl_for_each_ptp_qvec(ptp, qvec) { + err = atl_alloc_qvec(qvec); + if (err) + goto free; + } + + err = atl_ptp_skb_ring_init(&ptp->skb_ring, nic->requested_rx_size); + if (err != 0) { + err = -ENOMEM; + goto free; + } + + return 0; + +free: + while (--qvec >= &ptp->qvec[0]) + atl_free_qvec(qvec); +#endif + + return err; +} + +int atl_ptp_ring_start(struct atl_nic *nic) +{ + int err = 0; +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct atl_queue_vec *qvec; + + if (!ptp) + return 0; + + atl_for_each_ptp_qvec(ptp, qvec) { + err = atl_start_qvec(qvec); + if (err) + goto stop; + } + + netif_napi_add(nic->ndev, ptp->napi, atl_ptp_poll, 64); + napi_enable(ptp->napi); + + return 0; + +stop: + while (--qvec >= &ptp->qvec[0]) + atl_stop_qvec(qvec); +#endif + + return err; +} + +void atl_ptp_ring_stop(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct atl_queue_vec *qvec; + + if (!ptp) + return; + + napi_disable(ptp->napi); + netif_napi_del(ptp->napi); + + atl_for_each_ptp_qvec(ptp, qvec) + atl_stop_qvec(qvec); +#endif +} + +void atl_ptp_ring_free(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return; + + atl_free_qvec(&ptp->qvec[ATL_PTPQ_PTP]); + + atl_ptp_skb_ring_release(&ptp->skb_ring); +#endif +} + +void atl_ptp_clock_init(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct timespec64 ts; + + ktime_get_real_ts64(&ts); + atl_ptp_settime(&ptp->ptp_info, &ts); +#endif +} + +int atl_ptp_init(struct atl_nic *nic) +{ + int err = 0; +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_mcp *mcp = &nic->hw.mcp; + struct atl_ptp *ptp; + + if (!mcp->ops->set_ptp) { + nic->ptp = NULL; + return 0; + } + + if (!(mcp->caps_ex & atl_fw2_ex_caps_phy_ptp_en)) { + nic->ptp = NULL; + return 0; + } + + ptp = kzalloc(sizeof(*ptp), GFP_KERNEL); + if (!ptp) { + err = -ENOMEM; + goto err_exit; + } + + ptp->nic = nic; + + ptp->qvec[ATL_PTPQ_PTP].type = ATL_QUEUE_PTP; + ptp->qvec[ATL_PTPQ_HWTS].type = ATL_QUEUE_HWTS; + + spin_lock_init(&ptp->ptp_lock); + spin_lock_init(&ptp->ptp_ring_lock); + + atl_ptp_tx_timeout_init(&ptp->ptp_tx_timeout); + + atomic_set(&ptp->offset_egress, 0); + atomic_set(&ptp->offset_ingress, 0); + + ptp->napi = &ptp->qvec[ATL_PTPQ_PTP].napi; + + ptp->idx_vector = ATL_IRQ_PTP; + + nic->ptp = ptp; + + /* enable ptp counter */ + nic->hw.link_state.ptp_available = true; + mcp->ops->set_ptp(&nic->hw, true); + atl_ptp_clock_init(nic); + + INIT_DELAYED_WORK(&ptp->poll_sync, &atl_ptp_poll_sync_work_cb); + ptp->eth_type_filter_idx = atl_reserve_filter(ATL_RXF_ETYPE); + ptp->udp_filter_idx = atl_reserve_filter(ATL_RXF_NTUPLE); + + return 0; + +err_exit: + if (ptp) + kfree(ptp->ptp_info.pin_config); + kfree(ptp); + nic->ptp = NULL; +#endif + + return err; +} + +int atl_ptp_register(struct atl_nic *nic) +{ + int err = 0; + +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp_offset_info ptp_offset_info; + enum atl_gpio_pin_function gpio_pin[3]; + struct atl_mcp *mcp = &nic->hw.mcp; + struct atl_ptp *ptp = nic->ptp; + struct ptp_clock *clock; + + if (!ptp) + return 0; + + err = atl_read_mcp_mem(&nic->hw, mcp->fw_stat_addr + atl_fw2_stat_ptp_offset, + &ptp_offset_info, sizeof(ptp_offset_info)); + if (err) + return err; + + err = atl_read_mcp_mem(&nic->hw, mcp->fw_stat_addr + atl_fw2_stat_gpio_pin, + &gpio_pin, sizeof(gpio_pin)); + if (err) + return err; + + atl_ptp_offset_init(&ptp_offset_info); + + ptp->ptp_info = atl_ptp_clock; + atl_ptp_gpio_init(nic, &ptp->ptp_info, &gpio_pin[0]); + clock = ptp_clock_register(&ptp->ptp_info, &nic->ndev->dev); + if (IS_ERR_OR_NULL(clock)) { + netdev_err(nic->ndev, "ptp_clock_register failed\n"); + err = PTR_ERR(clock); + goto err_exit; + } + ptp->ptp_clock = clock; + +err_exit: +#endif + + return err; +} + +void atl_ptp_unregister(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return; + + ptp_clock_unregister(ptp->ptp_clock); +#endif +} + +void atl_ptp_free(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_mcp *mcp = &nic->hw.mcp; + struct atl_ptp *ptp = nic->ptp; + + if (!ptp) + return; + + atl_release_filter(ATL_RXF_ETYPE); + atl_release_filter(ATL_RXF_NTUPLE); + + cancel_delayed_work_sync(&ptp->poll_sync); + /* disable ptp */ + mcp->ops->set_ptp(&nic->hw, false); + + kfree(ptp->ptp_info.pin_config); + + kfree(ptp); + nic->ptp = NULL; +#endif +} + +struct ptp_clock *atl_ptp_get_ptp_clock(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + return nic->ptp->ptp_clock; +#else + return NULL; +#endif +} + +int atl_ptp_link_change(struct atl_nic *nic) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + struct atl_ptp *ptp = nic->ptp; + struct atl_hw *hw = &nic->hw; + + if (!ptp) + return 0; + + if (hw->mcp.ops->check_link(hw)) + atl_ptp_start_work(ptp); + else + cancel_delayed_work_sync(&ptp->poll_sync); +#endif + + return 0; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.h new file mode 100644 index 000000000000..4449b0044e7b --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ptp.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2014-2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef ATL_PTP_H +#define ATL_PTP_H + +#include + +#include "atl_compat.h" + +struct atl_nic; +struct atl_queue_vec; + +/* Common functions */ +int atl_ptp_init(struct atl_nic *nic); +int atl_ptp_register(struct atl_nic *nic); + +void atl_ptp_unregister(struct atl_nic *nic); +void atl_ptp_free(struct atl_nic *nic); + +int atl_ptp_irq_alloc(struct atl_nic *nic); +void atl_ptp_irq_free(struct atl_nic *nic); + +int atl_ptp_ring_alloc(struct atl_nic *nic); +void atl_ptp_ring_free(struct atl_nic *nic); + +int atl_ptp_ring_start(struct atl_nic *nic); +void atl_ptp_ring_stop(struct atl_nic *nic); + +void atl_ptp_work(struct atl_nic *nic); + +void atl_ptp_tm_offset_set(struct atl_nic *nic, unsigned int mbps); + +void atl_ptp_clock_init(struct atl_nic *nic); + +int atl_ptp_qvec_intr(struct atl_queue_vec *qvec); + +/* Traffic processing functions */ +netdev_tx_t atl_ptp_start_xmit(struct atl_nic *nic, struct sk_buff *skb); +void atl_ptp_tx_hwtstamp(struct atl_nic *nic, u64 timestamp); + +/* Check for PTP availability before calling! */ +void atl_ptp_hwtstamp_config_get(struct atl_nic *nic, + struct hwtstamp_config *config); +int atl_ptp_hwtstamp_config_set(struct atl_nic *nic, + struct hwtstamp_config *config); + +/* Return whether ring belongs to PTP or not*/ +bool atl_is_ptp_ring(struct atl_nic *nic, struct atl_desc_ring *ring); +u16 atl_ptp_extract_ts(struct atl_nic *nic, struct sk_buff *skb, u8 *p, + unsigned int len); + +struct ptp_clock *atl_ptp_get_ptp_clock(struct atl_nic *nic); + +int atl_ptp_link_change(struct atl_nic *nic); + +#endif /* ATL_PTP_H */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_regs.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_regs.h new file mode 100644 index 000000000000..0edd7dd4d795 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_regs.h @@ -0,0 +1,224 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_REGS_H_ +#define _ATL_REGS_H_ + +#define ATL_REG_STRIDE(base, stride, idx) ((base) + (stride) * (idx)) + +/* Ring registers common for Rx and Tx */ +#define ATL_RING_OFFT(ring, offt) \ + (((struct atl_hw_ring *)(ring))->reg_base + (offt)) +#define ATL_RING_BASE_LSW(ring) ATL_RING_OFFT(ring, 0) +#define ATL_RING_BASE_MSW(ring) ATL_RING_OFFT(ring, 4) +#define ATL_RING_CTL(ring) ATL_RING_OFFT(ring, 8) +#define ATL_RING_HEAD(ring) ATL_RING_OFFT(ring, 0xc) +#define ATL_RING_TAIL(ring) ATL_RING_OFFT(ring, 0x10) +#define ATL_RING_STS(ring) ATL_RING_OFFT(ring, 0x14) + +/* MIF @ 0x0000*/ +#define ATL_GLOBAL_STD_CTRL 0 +#define ATL_GLOBAL_FW_ID 0xc +#define ATL_GLOBAL_CHIP_ID 0x10 +#define ATL_GLOBAL_CHIP_REV 0x14 +#define ATL_GLOBAL_FW_IMAGE_ID 0x18 +#define ATL_GLOBAL_MIF_ID 0x1c +#define ATL_GLOBAL_MBOX_CTRL 0x200 +#define ATL_GLOBAL_MBOX_CRC 0x204 +#define ATL_GLOBAL_MBOX_ADDR 0x208 +#define ATL_GLOBAL_MBOX_DATA 0x20c +#define ATL_GLOBAL_MDIO_CTL 0x280 +#define ATL_GLOBAL_MDIO_CMD 0x284 +#define ATL_GLOBAL_MDIO_WDATA 0x288 +#define ATL_GLOBAL_MDIO_ADDR 0x28c +#define ATL_GLOBAL_MDIO_RDATA 0x290 +#define ATL2_MIF_BOOT_READ_REQ_ADR 0x328 +#define ATL2_MIF_BOOT_READ_REQ_LEN 0x32c +#define ATL2_MIF_BOOT_REG_ADR 0x3040 +#define ATL2_MCP_HOST_REQ_INT 0x0F00 +#define ATL2_MCP_HOST_REQ_INT_CLR 0x0F08 +#define ATL2_MCP_HOST_REQ_INT_MASK(idx) ATL_REG_STRIDE(0x0f0c, 4, (idx)) +#define ATL2_MIF_SHARED_BUFFER_BOOT(idx) ATL_REG_STRIDE(0x00010000, 0x4, (idx)) +#define ATL2_MIF_SHARED_BUFFER_IN(idx) ATL_REG_STRIDE(0x00012000, 0x4, (idx)) +#define ATL2_MIF_SHARED_BUFFER_OUT(idx) ATL_REG_STRIDE(0x00013000, 0x4, (idx)) +#define ATL2_MIF_HOST_FINISHED_WRITE 0x0e00 +#define ATL2_MIF_MCP_FINISHED_READ 0x0e04 + +/* Scratch pads numbered starting from 1 */ +#define ATL_MCP_SCRATCH(idx) ATL_REG_STRIDE(0x300 - 0x4, 0x4, idx) +#define ATL_MCP_SEM(idx) ATL_REG_STRIDE(0x3a0, 0x4, idx) +#define ATL_MCP_SEM_MDIO 0 +#define ATL_MCP_SEM_MSM 1 +#define ATL_MCP_SEM_MEM 2 +#define ATL2_MCP_SEM_ACT_RSLVR 3 +#define ATL_GLOBAL_CTRL2 0x404 +#define ATL_GLOBAL_DAISY_CHAIN_STS1 0x704 + +enum mcp_scratchpad { + FW2_MBOX_DATA = 11, /* 0x328 */ + FW2_MBOX_CMD = 12, /* 0x32c */ + FW2_RPC_DATA = 14, /* 0x334 */ + FW_STAT_STRUCT = 25, /* 0x360 */ + FW2_EFUSE_SHADOW = 26, /* 0x364 */ + FW1_LINK_REQ = 27, + FW2_LINK_REQ_LOW = 27, /* 0x368 */ + FW1_LINK_STS = 28, + FW2_LINK_REQ_HIGH = 28, /* 0x36c */ + FW2_LINK_RES_LOW = 29, /* 0x370 */ + FW1_EFUSE_SHADOW = 30, + FW2_LINK_RES_HIGH = 30, /* 0x374 */ + FW3_EXT_REQ = 31, /* 0x378 */ + FW3_EXT_RES = 32, /* 0x37c */ + RBL_STS = 35, /* 0x388 */ +}; + +/* INTR @ 0x2000 */ +#define ATL_INTR_STS 0x2000 +#define ATL_INTR_MSK 0x2010 +#define ATL_INTR_MSK_SET 0x2060 +#define ATL_INTR_MSK_CLEAR 0x2070 +#define ATL_INTR_AUTO_CLEAR 0x2080 +#define ATL_INTR_AUTO_MASK 0x2090 +#define ATL_INTR_RING_INTR_MAP(idx) ATL_REG_STRIDE(0x2100, 0x4, (idx) >> 1) +#define ATL_INTR_GEN_INTR_MAP4 0x218c +#define ATL_INTR_RSC_EN 0x2200 +#define ATL_INTR_RSC_DELAY 0x2204 +#define ATL_INTR_CTRL 0x2300 +#define ATL_INTR_THRTL(idx) ATL_REG_STRIDE(0x2800, 4, idx) + +/* MPI @ 0x4000 */ +#define ATL_MPI_CTRL1 0x4000 +#define ATL_MPI_MSM_CTRL 0x4018 +#define ATL_MPI_MSM_ADDR 0x4400 +#define ATL_MPI_MSM_WR 0x4404 +#define ATL_MPI_MSM_RD 0x4408 + +/* RX @ 0x5000 */ +#define ATL_RX_CTRL1 0x5000 +#define ATL_RX_SPARE_CTRL0 0x50A0 +#define ATL_RX_SPARE_CTRL1 0x50A4 +#define ATL2_RX_FLT_L2_BC_TAG 0x50F0 +#define ATL_RX_FLT_CTRL1 0x5100 +#define ATL_RX_FLT_CTRL2 0x5104 +#define ATL_UC_FLT_NUM 37 +#define ATL_RX_UC_FLT_REG1(idx) ATL_REG_STRIDE(0x5110, 8, idx) +#define ATL_RX_UC_FLT_REG2(idx) ATL_REG_STRIDE(0x5114, 8, idx) +#define ATL_MC_FLT_NUM 8 +#define ATL_RX_MC_FLT(idx) ATL_REG_STRIDE(0x5250, 4, idx) +#define ATL_RX_MC_FLT_MSK 0x5270 +#define ATL_RX_VLAN_FLT_CTRL1 0x5280 +#define ATL_VLAN_FLT_NUM 16 +#define ATL_RX_VLAN_FLT(idx) ATL_REG_STRIDE(0x5290, 4, idx) +#define ATL_RX_ETYPE_FLT(idx) ATL_REG_STRIDE(0x5300, 4, idx) +#define ATL2_RX_ETYPE_TAG(idx) ATL_REG_STRIDE(0x5340, 4, idx) +#define ATL_ETYPE_FLT_NUM 16 +#define ATL_NTUPLE_CTRL(idx) ATL_REG_STRIDE(0x5380, 4, idx) +#define ATL_NTUPLE_SADDR(idx) ATL_REG_STRIDE(0x53b0, 4, idx) +#define ATL_NTUPLE_DADDR(idx) ATL_REG_STRIDE(0x53d0, 4, idx) +#define ATL_NTUPLE_SPORT(idx) ATL_REG_STRIDE(0x5400, 4, idx) +#define ATL_NTUPLE_DPORT(idx) ATL_REG_STRIDE(0x5420, 4, idx) +#define ATL_NTUPLE_FLT_NUM 8 +#define ATL_NTUPLE_V6_FLT_NUM 2 +#define ATL_RX_FLEX_FLT_CTRL(idx) ATL_REG_STRIDE(0x5460, 0x20, idx) +#define ATL_FLEX_FLT_NUM 2 +#define ATL_RX_RSS_CTRL 0x54c0 +#define ATL_RX_RSS_KEY_ADDR 0x54d0 +#define ATL_RX_RSS_KEY_WR_DATA 0x54d4 +#define ATL_RX_RSS_KEY_RD_DATA 0x54d8 +#define ATL_RX_RSS_TBL_ADDR 0x54e0 +#define ATL_RX_RSS_TBL_WR_DATA 0x54e4 +#define ATL_RX_RSS_TBL_RD_DATA 0x54e8 +#define ATL2_RX_RSS_HASH_TYPE_ADR 0x54C8 +#define ATL_RX_RPF_DBG_CNT_CTRL 0x5518 +#define ATL_RX_RPF_HOST_CNT_LO 0x552c +#define ATL_RX_RPF_HOST_CNT_HI 0x5530 +#define ATL_RX_RPF_LOST_CNT_LO 0x554c +#define ATL_RX_RPF_LOST_CNT_HI 0x5550 +#define ATL_RX_PO_CTRL1 0x5580 +#define ATL_RX_LRO_CTRL1 0x5590 +#define ATL_RX_LRO_CTRL2 0x5594 +#define ATL_RX_LRO_PKT_LIM_EN 0x5598 +#define ATL_RX_LRO_PKT_LIM(idx) ATL_REG_STRIDE(0x55a0, 4, (idx) >> 3) +#define ATL_RX_LRO_TMRS 0x5620 +#define ATL_RX_PBUF_CTRL1 0x5700 +#define ATL_RX_PBUF_REG1(idx) ATL_REG_STRIDE(0x5710, 0x10, idx) +#define ATL_RX_PBUF_REG2(idx) ATL_REG_STRIDE(0x5714, 0x10, idx) +#define ATL2_RX_Q_TO_TC_MAP(tc) ATL_REG_STRIDE(0x5900, 0x4, tc) +#define ATL_RX_INTR_CTRL 0x5a30 +#define ATL_RX_INTR_MOD_CTRL(idx) ATL_REG_STRIDE(0x5a40, 4, idx) +#define ATL2_RPF_RSS_REDIR(TC, INDEX) (0x6200 + \ + (0x100 * (TC > 3 ? 1 : 0)) + (INDEX) * 4) +#define ATL2_RPF_L3_FLT(filter) ATL_REG_STRIDE(0x6500, 0x4, filter) +#define ATL2_RPF_L3_SA(filter) ATL_REG_STRIDE(0x6400, 0x10, filter) +#define ATL2_RPF_L3_DA(filter) ATL_REG_STRIDE(0x6480, 0x10, filter) +#define ATL2_RPF_L4_FLT(filter) ATL_REG_STRIDE(0x6520, 0x4, filter) +#define ATL2_RPF_REC_TAB_EN 0x00006ff0 +#define ATL2_RPF_ACT_RSLVR_REQ_TAG(filter) ATL_REG_STRIDE(0x14000, 0x10, filter) +#define ATL2_RPF_ACT_RSLVR_TAG_MASK(filter) ATL_REG_STRIDE(0x14004, 0x10, filter) +#define ATL2_RPF_ACT_RSLVR_ACTN(filter) ATL_REG_STRIDE(0x14008, 0x10, filter) + +/* Rx rings */ +#define ATL_RX_RING(idx) ATL_REG_STRIDE(0x5b00, 0x20, idx) +#define ATL_RX_RING_BASE_LSW(ring) ATL_RING_BASE_LSW(ring) +#define ATL_RX_RING_BASE_MSW(ring) ATL_RING_BASE_MSW(ring) +#define ATL_RX_RING_CTL(ring) ATL_RING_CTL(ring) +#define ATL_RX_RING_HEAD(ring) ATL_RING_HEAD(ring) +#define ATL_RX_RING_TAIL(ring) ATL_RING_TAIL(ring) +#define ATL_RX_RING_STS(ring) ATL_RING_STS(ring) +#define ATL_RX_RING_BUF_SIZE(ring) ATL_RING_OFFT(ring, 0x18) +#define ATL_RX_RING_THRESH(ring) ATL_RING_OFFT(ring, 0x1c) + +#define ATL_RX_DMA_STATS_CNT7 0x6818 + +/* TX @ 0x7000 */ +#define ATL_TX_CTRL1 0x7000 +#define ATL_TX_PO_CTRL1 0x7800 +#define ATL_TX_LSO_CTRL 0x7810 +#define ATL_TX_LSO_TCP_CTRL1 0x7820 +#define ATL_TX_LSO_TCP_CTRL2 0x7824 +#define ATL_TX_PBUF_CTRL1 0x7900 +#define ATL_TX_PBUF_REG1(idx) ATL_REG_STRIDE(0x7910, 0x10, idx) +#define ATL_TX_PBUF_REG2(idx) ATL_REG_STRIDE(0x7914, 0x10, idx) +#define ATL_TX_INTR_CTRL 0x7b40 +#define ATL2_TX_Q_TO_TC_MAP(tc) ATL_REG_STRIDE(0x799c, 0x4, tc) + +/* Tx rings */ +#define ATL_TX_RING(idx) ATL_REG_STRIDE(0x7c00, 0x40, idx) +#define ATL_TX_RING_BASE_LSW(ring) ATL_RING_BASE_LSW(ring) +#define ATL_TX_RING_BASE_MSW(ring) ATL_RING_BASE_MSW(ring) +#define ATL_TX_RING_CTL(ring) ATL_RING_CTL(ring) +#define ATL_TX_RING_HEAD(ring) ATL_RING_HEAD(ring) +#define ATL_TX_RING_TAIL(ring) ATL_RING_TAIL(ring) +#define ATL_TX_RING_STS(ring) ATL_RING_STS(ring) +#define ATL_TX_RING_THRESH(ring) ATL_RING_OFFT(ring, 0x18) +#define ATL_TX_RING_HEAD_WB_LSW(ring) ATL_RING_OFFT(ring, 0x1c) +#define ATL_TX_RING_HEAD_WB_MSW(ring) ATL_RING_OFFT(ring, 0x20) + +#define ATL_TX_INTR_MOD_CTRL(idx) ATL_REG_STRIDE(0x8980, 0x4, idx) +#define ATL2_TX_INTR_MOD_CTRL(idx) ATL_REG_STRIDE(0x7c28, 0x40, idx) + +/* MSM */ +#define ATL_MSM_GEN_CTRL 0x8 +#define ATL_MSM_GEN_STS 0x40 +#define ATL_MSM_TX_LPI_DELAY 0x78 +#define ATL_MSM_CTR_RX_PKTS_GOOD 0x88 +#define ATL_MSM_CTR_RX_FCS_ERRS 0x90 +#define ATL_MSM_CTR_RX_ALIGN_ERRS 0x98 +#define ATL_MSM_CTR_TX_PAUSE 0xa0 +#define ATL_MSM_CTR_RX_PAUSE 0xa8 +#define ATL_MSM_CTR_RX_OCTETS_LO 0xd8 +#define ATL_MSM_CTR_RX_OCTETS_HI 0xdc +#define ATL_MSM_CTR_RX_MULTICAST 0xE8 +#define ATL_MSM_CTR_RX_BROADCAST 0xF0 +#define ATL_MSM_CTR_RX_ERRS 0x120 + + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.c new file mode 100644 index 000000000000..d0b1b3ca3687 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.c @@ -0,0 +1,2199 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atl_ring.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "atl_trace.h" +#include "atl_fwdnl.h" +#include "atl_hw_ptp.h" + +static inline uint32_t fetch_tx_head(struct atl_desc_ring *ring) +{ +#ifdef ATL_TX_HEAD_WB + //XXX +#else + return atl_read(ring_hw(ring), ATL_TX_RING_HEAD(ring)); +#endif +} + +int atl_tx_full(struct atl_desc_ring *ring, int needed) +{ + struct atl_nic *nic = ring->nic; + + if (likely(ring_space(ring) >= needed)) + return 0; + + netif_stop_subqueue(nic->ndev, ring->qvec->idx); + atl_nic_dbg("Stopping tx queue\n"); + + smp_mb(); + + // Check if another CPU freed some space + if (likely(ring_space(ring) < needed)) + return -EAGAIN; + + netif_start_subqueue(nic->ndev, ring->qvec->idx); + atl_nic_dbg("Restarting tx queue in %s...\n", __func__); + atl_update_ring_stat(ring, tx.tx_restart, 1); + return 0; +} + +static void atl_txbuf_free(struct atl_txbuf *txbuf, struct device *dev, + uint32_t idx) +{ + if (txbuf->skb) { + if (dma_unmap_len(txbuf, len)) { + dma_unmap_single(dev, dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), + DMA_TO_DEVICE); + trace_atl_dma_unmap_head(-1, idx, + dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), + txbuf->skb); + } + dev_kfree_skb_any(txbuf->skb); + } else if (dma_unmap_len(txbuf, len)) { + dma_unmap_page(dev, dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), + DMA_TO_DEVICE); + trace_atl_dma_unmap_frag(-1, idx, dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), txbuf->skb); + } + + txbuf->last = -1; + txbuf->skb = NULL; + dma_unmap_len_set(txbuf, len, 0); +} + +static inline struct netdev_queue *atl_txq(struct atl_desc_ring *ring) +{ + return netdev_get_tx_queue(ring->nic->ndev, ring->qvec->idx); +} + +unsigned int atl_tx_free_low = MAX_SKB_FRAGS + 4; +module_param_named(tx_free_low, atl_tx_free_low, uint, 0644); + +unsigned int atl_tx_free_high = MAX_SKB_FRAGS * 3; +module_param_named(tx_free_high, atl_tx_free_high, uint, 0644); + +static netdev_tx_t atl_map_xmit_skb(struct sk_buff *skb, + struct atl_desc_ring *ring, struct atl_txbuf *first_buf) +{ + int idx = ring->tail; + struct device *dev = &ring->nic->hw.pdev->dev; + struct atl_tx_desc *desc = &ring->desc.tx; + skb_frag_t *frag; + /* Header's DMA mapping must be stored in the txbuf that has + * ->skb set, even if it corresponds to the context + * descriptor and not the first data descriptor + */ + struct atl_txbuf *txbuf = first_buf; + unsigned int len = skb_headlen(skb); + unsigned int frags = skb_shinfo(skb)->nr_frags; + dma_addr_t daddr = dma_map_single(dev, skb->data, len, + DMA_TO_DEVICE); + trace_atl_dma_map_head(-1, idx, daddr, len, skb, skb->data); + + for (frag = &skb_shinfo(skb)->frags[0];; frag++) { + if (dma_mapping_error(dev, daddr)) + goto err_dma; + + dma_unmap_len_set(txbuf, len, len); + dma_unmap_addr_set(txbuf, daddr, daddr); + + desc->daddr = cpu_to_le64(daddr); + while (len > ATL_DATA_PER_TXD) { + desc->len = cpu_to_le16(ATL_DATA_PER_TXD); + trace_atl_tx_descr(ring->qvec->idx, idx, (u64 *)desc); + ring->hw.descs[idx].tx = *desc; + bump_ptr(idx, ring, 1); + daddr += ATL_DATA_PER_TXD; + len -= ATL_DATA_PER_TXD; + desc->daddr = cpu_to_le64(daddr); + } + desc->len = cpu_to_le16(len); + + if (!frags) + break; + + trace_atl_tx_descr(ring->qvec->idx, idx, (u64 *)desc); + ring->hw.descs[idx].tx = *desc; + bump_ptr(idx, ring, 1); + txbuf = &ring->txbufs[idx]; + len = skb_frag_size(frag); + daddr = skb_frag_dma_map(dev, frag, 0, len, + DMA_TO_DEVICE); + trace_atl_dma_map_frag(frag - &skb_shinfo(skb)->frags[0], idx, + daddr, len, skb, skb_frag_address(frag)); + + frags--; + } + + //Last descriptor + desc->eop = 1; +#if defined(ATL_TX_DESC_WB) || defined(ATL_TX_HEAD_WB) + desc->cmd |= tx_desc_cmd_wb; +#endif + trace_atl_tx_descr(ring->qvec->idx, idx, (u64 *)desc); + ring->hw.descs[idx].tx = *desc; + first_buf->last = idx; + bump_ptr(idx, ring, 1); + ring->txbufs[idx].last = -1; + ring->tail = idx; + + /* Stop queue if no space for another packet */ + atl_tx_full(ring, atl_tx_free_low); + + /* Delay bumping the HW tail if another packet is pending and + * there's space for it. + */ + if (skb_xmit_more(skb) && !netif_xmit_stopped(atl_txq(ring))) + return NETDEV_TX_OK; + + wmb(); + atl_write(ring_hw(ring), ATL_TX_RING_TAIL(ring), ring->tail); + + return NETDEV_TX_OK; + +err_dma: + dev_err(dev, "atl_map_skb failed\n"); + for (;;) { + atl_txbuf_free(txbuf, dev, idx); + if (txbuf == first_buf) + break; + bump_ptr(idx, ring, -1); + txbuf = &ring->txbufs[idx]; + } + ring->tail = idx; + atl_update_ring_stat(ring, tx.dma_map_failed, 1); + return -EFAULT; +} + +static uint32_t atl_insert_context(struct atl_txbuf *txbuf, + struct atl_desc_ring *ring, unsigned int *len) +{ + struct sk_buff *skb = txbuf->skb; + struct atl_tx_ctx *ctx; + unsigned int hdr_len; + uint32_t tx_cmd = 0; + int mss; + DECLARE_SCRATCH_DESC(scratch); + + ctx = &DESC_PTR(ring, ring->tail, scratch)->ctx; + + memset(ctx, 0, sizeof(*ctx)); + + txbuf->bytes = *len; + txbuf->packets = 1; + + mss = skb_shinfo(skb)->gso_size; + + if (mss && (skb_shinfo(skb)->gso_type & + (SKB_GSO_TCPV4 | SKB_GSO_TCPV6))) { + tx_cmd |= tx_desc_cmd_lso | tx_desc_cmd_l4cs; + ctx->mss_len = mss; + ctx->cmd = ctx_cmd_tcp; + + ctx->l2_len = skb_network_offset(skb); + + if (skb_is_gso_v6(skb)) + ctx->cmd |= ctx_cmd_ipv6; + + ctx->l3_len = skb_transport_offset(skb) - ctx->l2_len; + ctx->l4_len = tcp_hdrlen(skb); + + hdr_len = ctx->l2_len + ctx->l3_len + ctx->l4_len; + + *len -= hdr_len; + txbuf->packets = skb_shinfo(skb)->gso_segs; + txbuf->bytes += (txbuf->packets - 1) * hdr_len; + } + + if (skb_vlan_tag_present(skb)) { + tx_cmd |= tx_desc_cmd_vlan; + ctx->vlan_tag = skb_vlan_tag_get(skb); + } + + if (tx_cmd) { + ctx->type = tx_desc_type_context; + ctx->idx = 0; + trace_atl_tx_context_descr(ring->qvec->idx, ring->tail, (u64 *)ctx); + COMMIT_DESC(ring, ring->tail, scratch); + bump_tail(ring, 1); + } + + return tx_cmd; +} + +netdev_tx_t atl_start_xmit(struct sk_buff *skb, struct net_device *ndev) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_desc_ring *ring = &nic->qvecs[skb->queue_mapping].tx; + + if (unlikely(nic->hw.link_state.ptp_datapath_up)) { + /* Hardware adds the Timestamp for PTPv2 802.AS1 + * and PTPv2 IPv4 UDP. + * We have to push even general 320 port messages to the ptp + * queue explicitly. This is a limitation of current firmware + * and hardware PTP design of the chip. Otherwise ptp stream + * will fail to sync + */ + if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) || + unlikely((ip_hdr(skb)->version == 4) && + (ip_hdr(skb)->protocol == IPPROTO_UDP) && + ((udp_hdr(skb)->dest == htons(319)) || + (udp_hdr(skb)->dest == htons(320)))) || + unlikely(eth_hdr(skb)->h_proto == htons(ETH_P_1588))) + return atl_ptp_start_xmit(nic, skb); + } + + skb_tx_timestamp(skb); + if (nic->priv_flags & ATL_PF_BIT(LPB_NET_DMA)) + return NETDEV_TX_BUSY; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + /* atl_max_queues is the number of standard queues. + * Extra queue is allocated for FWD processing. + */ + if (unlikely(skb->queue_mapping >= nic->nvecs)) + return atlfwd_nl_xmit(skb, ndev); +#endif + + if (atl_tx_full(ring, skb_shinfo(skb)->nr_frags + 4)) { + atl_update_ring_stat(ring, tx.tx_busy, 1); + return NETDEV_TX_BUSY; + } + + return atl_map_skb(skb, ring); +} + +netdev_tx_t atl_map_skb(struct sk_buff *skb, struct atl_desc_ring *ring) +{ + unsigned int len = skb->len; + struct atl_tx_desc *desc; + struct atl_txbuf *txbuf; + uint32_t cmd_from_ctx; + + txbuf = &ring->txbufs[ring->tail]; + + txbuf->skb = skb; + cmd_from_ctx = atl_insert_context(txbuf, ring, &len); + + /* use ring->desc unconditionally as it will serve as a + * template for all descriptors + */ + desc = &ring->desc.tx; + + memset(desc, 0, sizeof(*desc)); + + desc->cmd = cmd_from_ctx; + desc->cmd |= tx_desc_cmd_fcs; + desc->ct_en = !!cmd_from_ctx; + desc->type = tx_desc_type_desc; + + desc->pay_len = len; + + if (skb->ip_summed == CHECKSUM_PARTIAL) { + uint8_t l4_proto = 0; + + switch (skb->protocol) { + case htons(ETH_P_IP): + desc->cmd |= tx_desc_cmd_ipv4cs; + l4_proto = ip_hdr(skb)->protocol; + break; + case htons(ETH_P_IPV6): + l4_proto = ipv6_hdr(skb)->nexthdr; + break; + } + + switch (l4_proto) { + case IPPROTO_TCP: + case IPPROTO_UDP: + desc->cmd |= tx_desc_cmd_l4cs; + break; + } + } + + return atl_map_xmit_skb(skb, ring, txbuf); +} + +unsigned int atl_tx_clean_budget = 256; +module_param_named(tx_clean_budget, atl_tx_clean_budget, uint, 0644); + +// Returns true if all work done +static bool atl_clean_tx(struct atl_desc_ring *ring) +{ + struct atl_nic *nic = ring->nic; + struct device *dev = &nic->hw.pdev->dev; + uint32_t first = READ_ONCE(ring->head); +#ifndef ATL_TX_DESC_WB + uint32_t done = atl_get_tx_head(ring); +#endif + uint32_t budget = atl_tx_clean_budget; + unsigned int bytes = 0, packets = 0; + struct atl_tx_desc *last_desc; + + atl_nic_dbg("descs in ring: %d\n", ring_occupied(ring)); + do { + struct atl_txbuf *txbuf = &ring->txbufs[first]; + struct sk_buff *skb = txbuf->skb; + uint32_t last = txbuf->last; + + if (last == -1) + break; + +#ifdef ATL_TX_DESC_WB + last_desc = &ring->hw.descs[last].tx; + + if (!last_desc->dd) + break; +#else + if ((first <= last && done >= first && done <= last) || + ((first > last) && (done >= first || done <= last))) + break; +#endif + + bump_ptr(last, ring, 1); + napi_consume_skb(txbuf->skb, budget); + trace_atl_dma_unmap_head(-1, first, + dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), skb); + + txbuf->skb = NULL; + txbuf->last = -1; + dma_unmap_single(dev, dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), DMA_TO_DEVICE); + dma_unmap_len_set(txbuf, len, 0); + + bytes += txbuf->bytes; + packets += txbuf->packets; + + for (bump_ptr(first, ring, 1); first != last; + bump_ptr(first, ring, 1)) { + txbuf = &ring->txbufs[first]; + if (dma_unmap_len(txbuf, len)) { + dma_unmap_page(dev, + dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), + DMA_TO_DEVICE); + trace_atl_dma_unmap_frag(-1, first, + dma_unmap_addr(txbuf, daddr), + dma_unmap_len(txbuf, len), skb); + dma_unmap_len_set(txbuf, len, 0); + } + } + } while (--budget); + + if (likely(ring->qvec->type != ATL_QUEUE_PTP)) { + u64_stats_update_begin(&ring->syncp); + ring->stats.tx.bytes += bytes; + ring->stats.tx.packets += packets; + u64_stats_update_end(&ring->syncp); + } + + ring->head = first; + + if (likely(ring->qvec->type != ATL_QUEUE_PTP) && + ring_space(ring) > atl_tx_free_high) { + struct net_device *ndev = nic->ndev; + + smp_mb(); + if (__netif_subqueue_stopped(ndev, ring->qvec->idx) && + test_bit(ATL_ST_RINGS_RUNNING, &nic->hw.state)) { + atl_nic_dbg("restarting tx queue\n"); + netif_wake_subqueue(ndev, ring->qvec->idx); + atl_update_ring_stat(ring, tx.tx_restart, 1); + } + } + + return !!budget; +} + +/* work around HW bugs in checksum calculation: + * - packets less than 60 octets + * - ip, tcp or udp checksum is 0xFFFF + * - non-zero padding + */ +static bool atl_checksum_workaround(struct sk_buff *skb, + struct atl_rx_desc_wb *desc) +{ + int ip_header_offset = 14; + int l4_header_offset = 0; + struct iphdr *ip; + struct ipv6hdr *ipv6; + struct tcphdr *tcp; + struct udphdr *udp; + + if (desc->pkt_len <= 60) + return true; + + if (((desc->pkt_type & atl_rx_pkt_type_vlan_msk) == + atl_rx_pkt_type_vlan) && + !(desc->rx_estat & atl_rx_estat_vlan_stripped)) + ip_header_offset += sizeof(struct vlan_hdr); + + if ((desc->pkt_type & atl_rx_pkt_type_vlan_msk) == + atl_rx_pkt_type_dbl_vlan) { + if (desc->rx_estat & atl_rx_estat_vlan_stripped) + ip_header_offset += sizeof(struct vlan_hdr); + else + ip_header_offset += sizeof(struct vlan_hdr) * 2; + } + + switch (desc->pkt_type & atl_rx_pkt_type_l3_msk) { + case atl_rx_pkt_type_ipv4: + ip = (struct iphdr *) &skb->data[ip_header_offset]; + + if (ip->check == 0xFFFF) + return true; + l4_header_offset = ip->ihl << 2; + /* padding inside Ethernet frame */ + if (ntohs(ip->tot_len) + ip_header_offset < desc->pkt_len) + return true; + break; + case atl_rx_pkt_type_ipv6: + ipv6 = (struct ipv6hdr *) &skb->data[ip_header_offset]; + l4_header_offset = sizeof(struct ipv6hdr); + /* padding inside Ethernet frame */ + if (ip_header_offset + sizeof(struct ipv6hdr) + + ntohs(ipv6->payload_len) < desc->pkt_len) + return true; + break; + default: + return false; + } + + switch (desc->pkt_type & atl_rx_pkt_type_l4_msk) { + case atl_rx_pkt_type_tcp: + tcp = (struct tcphdr *) &skb->data[ip_header_offset + + l4_header_offset]; + + if (tcp->check == 0xFFFF) + return true; + break; + case atl_rx_pkt_type_udp: + udp = (struct udphdr *) &skb->data[ip_header_offset + + l4_header_offset]; + if (udp->check == 0xFFFF) + return true; + /* padding inside IP frame */ + if (l4_header_offset + ntohs(udp->len) < desc->pkt_len) + return true; + break; + default: + return false; + } + + return false; +} + +bool atl_rx_checksum(struct sk_buff *skb, struct atl_rx_desc_wb *desc, + struct atl_desc_ring *ring) +{ + struct atl_nic *nic = ring->nic; + struct net_device *ndev = nic->ndev; + int csum_ok = 1; + + skb_checksum_none_assert(skb); + + if (desc->rx_stat & atl_rx_stat_mac_err) { + atl_update_ring_stat(ring, rx.mac_err, 1); + atl_nic_dbg("rx MAC err: rx_stat %d pkt_type %d len %d\n", + desc->rx_stat, desc->pkt_type, desc->pkt_len); + goto drop; + } + + if (!(ndev->features & NETIF_F_RXCSUM)) + return true; + + switch (desc->pkt_type & atl_rx_pkt_type_l3_msk) { + case atl_rx_pkt_type_ipv4: + csum_ok &= !(desc->rx_stat & atl_rx_stat_ipv4_err); + /* Fallthrough */ + case atl_rx_pkt_type_ipv6: + break; + default: + return true; + } + + switch (desc->pkt_type & atl_rx_pkt_type_l4_msk) { + case atl_rx_pkt_type_tcp: + case atl_rx_pkt_type_udp: + csum_ok &= !(desc->rx_stat & atl_rx_stat_l4_err); + break; + default: + return true; + } + + if (csum_ok) { + skb->ip_summed = CHECKSUM_UNNECESSARY; + return true; + } else { + if (atl_checksum_workaround(skb, desc)) + return true; + } + + atl_update_ring_stat(ring, rx.csum_err, 1); + + atl_nic_dbg("bad rx checksum: rx_stat %d pkt_type %d len %d\n", + desc->rx_stat, desc->pkt_type, desc->pkt_len); + + if (ndev->features & NETIF_F_RXALL) + return true; + +drop: + dev_kfree_skb_any(skb); + return false; +} + +void atl_rx_hash(struct sk_buff *skb, struct atl_rx_desc_wb *desc, + struct net_device *ndev) +{ + uint8_t rss_type = desc->rss_type; + + if (!(ndev->features & NETIF_F_RXHASH) || rss_type < 2 || rss_type > 7) + return; + + skb_set_hash(skb, le32_to_cpu(desc->rss_hash), + (rss_type > 3 && rss_type < 8) ? PKT_HASH_TYPE_L4 : + PKT_HASH_TYPE_L3); +} + +static int atl_napi_receive_skb(struct atl_desc_ring *ring, struct sk_buff *skb) +{ + bool is_ptp_ring = atl_is_ptp_ring(ring->nic, ring); + struct net_device *ndev = ring->nic->ndev; + struct napi_struct *napi = &ring->qvec->napi; + + /* Send all PTP traffic to 0 queue */ + skb_record_rx_queue(skb, is_ptp_ring ? 0 : ring->qvec->idx); + skb->protocol = eth_type_trans(skb, ndev); + napi_gro_receive(napi, skb); + + return 0; +} + +static bool atl_rx_packet(struct sk_buff *skb, struct atl_rx_desc_wb *desc, + struct atl_desc_ring *ring, + rx_skb_handler_t rx_skb_func) +{ + struct net_device *ndev = ring->nic->ndev; + + if (!atl_rx_checksum(skb, desc, ring)) + return false; + + if (!skb_is_nonlinear(skb) && eth_skb_pad(skb)) + return false; + + if ((ndev->features & NETIF_F_HW_VLAN_CTAG_RX) && + (desc->rx_estat & atl_rx_estat_vlan_stripped)) { + __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), + le16_to_cpu(desc->vlan_tag)); + } + + atl_rx_hash(skb, desc, ndev); + + if (skb->pkt_type == PACKET_MULTICAST) + atl_update_ring_stat(ring, rx.multicast, 1); + + rx_skb_func(ring, skb); + + return true; +} + +unsigned int atl_rx_linear; +module_param_named(rx_linear, atl_rx_linear, uint, 0444); + +/* DMA mappings of buffer pages are accounted via struct + * atl_rxpage. Being mapped counts as a single additional reference + * for the target page. + */ +static int atl_get_page(struct atl_pgref *pgref, unsigned int order, + struct device *dev, bool atomic) +{ + struct atl_rxpage *rxpage; + struct page *page; + dma_addr_t daddr; + int ret = -ENOMEM; + gfp_t flags = atomic ? GFP_ATOMIC | __GFP_NOWARN : GFP_KERNEL; + + rxpage = kmalloc(sizeof(*rxpage), flags); + if (unlikely(!rxpage)) + return ret; + + page = __dev_alloc_pages(flags, order); + if (unlikely(!page)) + goto free_rxpage; + + daddr = dma_map_page(dev, page, 0, PAGE_SIZE << order, DMA_FROM_DEVICE); + trace_atl_dma_map_rxbuf(-1, -1, daddr, PAGE_SIZE << order, NULL, + page_to_virt(page)); + + if (unlikely(dma_mapping_error(dev, daddr))) + goto free_page; + + rxpage->page = page; + rxpage->daddr = daddr; + rxpage->order = order; + rxpage->mapcount = 1; + + pgref->rxpage = rxpage; + pgref->pg_off = 0; + + return 0; + +free_page: + __free_pages(page, order); +free_rxpage: + kfree(rxpage); + + return ret; +} + +static int atl_get_pages(struct atl_rxbuf *rxbuf, + struct atl_desc_ring *ring, bool atomic) +{ + int ret; + struct device *dev = &ring->nic->hw.pdev->dev; + + if (likely((rxbuf->head.rxpage || atl_rx_linear) + && rxbuf->data.rxpage)) + return 0; + + if (!rxbuf->head.rxpage && !atl_rx_linear) { + ret = atl_get_page(&rxbuf->head, ATL_RX_HEAD_ORDER, + dev, atomic); + if (ret) { + atl_update_ring_stat(ring, + rx.alloc_head_page_failed, 1); + return ret; + } + atl_update_ring_stat(ring, rx.alloc_head_page, 1); + } + + if (!rxbuf->data.rxpage) { + ret = atl_get_page(&rxbuf->data, ATL_RX_DATA_ORDER, + dev, atomic); + if (ret) { + atl_update_ring_stat(ring, + rx.alloc_data_page_failed, 1); + return ret; + } + atl_update_ring_stat(ring, rx.alloc_data_page, 1); + } + + return 0; +} + +static inline void atl_fill_rx_desc(struct atl_desc_ring *ring, + struct atl_rxbuf *rxbuf) +{ + struct atl_rx_desc *desc; + DECLARE_SCRATCH_DESC(scratch); + + desc = &DESC_PTR(ring, ring->tail, scratch)->rx; + + desc->daddr = atl_buf_daddr(&rxbuf->data) + + (atl_rx_linear ? ATL_RX_HEADROOM : 0); + + /* Assigning haddr clears dd as bufs are cacheline-aligned + * and ATL_RX_HEADROOM is even + */ + desc->haddr = atl_rx_linear ? 0 : + atl_buf_daddr(&rxbuf->head) + ATL_RX_HEADROOM; + + trace_atl_fill_rx_desc(ring->tail, desc); + COMMIT_DESC(ring, ring->tail, scratch); +} + +static int atl_fill_rx(struct atl_desc_ring *ring, uint32_t count, bool atomic) +{ + int ret = 0; + + while (count) { + struct atl_rxbuf *rxbuf = &ring->rxbufs[ring->tail]; + + ret = atl_get_pages(rxbuf, ring, atomic); + if (ret) + break; + + atl_fill_rx_desc(ring, rxbuf); + bump_tail(ring, 1); + count--; + } + + /* If tail ptr passed the next_to_recycle ptr, clamp the + * latter to the former. + */ + if (ring->next_to_recycle < ring->head ? + ring->next_to_recycle < ring->tail && + ring->tail < ring->head : + ring->tail > ring->next_to_recycle || + ring->tail < ring->head) + ring->next_to_recycle = ring->tail; + + wmb(); + atl_write(ring_hw(ring), ATL_RX_RING_TAIL(ring), ring->tail); + return ret; +} + +static int atl_fill_hwts_rx(struct atl_desc_ring *ring, uint32_t count, bool atomic) +{ + struct atl_rx_desc *desc; + + while (count) { + DECLARE_SCRATCH_DESC(scratch); + + desc = &DESC_PTR(ring, ring->tail, scratch)->rx; + desc->daddr = ring->hw.daddr + ring->hw.size * sizeof(*ring->hw.descs); + desc->haddr = 0; + + COMMIT_DESC(ring, ring->tail, scratch); + + bump_tail(ring, 1); + count--; + } + + return 0; +} + +static inline void atl_get_rxpage(struct atl_pgref *pgref) +{ + pgref->rxpage->mapcount++; +} + +static inline void __atl_free_rxpage(struct atl_rxpage *rxpage, + struct device *dev) +{ + unsigned int len = PAGE_SIZE << rxpage->order; + + dma_unmap_page(dev, rxpage->daddr, len, DMA_FROM_DEVICE); + trace_atl_dma_unmap_rxbuf(-1, -1, rxpage->daddr, len, NULL); + + /* Drop the ref for dma mapping. */ + __free_pages(rxpage->page, rxpage->order); + kfree(rxpage); +} + +static inline void atl_put_rxpage(struct atl_pgref *pgref, struct device *dev) +{ + struct atl_rxpage *rxpage = pgref->rxpage; + + if (!rxpage) + return; + + if (--rxpage->mapcount) + return; + + __atl_free_rxpage(rxpage, dev); + pgref->rxpage = 0; +} + +static bool atl_recycle_or_put_page(struct atl_pgref *pgref, + unsigned int buf_len, struct device *dev) +{ + unsigned int order = pgref->rxpage->order; + unsigned int size = PAGE_SIZE << order; + struct page *page = pgref->rxpage->page; + + if (!page_is_pfmemalloc(page) && pgref->pg_off + buf_len < size) + return true; + + atl_put_rxpage(pgref, dev); + + return false; +} + +static void atl_maybe_recycle_rxbuf(struct atl_desc_ring *ring, + struct atl_rxbuf *rxbuf) +{ + int reused = 0; + struct atl_pgref *head = &rxbuf->head, *data = &rxbuf->data; + struct atl_rxbuf *new = &ring->rxbufs[ring->next_to_recycle]; + unsigned int data_len = ATL_RX_BUF_SIZE + + (atl_rx_linear ? ATL_RX_HDR_OVRHD : 0); + struct device *dev = &ring->nic->hw.pdev->dev; + + if (!atl_rx_linear && + atl_recycle_or_put_page(head, ATL_RX_HDR_SIZE + ATL_RX_HDR_OVRHD, + dev)) { + new->head = *head; + reused = 1; + atl_update_ring_stat(ring, rx.reused_head_page, 1); + } + head->rxpage = 0; + + if (atl_recycle_or_put_page(data, data_len, dev)) { + new->data = *data; + reused = 1; + atl_update_ring_stat(ring, rx.reused_data_page, 1); + } + data->rxpage = 0; + + if (reused) + bump_ptr(ring->next_to_recycle, ring, 1); +} + +static unsigned int atl_data_len(struct atl_rx_desc_wb *wb) +{ + unsigned int len = le16_to_cpu(wb->pkt_len); + + if (!wb->eop) + return ATL_RX_BUF_SIZE; + + if (!wb->rsc_cnt && wb->sph) + len -= wb->hdr_len; + + len &= ATL_RX_BUF_SIZE - 1; + return len ?: ATL_RX_BUF_SIZE; +} + +static void atl_sync_range(struct atl_desc_ring *ring, + struct atl_pgref *pgref, unsigned int offt, unsigned int len) +{ + dma_addr_t daddr = pgref->rxpage->daddr; + unsigned int pg_off = pgref->pg_off + offt; + struct device *dev = &ring->nic->hw.pdev->dev; + + dma_sync_single_range_for_cpu(dev, daddr, pg_off, len, DMA_FROM_DEVICE); + trace_atl_sync_rx_range(-1, daddr, pg_off, len); +} + +static struct sk_buff *atl_init_skb(struct atl_desc_ring *ring, + struct atl_rxbuf *rxbuf, struct atl_rx_desc_wb *wb) +{ + struct sk_buff *skb; + unsigned int hdr_len, alloc, tailroom, len; + unsigned int data_len = atl_data_len(wb); + void *hdr; + struct atl_pgref *pgref; + struct atl_nic *nic = ring->nic; + + if (atl_rx_linear) { + if (!wb->eop) { + atl_nic_err("Multi-frag packet in linear mode\n"); + atl_update_ring_stat(ring, rx.linear_dropped, 1); + return (void *)-1l; + } + + hdr_len = len = data_len; + tailroom = 0; + pgref = &rxbuf->data; + } else { + hdr_len = wb->hdr_len; + if (hdr_len == 0) { + atl_nic_err("Header parse error\n"); + return (void *)-1l; + } + + /* If entire packet fits into ATL_RX_HDR_SIZE, reserve + * enough space to pull the data part into skb head + * and make it linear, otherwise allocate space for + * hdr_len only + */ + len = (wb->sph ? hdr_len : 0) + data_len; + if (!wb->eop || len > ATL_RX_HDR_SIZE) + len = hdr_len; + + /* reserve space for potential __pskb_pull_tail() */ + tailroom = min(ATL_RX_TAILROOM, ATL_RX_HDR_SIZE - len); + pgref = &rxbuf->head; + } + + if (atl_rx_linear || (wb->sph && (wb->eop || !wb->rsc_cnt))) + atl_sync_range(ring, pgref, + ATL_RX_HEADROOM, hdr_len); + + alloc = len + tailroom + ATL_RX_HEADROOM; + alloc += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + alloc = SKB_DATA_ALIGN(alloc); + + hdr = atl_buf_vaddr(pgref); + skb = build_skb(hdr, alloc); + if (unlikely(!skb)) { + atl_update_ring_stat(ring, rx.alloc_skb_failed, 1); + return NULL; + } + + if (wb->rsc_cnt && !wb->eop) { + struct atl_cb *atl_cb = ATL_CB(skb); + + /* First frag of a multi-frag RSC packet. Either head or + * data buffer, depending on whether the header was + * split off by HW, might still be accessed by + * RSC. Delay processing till EOP. + */ + if (wb->sph) { + atl_cb->pgref = *pgref; + atl_cb->head = true; + /* Safe to sync the data buf. !wb->eop + * implies the data buffer is completely filled. + */ + atl_sync_range(ring, &rxbuf->data, 0, ATL_RX_BUF_SIZE); + } else { + atl_cb->pgref = rxbuf->data; + atl_cb->head = false; + /* No need to sync head fragment as nothing + * was DMA'd into it + */ + } + atl_get_rxpage(&atl_cb->pgref); + } + + pgref->pg_off += alloc; + page_ref_inc(pgref->rxpage->page); + + if (!atl_rx_linear && !wb->sph) { + atl_nic_dbg("Header not split despite non-zero hdr_len (%d)\n", + hdr_len); + /* Make skb head empty -- will copy the real header + * from the data buffer later + */ + hdr_len = 0; + } + + skb_reserve(skb, ATL_RX_HEADROOM); + skb_put(skb, hdr_len); + return skb; +} + +static inline void atl_skb_put_data(struct sk_buff *skb, + void *data, unsigned int len) +{ + memcpy(skb_tail_pointer(skb), data, len); + skb->tail += len; + skb->len += len; +} + +static struct sk_buff *atl_process_rx_frag(struct atl_desc_ring *ring, + struct atl_rxbuf *rxbuf, struct atl_rx_desc_wb *wb) +{ + bool first_frag = false; + bool hdr_split = !!wb->sph; + unsigned int hdr_len, data_len, aligned_data_len; + unsigned int data_offt = 0, to_pull = 0; + struct sk_buff *skb = rxbuf->skb; + struct atl_cb *atl_cb; + struct atl_pgref *headref = &rxbuf->head, *dataref = &rxbuf->data; + struct device *dev = &ring->nic->hw.pdev->dev; + bool is_ptp_ring = atl_is_ptp_ring(ring->nic, ring); + + if (unlikely(wb->rdm_err)) { + if (skb && skb != (void *)-1l) + dev_kfree_skb_any(skb); + + skb = (void *)-1l; + } + + if (!skb) { + /* First buffer of a packet */ + skb = atl_init_skb(ring, rxbuf, wb); + first_frag = true; + } else + rxbuf->skb = NULL; + + if (unlikely(!skb || skb == (void *)-1l)) + return skb; + + hdr_len = wb->hdr_len; + data_len = atl_data_len(wb); + if (is_ptp_ring) { + data_len -= atl_ptp_extract_ts(ring->nic, skb, atl_buf_vaddr(dataref), + data_len); + } + + if (atl_rx_linear) { + /* Linear skb mode. The entire packet was DMA'd into + * the data buffer and skb has already been built + * around it and dataref's pg_off has been increased + * in atl_init_skb() + */ + + atl_maybe_recycle_rxbuf(ring, rxbuf); + return skb; + } + + /* Align the start of the next buffer in the page. This also + * serves as truesize increment when the paged frag is added + * to skb. + */ + aligned_data_len = ALIGN(data_len, L1_CACHE_BYTES); + + if (first_frag && !hdr_split) + /* Header was not split off, so skip over it + * when adding the paged frag + */ + data_offt = hdr_len; + + if (!first_frag || wb->eop || !wb->rsc_cnt) { + atl_sync_range(ring, dataref, 0, data_len); + + /* If header was not split off by HW, remember to pull + * it into the skb head later. The rest of the data + * buffer might need to be pulled too for small + * packets, so delay the actual copying till later + */ + if (first_frag && !hdr_split) + to_pull = hdr_len; + } + + /* If the entire packet fits within ATL_RX_HDR_SIZE bytes, + * pull it into the skb head. This handles the header not + * having been split by HW case correctly too, as + * skb_headlen() will be zero in that case and data_len will + * hold the whole packet length. + */ + if (first_frag && skb_headlen(skb) + data_len <= ATL_RX_HDR_SIZE) { + to_pull = data_len; + /* Recycle the data buffer as we're copying the + * contents to skb head. + */ + aligned_data_len = 0; + } else { + /* Add the data buffer to paged frag list, skipping + * the un-split header if any -- it will be copied to + * skb head later. + */ + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, + dataref->rxpage->page, dataref->pg_off + data_offt, + data_len - data_offt, aligned_data_len); + page_ref_inc(dataref->rxpage->page); + } + + if (to_pull) + atl_skb_put_data(skb, atl_buf_vaddr(dataref), to_pull); + + /* Update the data buf's pg_off to point to free + * space. Header buf's offset was updated in atl_init_skb() + * for first frag of the packet only. + */ + dataref->pg_off += aligned_data_len; + atl_maybe_recycle_rxbuf(ring, rxbuf); + + if (first_frag || !wb->eop || !wb->rsc_cnt) + return skb; + + /* The last descriptor of RSC packet is done, unmap the head + * fragment. + */ + atl_cb = ATL_CB(skb); + + headref = &atl_cb->pgref; + if (unlikely(!headref->rxpage)) + return skb; + + if (likely(atl_cb->head)) { + atl_sync_range(ring, headref, ATL_RX_HEADROOM, hdr_len); + atl_put_rxpage(headref, dev); + } else { + atl_sync_range(ring, headref, 0, ATL_RX_BUF_SIZE); + /* Data buf's sync being delayed implies header was + * not split off by HW. Fix that now. + */ + atl_skb_put_data(skb, atl_buf_vaddr(headref), hdr_len); + atl_put_rxpage(headref, dev); + } + + return skb; +} + +unsigned int atl_rx_refill_batch = 16; +module_param_named(rx_refill_batch, atl_rx_refill_batch, uint, 0644); + +int atl_clean_rx(struct atl_desc_ring *ring, int budget, + rx_skb_handler_t rx_skb_func) +{ + unsigned int refill_batch = + min_t(typeof(atl_rx_refill_batch), atl_rx_refill_batch, + ring->hw.size - 1); + unsigned int packets = 0; + unsigned int bytes = 0; + struct sk_buff *skb; + + while (packets < budget) { + uint32_t space = ring_space(ring); + struct atl_rx_desc_wb *wb; + struct atl_rxbuf *rxbuf; + unsigned int len; + DECLARE_SCRATCH_DESC(scratch); + + if (space >= refill_batch) + atl_fill_rx(ring, space, true); + + rxbuf = &ring->rxbufs[ring->head]; + + wb = &DESC_PTR(ring, ring->head, scratch)->wb; + FETCH_DESC(ring, ring->head, scratch); + + if (!wb->dd) + break; + DESC_RMB(); + + trace_atl_rx_descr(ring->qvec->idx, ring->head, (u64 *)wb); + + skb = atl_process_rx_frag(ring, rxbuf, wb); + + /* Treat allocation errors as transient and retry later */ + if (!skb) { + struct atl_nic *nic = ring->nic; + + atl_nic_err("failed to alloc skb for RX packet\n"); + break; + } + + if (skb == (void *)-1l) + atl_maybe_recycle_rxbuf(ring, rxbuf); + + bump_head(ring, 1); + if (!wb->eop) { + uint32_t next = wb->rsc_cnt ? + le16_to_cpu(wb->next_desp) : + ring->head; + /* If atl_process_rx_flags() returned any + * other error this propagates the error to + * the next descriptor of the packet, + * preventing it from being treated as a start + * of a new packet later. + */ + ring->rxbufs[next].skb = skb; + atl_update_ring_stat(ring, rx.non_eop_descs, 1); + continue; + } + + if (skb == (void *)-1l) + continue; + + len = skb->len; + if (atl_rx_packet(skb, wb, ring, rx_skb_func)) { + packets++; + bytes += len; + } + } + + u64_stats_update_begin(&ring->syncp); + ring->stats.rx.bytes += bytes; + ring->stats.rx.packets += packets; + u64_stats_update_end(&ring->syncp); + + return packets; +} + +int atl_clean_hwts_rx(struct atl_desc_ring *ring, int budget) +{ + unsigned int refill_batch = + min_t(typeof(atl_rx_refill_batch), atl_rx_refill_batch, + ring->hw.size - 1); + unsigned int packets = 0; + + while (packets < budget) { + uint32_t space = ring_space(ring); + struct atl_rx_desc_hwts_wb *wb; + struct atl_rxbuf *rxbuf; + u64 ns; + DECLARE_SCRATCH_DESC(scratch); + + if (space >= refill_batch) + atl_fill_hwts_rx(ring, space, true); + + rxbuf = &ring->rxbufs[ring->head]; + + wb = &DESC_PTR(ring, ring->head, scratch)->hwts_wb; + FETCH_DESC(ring, ring->head, scratch); + + if (!wb->dd) + break; + DESC_RMB(); + + hw_atl_extract_hwts(&ring->nic->hw, wb, &ns); + atl_ptp_tx_hwtstamp(ring->nic, ns); + + bump_head(ring, 1); + packets++; + } + + return packets; +} + +unsigned int atl_min_intr_delay = 10; +module_param_named(min_intr_delay, atl_min_intr_delay, uint, 0644); + +static void atl_set_intr_throttle(struct atl_queue_vec *qvec) +{ + struct atl_hw *hw = &qvec->nic->hw; + atl_write(hw, ATL_INTR_THRTL(atl_qvec_intr(qvec)), + 1 << 0x1f | ((atl_min_intr_delay / 2) & 0x1ff) << 0x10); +} + +static int atl_poll(struct napi_struct *napi, int budget) +{ + struct atl_queue_vec *qvec; + + qvec = container_of(napi, struct atl_queue_vec, napi); + + return atl_poll_qvec(qvec, budget); +} + +int atl_poll_qvec(struct atl_queue_vec *qvec, int budget) +{ + struct atl_nic *nic; + bool clean_done; + int rx_cleaned; + + nic = qvec->nic; + + clean_done = atl_clean_tx(&qvec->tx); + rx_cleaned = atl_clean_rx(&qvec->rx, budget, atl_napi_receive_skb); + + clean_done &= (rx_cleaned < budget); + + if (!clean_done) + return budget; + + if (likely(qvec->type != ATL_QUEUE_PTP)) { + napi_complete_done(&qvec->napi, rx_cleaned); + atl_intr_enable(&nic->hw, BIT(atl_qvec_intr(qvec))); + /* atl_set_intr_throttle(&nic->hw, qvec->idx); */ + } + return rx_cleaned; +} + +/* XXX NOTE: only checked on device probe for now */ +#ifdef CONFIG_PCI_MSI +bool atl_enable_msi = true; +#else +bool atl_enable_msi /*= false*/; +#endif +module_param_named(msi, atl_enable_msi, bool, 0444); +bool atl_wq_non_msi /*= false*/; +module_param_named(wq_non_msi, atl_wq_non_msi, bool, 0444); + +static int atl_config_interrupts(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + unsigned int flags; + int ret; + + if (atl_enable_msi) { + int nvecs; + + nvecs = min_t(unsigned int, nic->requested_nvecs, + num_present_cpus()); + flags = PCI_IRQ_MSIX | PCI_IRQ_MSI; + ret = pci_alloc_irq_vectors(hw->pdev, + ATL_NUM_NON_RING_IRQS + 1, + ATL_NUM_NON_RING_IRQS + nvecs, + flags); + + /* pci_alloc_irq_vectors() never allocates less + * than min_vectors + */ + if (ret > 0) { + ret -= ATL_NUM_NON_RING_IRQS; + nic->flags |= ATL_FL_MULTIPLE_VECTORS; + return ret; + } + } + + atl_nic_warn("Couldn't allocate MSI-X / MSI vectors, falling back to legacy interrupts\n"); + + flags = PCI_IRQ_LEGACY; + ret = pci_alloc_irq_vectors(hw->pdev, 1, 1, flags); + if (ret < 0) { + atl_nic_err("Couldn't allocate legacy IRQ\n"); + return ret; + } + + nic->flags &= ~ATL_FL_MULTIPLE_VECTORS; + + return min_t(unsigned int, atl_max_queues_non_msi, num_present_cpus()); +} + +irqreturn_t atl_ring_irq(int irq, void *priv) +{ + struct napi_struct *napi = priv; + + napi_schedule_irqoff(napi); + return IRQ_HANDLED; +} + +void atl_ring_work(struct work_struct *work) +{ + struct legacy_irq_work *irq_work = to_irq_work(work); + + napi_schedule(irq_work->napi); +} + +void atl_clear_datapath(struct atl_nic *nic) +{ + int i; + struct atl_queue_vec *qvecs = nic->qvecs; + + /* If atl_reconfigure() have failed previously, + * atl_clear_datapath() can be called again on + * pci_ops->remove(), without an intervening + * atl_setup_datapath(). + */ + if (!test_and_clear_bit(ATL_ST_CONFIGURED, &nic->hw.state)) + return; + + atl_ptp_irq_free(nic); + atl_free_link_intr(nic); + + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) { + for (i = 0; i < nic->nvecs; i++) { + int vector = pci_irq_vector(nic->hw.pdev, + i + ATL_NUM_NON_RING_IRQS); + irq_set_affinity_hint(vector, NULL); + } + } + + pci_free_irq_vectors(nic->hw.pdev); + + if (!qvecs) + return; + + for (i = 0; i < nic->nvecs; i++) { + if (unlikely(!(nic->flags & ATL_FL_MULTIPLE_VECTORS))) + cancel_work_sync(qvecs[i].work); + netif_napi_del(&qvecs[i].napi); + } + + atl_ptp_ring_stop(nic); + + kfree(to_irq_work(qvecs[0].work)); + kfree(qvecs); + nic->qvecs = NULL; +} + +static void atl_calc_affinities(struct atl_nic *nic) +{ + int i; + unsigned int cpu; + + get_online_cpus(); + cpu = cpumask_first(cpu_online_mask); + + for (i = 0; i < nic->nvecs; i++) { + cpumask_t *cpumask = &nic->qvecs[i].affinity_hint; + + /* If more vectors got allocated (based on + * cpu_present_mask) than cpus currently online, + * spread the remaining vectors among online cpus. + */ + if (cpu >= nr_cpumask_bits) + cpu = cpumask_first(cpu_online_mask); + + cpumask_clear(cpumask); + cpumask_set_cpu(cpu, cpumask); + cpu = cpumask_next(cpu, cpu_online_mask); + } + put_online_cpus(); +} + +void atl_init_qvec(struct atl_nic *nic, struct atl_queue_vec *qvec, int idx) +{ + qvec->nic = nic; + qvec->idx = idx; + + qvec->rx.hw.reg_base = ATL_RX_RING(idx); + qvec->rx.hw.size = nic->requested_rx_size; + qvec->rx.nic = nic; + qvec->rx.qvec = qvec; + + qvec->tx.hw.reg_base = ATL_TX_RING(idx); + qvec->tx.hw.size = nic->requested_tx_size; + qvec->tx.nic = nic; + qvec->tx.qvec = qvec; + + u64_stats_init(&qvec->rx.syncp); + u64_stats_init(&qvec->tx.syncp); + + if (likely(qvec->type == ATL_QUEUE_REGULAR)) + netif_napi_add(nic->ndev, &qvec->napi, atl_poll, 64); +} + +int atl_setup_datapath(struct atl_nic *nic) +{ + struct legacy_irq_work *irq_work = NULL; + struct atl_queue_vec *qvec; + int nvecs, i, ret; + + nvecs = atl_config_interrupts(nic); + if (nvecs < 0) + return nvecs; + nic->nvecs = nvecs; + + qvec = kcalloc(nvecs, sizeof(*qvec), GFP_KERNEL); + if (!qvec) { + atl_nic_err("Couldn't alloc qvecs\n"); + ret = -ENOMEM; + goto err_alloc; + } + nic->qvecs = qvec; + + if (unlikely(!(nic->flags & ATL_FL_MULTIPLE_VECTORS))) { + irq_work = kcalloc(nvecs, sizeof(*irq_work), GFP_KERNEL); + if (!irq_work) { + ret = -ENOMEM; + goto err_alloc_work; + } + } + + ret = atl_alloc_link_intr(nic); + if (ret) + goto err_link_intr; + + ret = atl_ptp_irq_alloc(nic); + if (ret < 0) + goto err_ptp_intr; + + ret = atl_ptp_ring_start(nic); + if (ret < 0) + goto err_ptp_ring; + + for (i = 0; i < nvecs; i++, qvec++) { + atl_init_qvec(nic, qvec, i); + + if (unlikely(irq_work)) { + INIT_WORK(&irq_work[i].work, atl_ring_work); + irq_work[i].napi = &qvec->napi; + qvec->work = &irq_work[i].work; + } + } + + atl_calc_affinities(nic); + + nic->max_mtu = atl_rx_linear ? ATL_MAX_RX_LINEAR_MTU : ATL_MAX_MTU; + + set_bit(ATL_ST_CONFIGURED, &nic->hw.state); + return 0; + +err_ptp_ring: + atl_ptp_irq_free(nic); + +err_ptp_intr: + atl_free_link_intr(nic); + +err_link_intr: + kfree(irq_work); + +err_alloc_work: + kfree(nic->qvecs); + nic->qvecs = NULL; + +err_alloc: + pci_free_irq_vectors(nic->hw.pdev); + + return ret; +} + +static inline void atl_free_rxpage(struct atl_pgref *pgref, struct device *dev) +{ + struct atl_rxpage *rxpage = pgref->rxpage; + + if (!rxpage) + return; + + /* Unmap, dropping the ref for being mapped */ + __atl_free_rxpage(rxpage, dev); + pgref->rxpage = 0; +} + +/* Releases any skbs that may have been queued on ring positions yet + * to be processes by poll. The buffers are kept to be re-used after + * resume / thaw. */ +void atl_clear_rx_bufs(struct atl_desc_ring *ring) +{ + unsigned int bufs = ring_occupied(ring); + struct device *dev = &ring->nic->hw.pdev->dev; + + while (bufs) { + struct atl_rxbuf *rxbuf = &ring->rxbufs[ring->head]; + struct sk_buff *skb = rxbuf->skb; + + if (skb) { + struct atl_pgref *pgref = &ATL_CB(skb)->pgref; + + atl_put_rxpage(pgref, dev); + dev_kfree_skb_any(skb); + rxbuf->skb = NULL; + } + + bump_head(ring, 1); + bufs--; + } +} + +static void atl_free_rx_bufs(struct atl_desc_ring *ring) +{ + struct device *dev = &ring->nic->hw.pdev->dev; + struct atl_rxbuf *rxbuf; + + if (!ring->rxbufs) + return; + + for (rxbuf = ring->rxbufs; + rxbuf < &ring->rxbufs[ring->hw.size]; rxbuf++) { + atl_free_rxpage(&rxbuf->head, dev); + atl_free_rxpage(&rxbuf->data, dev); + } +} + +static void atl_free_tx_bufs(struct atl_desc_ring *ring) +{ + unsigned int bufs = ring_occupied(ring); + + if (!ring->txbufs) + return; + + while (bufs) { + struct atl_txbuf *txbuf; + + bump_tail(ring, -1); + txbuf = &ring->txbufs[ring->tail]; + + atl_txbuf_free(txbuf, &ring->nic->hw.pdev->dev, ring->tail); + bufs--; + } +} + +static size_t atl_ring_extra_size(struct atl_desc_ring *ring) +{ + switch (ring->qvec->type) { + case ATL_QUEUE_REGULAR: + case ATL_QUEUE_PTP: + return 0; + case ATL_QUEUE_HWTS: + return ATL_RX_BUF_SIZE; + default: + WARN_ONCE(true, "Unknown queue type\n"); + break; + } + + return 0; +} + +static void atl_free_ring(struct atl_desc_ring *ring) +{ + size_t extra = atl_ring_extra_size(ring); + + if (ring->bufs) { + vfree(ring->bufs); + ring->bufs = 0; + } + + atl_free_descs(ring->nic, &ring->hw, extra); +} + +static int atl_alloc_ring(struct atl_desc_ring *ring, size_t buf_size, + char *type) +{ + size_t extra = atl_ring_extra_size(ring); + struct atl_nic *nic = ring->nic; + int idx = ring->qvec->idx; + int ret; + + ret = atl_alloc_descs(nic, &ring->hw, extra); + if (ret) { + atl_nic_err("Couldn't alloc %s[%d] descriptors\n", type, idx); + return ret; + } + + if (likely(ring->qvec->type != ATL_QUEUE_HWTS)) { + ring->bufs = vzalloc(ring->hw.size * buf_size); + if (!ring->bufs) { + ret = -ENOMEM; + goto free; + } + } + + ring->head = ring->tail = + atl_read(&nic->hw, ATL_RING_HEAD(ring)) & 0x1fff; + return 0; + +free: + atl_free_ring(ring); + return ret; +} + +static void atl_set_affinity(int vector, struct atl_queue_vec *qvec) +{ + cpumask_t *cpumask = qvec ? &qvec->affinity_hint : NULL; + + irq_set_affinity_hint(vector, cpumask); +} + +static int atl_alloc_qvec_intr(struct atl_queue_vec *qvec) +{ + struct atl_nic *nic = qvec->nic; + int vector; + int ret; + + snprintf(qvec->name, sizeof(qvec->name), "%s-ring-%d", + nic->ndev->name, qvec->idx); + + if (!(nic->flags & ATL_FL_MULTIPLE_VECTORS)) + return 0; + + vector = pci_irq_vector(nic->hw.pdev, atl_qvec_intr(qvec)); + ret = request_irq(vector, atl_ring_irq, IRQF_NO_SUSPEND, + qvec->name, &qvec->napi); + if (ret) { + atl_nic_err("request MSI ring vector failed: %d\n", -ret); + return ret; + } + + atl_set_affinity(vector, qvec); + + return 0; +} + +static void atl_free_qvec_intr(struct atl_queue_vec *qvec) +{ + struct atl_nic *nic = qvec->nic; + int vector; + + if (unlikely(!nic)) + return; + + if (!(nic->flags & ATL_FL_MULTIPLE_VECTORS)) + return; + + vector = pci_irq_vector(nic->hw.pdev, atl_qvec_intr(qvec)); + atl_set_affinity(vector, NULL); + free_irq(vector, &qvec->napi); +} + +int atl_alloc_qvec(struct atl_queue_vec *qvec) +{ + struct atl_txbuf *txbuf; + int count = qvec->tx.hw.size; + int ret = 0; + + switch (qvec->type) { + case ATL_QUEUE_REGULAR: + ret = atl_alloc_qvec_intr(qvec); + break; + case ATL_QUEUE_PTP: + case ATL_QUEUE_HWTS: + break; + default: + WARN_ONCE(true, "Unknown queue type\n"); + break; + } + + if (ret) + return ret; + + if (likely(qvec->type != ATL_QUEUE_HWTS)) { + ret = atl_alloc_ring(&qvec->tx, sizeof(struct atl_txbuf), "tx"); + if (ret) + goto free_irq; + } + + ret = atl_alloc_ring(&qvec->rx, sizeof(struct atl_rxbuf), "rx"); + if (ret) + goto free_tx; + + if (likely(qvec->type != ATL_QUEUE_HWTS)) { + for (txbuf = qvec->tx.txbufs; count; count--) + (txbuf++)->last = -1; + } + + return 0; + +free_tx: + if (likely(qvec->type != ATL_QUEUE_HWTS)) + atl_free_ring(&qvec->tx); +free_irq: + switch (qvec->type) { + case ATL_QUEUE_REGULAR: + atl_free_qvec_intr(qvec); + break; + default: + break; + } + + return ret; +} + +void atl_free_qvec(struct atl_queue_vec *qvec) +{ + struct atl_desc_ring *rx = &qvec->rx; + struct atl_desc_ring *tx = &qvec->tx; + + atl_free_rx_bufs(rx); + atl_free_ring(rx); + + if (likely(qvec->type != ATL_QUEUE_HWTS)) + atl_free_ring(tx); + + switch (qvec->type) { + case ATL_QUEUE_REGULAR: + atl_free_qvec_intr(qvec); + break; + case ATL_QUEUE_PTP: + case ATL_QUEUE_HWTS: + break; + default: + WARN_ONCE(true, "Unknown queue type\n"); + break; + } +} + +int atl_alloc_rings(struct atl_nic *nic) +{ + struct atl_queue_vec *qvec; + int ret; + + atl_for_each_qvec(nic, qvec) { + ret = atl_alloc_qvec(qvec); + if (ret) + goto free; + } + + return 0; + +free: + while(--qvec >= &nic->qvecs[0]) + atl_free_qvec(qvec); + + return ret; +} + +void atl_free_rings(struct atl_nic *nic) +{ + struct atl_queue_vec *qvec; + + atl_for_each_qvec(nic, qvec) + atl_free_qvec(qvec); + +} + +static unsigned int atl_rx_mod_hyst = 10, atl_tx_mod_hyst = 10; +module_param_named(rx_mod_hyst, atl_rx_mod_hyst, uint, 0644); +module_param_named(tx_mod_hyst, atl_tx_mod_hyst, uint, 0644); + +static void atl_set_intr_mod_qvec(struct atl_queue_vec *qvec) +{ + struct atl_nic *nic = qvec->nic; + struct atl_hw *hw = &nic->hw; + unsigned int min, max; + int idx = qvec->idx; + uint32_t reg; + + min = nic->rx_intr_delay - atl_min_intr_delay; + max = min + atl_rx_mod_hyst; + + atl_write(hw, ATL_RX_INTR_MOD_CTRL(idx), + (max / 2) << 0x10 | (min / 2) << 8 | 2); + + min = nic->tx_intr_delay - atl_min_intr_delay; + max = min + atl_tx_mod_hyst; + + if (hw->chip_id == ATL_ANTIGUA) + reg = ATL2_TX_INTR_MOD_CTRL(idx); + else + reg = ATL_TX_INTR_MOD_CTRL(idx); + atl_write(hw, reg, (max / 2) << 0x10 | (min / 2) << 8 | 2); +} + +void atl_set_intr_mod(struct atl_nic *nic) +{ + struct atl_queue_vec *qvec; + + atl_for_each_qvec(nic, qvec) + atl_set_intr_mod_qvec(qvec); +} + +int atl_init_rx_ring(struct atl_desc_ring *rx) +{ + struct atl_hw *hw = &rx->nic->hw; + struct atl_rxbuf *rxbuf; + int ret = 0; + + rx->head = rx->tail = atl_read(hw, ATL_RING_HEAD(rx)) & 0xffff; + if (rx->head > 0x1FFF) + return -EIO; + + switch (rx->qvec->type) { + case ATL_QUEUE_HWTS: + ret = atl_fill_hwts_rx(rx, ring_space(rx), false); + break; + case ATL_QUEUE_PTP: + case ATL_QUEUE_REGULAR: + ret = atl_fill_rx(rx, ring_space(rx), false); + break; + default: + WARN_ONCE(true, "Unknown queue type\n"); + break; + } + + if (ret) + return ret; + + if (likely(rx->qvec->type != ATL_QUEUE_HWTS)) { + rx->next_to_recycle = rx->tail; + /* rxbuf at ->next_to_recycle is always kept empty so that + * atl_maybe_recycle_rxbuf() always have a spot to recycle into + * without overwriting a pgref to an already allocated page, + * leaking memory. It's also the guard element in the ring + * that keeps ->tail from overrunning ->head. If it's nonempty + * on ring init (e.g. after a sleep-wake cycle) just release + * the pages. + */ + rxbuf = &rx->rxbufs[rx->next_to_recycle]; + atl_put_rxpage(&rxbuf->head, &hw->pdev->dev); + atl_put_rxpage(&rxbuf->data, &hw->pdev->dev); + } + + return 0; +} + +int atl_init_tx_ring(struct atl_desc_ring *tx) +{ + struct atl_hw *hw = &tx->nic->hw; + + tx->head = tx->tail = atl_read(hw, ATL_RING_HEAD(tx)) & 0xffff; + if (tx->head > 0x1FFF) + return -EIO; + + return 0; +} + +static void atl_start_rx_ring(struct atl_desc_ring *ring) +{ + struct atl_hw *hw = &ring->nic->hw; + int idx = ring->qvec->idx; + unsigned int rx_ctl; + + atl_write(hw, ATL_RING_BASE_LSW(ring), ring->hw.daddr); + atl_write(hw, ATL_RING_BASE_MSW(ring), upper_32_bits(ring->hw.daddr)); + + atl_write(hw, ATL_RX_RING_TAIL(ring), ring->tail); + switch (ring->qvec->type) { + case ATL_QUEUE_REGULAR: + atl_write(hw, ATL_RX_RING_BUF_SIZE(ring), + (ATL_RX_HDR_SIZE / 64) << 8 | ATL_RX_BUF_SIZE / 1024); + break; + case ATL_QUEUE_PTP: + case ATL_QUEUE_HWTS: + atl_write(hw, ATL_RX_RING_BUF_SIZE(ring), + ATL_RX_BUF_SIZE / 1024); + break; + default: + WARN_ONCE(true, "Unknown queue type\n"); + break; + } + atl_write(hw, ATL_RX_RING_THRESH(ring), 8 << 0x10 | 24 << 0x18); + + /* LRO */ + atl_write_bits(hw, ATL_RX_LRO_PKT_LIM(idx), + (idx & 7) * 4, 2, 3); + + /* Enable ring | VLAN offload */ + rx_ctl = BIT(31) | BIT(29) | ring->hw.size; + switch (ring->qvec->type) { + case ATL_QUEUE_REGULAR: + /* Enable header split in non-linear mode */ + rx_ctl |= (atl_rx_linear ? 0 : BIT(28)); + break; + case ATL_QUEUE_PTP: + case ATL_QUEUE_HWTS: + break; + default: + break; + } + atl_write(hw, ATL_RX_RING_CTL(ring), rx_ctl); +} + +static void atl_start_tx_ring(struct atl_desc_ring *ring) +{ + struct atl_nic *nic = ring->nic; + struct atl_hw *hw = &nic->hw; + + atl_write(hw, ATL_RING_BASE_LSW(ring), ring->hw.daddr); + atl_write(hw, ATL_RING_BASE_MSW(ring), upper_32_bits(ring->hw.daddr)); + + /* Enable TSO on all active Tx rings */ + atl_write(hw, ATL_TX_LSO_CTRL, BIT(nic->nvecs) - 1); + + atl_write(hw, ATL_TX_RING_TAIL(ring), ring->tail); + switch (ring->qvec->type) { + case ATL_QUEUE_REGULAR: + atl_write(hw, ATL_TX_RING_THRESH(ring), 8 << 8 | 8 << 0x10 | + 24 << 0x18); + break; + case ATL_QUEUE_PTP: + case ATL_QUEUE_HWTS: + atl_write(hw, ATL_TX_RING_THRESH(ring), 0); + break; + default: + WARN_ONCE(true, "Unknown queue type\n"); + break; + } + atl_write(hw, ATL_TX_RING_CTL(ring), BIT(31) | ring->hw.size); +} + +int atl_start_qvec(struct atl_queue_vec *qvec) +{ + struct atl_desc_ring *rx = &qvec->rx; + struct atl_desc_ring *tx = &qvec->tx; + struct atl_hw *hw = &qvec->nic->hw; + int intr = atl_qvec_intr(qvec); + int ret; + + ret = atl_init_rx_ring(rx); + if (ret) + return ret; + if (likely(qvec->type != ATL_QUEUE_HWTS)) { + ret = atl_init_tx_ring(tx); + if (ret) + return ret; + } + + /* Map ring interrups into corresponding cause bit*/ + atl_set_intr_bits(hw, qvec->idx, intr, intr); + atl_set_intr_throttle(qvec); + + if (likely(qvec->type == ATL_QUEUE_REGULAR)) + napi_enable(&qvec->napi); + atl_set_intr_mod_qvec(qvec); + atl_intr_enable(hw, BIT(intr)); + + if (likely(qvec->type != ATL_QUEUE_HWTS)) + atl_start_tx_ring(tx); + atl_start_rx_ring(rx); + + return 0; +} + +void atl_stop_qvec(struct atl_queue_vec *qvec) +{ + struct atl_desc_ring *rx = &qvec->rx; + struct atl_desc_ring *tx = &qvec->tx; + struct atl_hw *hw = &qvec->nic->hw; + + /* Disable and reset rings */ + atl_write(hw, ATL_RING_CTL(rx), BIT(25)); + if (likely(qvec->type != ATL_QUEUE_HWTS)) + atl_write(hw, ATL_RING_CTL(tx), BIT(25)); + udelay(10); + atl_write(hw, ATL_RING_CTL(rx), 0); + if (likely(qvec->type != ATL_QUEUE_HWTS)) + atl_write(hw, ATL_RING_CTL(tx), 0); + + atl_intr_disable(hw, BIT(atl_qvec_intr(qvec))); + if (likely(qvec->type == ATL_QUEUE_REGULAR)) + napi_disable(&qvec->napi); + + if (likely(qvec->type != ATL_QUEUE_HWTS)) { + atl_clear_rx_bufs(rx); + atl_free_tx_bufs(tx); + } +} + +static void atl_set_lro(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + uint32_t val = nic->ndev->features & NETIF_F_LRO ? + BIT(nic->nvecs) - 1 : 0; + + if (val) + atl_nic_warn("There are unresolved issues with LRO, enabling it isn't recommended for now\n"); + + atl_write_bits(hw, ATL_RX_LRO_CTRL1, 0, nic->nvecs, val); + atl_write_bits(hw, ATL_INTR_RSC_EN, 0, nic->nvecs, val); +} + +int atl_start_rings(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + uint32_t mask; + struct atl_queue_vec *qvec; + int ret; + + if (test_bit(ATL_ST_RINGS_RUNNING, &hw->state)) + return 0; + + if (nic->flags & ATL_FL_MULTIPLE_VECTORS) { + mask = BIT(nic->nvecs + ATL_NUM_NON_RING_IRQS) - + BIT(ATL_NUM_NON_RING_IRQS); + /* Enable auto-masking of ring interrupts on intr generation */ + atl_set_bits(hw, ATL_INTR_AUTO_MASK, mask); + /* Enable status auto-clear on intr generation */ + atl_set_bits(hw, ATL_INTR_AUTO_CLEAR, mask); + } + + atl_set_lro(nic); + ret = atl_set_rss_tbl(hw); + if (ret) + return ret; + + atl_for_each_qvec(nic, qvec) { + ret = atl_start_qvec(qvec); + if (ret) + goto stop; + } + + set_bit(ATL_ST_RINGS_RUNNING, &hw->state); + netif_tx_start_all_queues(nic->ndev); + + return 0; + +stop: + while (--qvec >= &nic->qvecs[0]) + atl_stop_qvec(qvec); + + return ret; +} + +void atl_clear_tdm_cache(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + atl_write_bit(hw, 0x7b00, 0, 1); + udelay(10); + atl_write_bit(hw, 0x7b00, 0, 0); +} + +void atl_clear_rdm_cache(struct atl_nic *nic) +{ + struct atl_hw *hw = &nic->hw; + + atl_write_bit(hw, 0x5a00, 0, 1); + udelay(10); + atl_write_bit(hw, 0x5a00, 0, 0); +} + +void atl_stop_rings(struct atl_nic *nic) +{ + struct atl_queue_vec *qvec; + + if (!test_and_clear_bit(ATL_ST_RINGS_RUNNING, &nic->hw.state)) + return; + + netif_tx_stop_all_queues(nic->ndev); + + atl_for_each_qvec(nic, qvec) + atl_stop_qvec(qvec); + + atl_clear_rdm_cache(nic); + atl_clear_tdm_cache(nic); +} + +int atl_set_features(struct net_device *ndev, netdev_features_t features) +{ + netdev_features_t changed = ndev->features ^ features; + + ndev->features = features; + + if (changed & NETIF_F_LRO) + atl_set_lro(netdev_priv(ndev)); + + return 0; +} + +void atl_get_ring_stats(struct atl_desc_ring *ring, + struct atl_ring_stats *stats) +{ + unsigned int start; + + do { + start = u64_stats_fetch_begin_irq(&ring->syncp); + memcpy(stats, &ring->stats, sizeof(*stats)); + } while (u64_stats_fetch_retry_irq(&ring->syncp, start)); +} + +#define atl_add_stats(_dst, _src) \ +do { \ + int i; \ + uint64_t *dst = (uint64_t *)(&(_dst)); \ + uint64_t *src = (uint64_t *)(&(_src)); \ + \ + for (i = 0; i < sizeof(_dst) / sizeof(uint64_t); i++) \ + dst[i] += src[i]; \ +} while (0) + +void atl_update_global_stats(struct atl_nic *nic) +{ + int i; + struct atl_ring_stats stats; + + if (!test_bit(ATL_ST_ENABLED, &nic->hw.state) || + test_bit(ATL_ST_RESETTING, &nic->hw.state) || + !test_bit(ATL_ST_CONFIGURED, &nic->hw.state)) + return; + + memset(&stats, 0, sizeof(stats)); + + spin_lock(&nic->stats_lock); + + memset(&nic->stats.rx, 0, sizeof(nic->stats.rx)); + memset(&nic->stats.tx, 0, sizeof(nic->stats.tx)); + + + for (i = 0; i < nic->nvecs; i++) { + atl_get_ring_stats(&nic->qvecs[i].rx, &stats); + atl_add_stats(nic->stats.rx, stats.rx); + + atl_get_ring_stats(&nic->qvecs[i].tx, &stats); + atl_add_stats(nic->stats.tx, stats.tx); + } + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + for (i = 0; i < ATL_NUM_FWD_RINGS; i++) { + if (atlfwd_nl_is_tx_fwd_ring_created(nic->ndev, i)) { + atl_fwd_get_ring_stats(nic->fwd.rings[ATL_FWDIR_TX][i], + &stats); + atl_add_stats(nic->stats.tx, stats.tx); + } + if (atlfwd_nl_is_rx_fwd_ring_created(nic->ndev, i)) { + atl_fwd_get_ring_stats(nic->fwd.rings[ATL_FWDIR_RX][i], + &stats); + atl_add_stats(nic->stats.rx, stats.rx); + } + } +#endif + + spin_unlock(&nic->stats_lock); +} + +void atl_get_stats64(struct net_device *ndev, + struct rtnl_link_stats64 *nstats) +{ + struct atl_nic *nic = netdev_priv(ndev); + struct atl_global_stats *stats = &nic->stats; + + atl_update_global_stats(nic); + + nstats->rx_bytes = stats->rx.bytes + stats->rx_fwd.bytes; + nstats->rx_packets = stats->rx.packets + stats->rx_fwd.packets; + nstats->tx_bytes = stats->tx.bytes; + nstats->tx_packets = stats->tx.packets; + nstats->rx_crc_errors = stats->rx.csum_err; + nstats->rx_frame_errors = stats->rx.mac_err; + nstats->rx_errors = nstats->rx_crc_errors + nstats->rx_frame_errors; + nstats->multicast = stats->rx.multicast; + nstats->tx_aborted_errors = stats->tx.dma_map_failed; + nstats->tx_errors = nstats->tx_aborted_errors; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.h new file mode 100644 index 000000000000..e35bab18a647 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring.h @@ -0,0 +1,235 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_RING_H_ +#define _ATL_RING_H_ + +#include + +#include "atl_common.h" +#include "atl_desc.h" +#include "atl_ring_desc.h" +#include "atl_ptp.h" + +//#define ATL_RINGS_IN_UC_MEM + +#define ATL_TX_DESC_WB +//#define ATL_TX_HEAD_WB + +#define ATL_RX_HEADROOM (NET_SKB_PAD + NET_IP_ALIGN) +#define ATL_RX_TAILROOM 64u +#define ATL_RX_HEAD_ORDER 0 +#define ATL_RX_DATA_ORDER 0 + +/* Header space in skb. Must be a multiple of L1_CACHE_BYTES */ +#define ATL_RX_HDR_SIZE 256u +#define ATL_RX_HDR_OVRHD SKB_DATA_ALIGN(ATL_RX_HEADROOM + \ + SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) +#define ATL_RX_BUF_SIZE 2048 + +#define ATL_MAX_RX_LINEAR_MTU (ATL_RX_BUF_SIZE - ETH_HLEN) + +#define ring_space(ring) \ + ({ \ + struct atl_desc_ring *__ring = (ring); \ + uint32_t space = READ_ONCE(__ring->head) - \ + READ_ONCE(__ring->tail) - 1; \ + (int32_t)space < 0 ? space + __ring->hw.size : space; \ + }) + +#define ring_occupied(ring) \ + ({ \ + struct atl_desc_ring *__ring = (ring); \ + uint32_t occupied = READ_ONCE(__ring->tail) - \ + READ_ONCE(__ring->head); \ + (int32_t)occupied < 0 ? occupied + __ring->hw.size \ + : occupied; \ + }) + +#define bump_ptr(ptr, ring, amount) \ + ({ \ + struct atl_desc_ring *__ring = (ring); \ + uint32_t __res = offset_ptr(ptr, &__ring->hw, amount); \ + (ptr) = __res; \ + __res; \ + }) + +/* These don't have to be atomic, because Tx tail is only adjusted + * in ndo->start_xmit which is serialized by the stack and the rest are + * only adjusted in NAPI poll which is serialized by NAPI */ +#define bump_tail(ring, amount) do { \ + struct atl_desc_ring *__ring = (ring); \ + uint32_t __ptr = READ_ONCE(__ring->tail); \ + __ring->tail = offset_ptr(__ptr, &__ring->hw, amount);\ + } while (0) + +#define bump_head(ring, amount) do { \ + struct atl_desc_ring *__ring = (ring); \ + uint32_t __ptr = READ_ONCE(__ring->head); \ + __ring->head = offset_ptr(__ptr, &__ring->hw, amount);\ + } while (0) + +struct atl_rxpage { + struct page *page; + dma_addr_t daddr; + unsigned mapcount; /* not atomic_t because accesses are + * serialized by NAPI */ + unsigned order; +}; + +struct atl_pgref { + struct atl_rxpage *rxpage; + unsigned pg_off; +}; + +struct atl_cb { + struct atl_pgref pgref; + bool head; +}; +#define ATL_CB(skb) ((struct atl_cb *)(skb)->cb) + +struct atl_rxbuf { + struct sk_buff *skb; + struct atl_pgref head; + struct atl_pgref data; +}; + +struct atl_txbuf { + struct sk_buff *skb; + uint32_t last; /* index of eop descriptor */ + unsigned bytes; + unsigned packets; + DEFINE_DMA_UNMAP_ADDR(daddr); + DEFINE_DMA_UNMAP_LEN(len); +}; + +struct legacy_irq_work { + struct work_struct work; + + struct napi_struct *napi; +}; +static inline struct legacy_irq_work *to_irq_work(struct work_struct *work) +{ + return container_of(work, struct legacy_irq_work, work); +}; + +enum atl_queue_type { + ATL_QUEUE_REGULAR, + ATL_QUEUE_PTP, + ATL_QUEUE_HWTS, +}; + +struct ____cacheline_aligned atl_queue_vec { + struct atl_desc_ring tx; + struct atl_desc_ring rx; + struct napi_struct napi; + struct atl_nic *nic; + unsigned idx; + char name[IFNAMSIZ + 10]; + cpumask_t affinity_hint; + enum atl_queue_type type; + struct work_struct *work; +}; + +#define atl_for_each_qvec(nic, qvec) \ + for (qvec = &(nic)->qvecs[0]; \ + qvec < &(nic)->qvecs[(nic)->nvecs]; qvec++) + +static inline struct atl_hw *ring_hw(struct atl_desc_ring *ring) +{ + return &ring->nic->hw; +} + +void atl_init_qvec(struct atl_nic *nic, struct atl_queue_vec *qvec, int idx); +int atl_alloc_qvec(struct atl_queue_vec *qvec); +void atl_free_qvec(struct atl_queue_vec *qvec); +int atl_start_qvec(struct atl_queue_vec *qvec); +void atl_stop_qvec(struct atl_queue_vec *qvec); +int atl_poll_qvec(struct atl_queue_vec *qvec, int budget); + +static inline int atl_qvec_intr(struct atl_queue_vec *qvec) +{ +#if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) + if (unlikely(qvec->idx >= qvec->nic->nvecs)) + return atl_ptp_qvec_intr(qvec); +#endif + + return qvec->idx + ATL_NUM_NON_RING_IRQS; +} + +static inline void *atl_buf_vaddr(struct atl_pgref *pgref) +{ + return page_to_virt(pgref->rxpage->page) + pgref->pg_off; +} + +static inline dma_addr_t atl_buf_daddr(struct atl_pgref *pgref) +{ + return pgref->rxpage->daddr + pgref->pg_off; +} + +void atl_get_ring_stats(struct atl_desc_ring *ring, + struct atl_ring_stats *stats); +#define atl_update_ring_stat(ring, stat, delta) \ +do { \ + struct atl_desc_ring *_ring = (ring); \ + \ + u64_stats_update_begin(&_ring->syncp); \ + _ring->stats.stat += (delta); \ + u64_stats_update_end(&_ring->syncp); \ +} while (0) + +int atl_init_rx_ring(struct atl_desc_ring *rx); +int atl_init_tx_ring(struct atl_desc_ring *tx); + +netdev_tx_t atl_map_skb(struct sk_buff *skb, struct atl_desc_ring *ring); +int atl_tx_full(struct atl_desc_ring *ring, int needed); + +typedef int (*rx_skb_handler_t)(struct atl_desc_ring *ring, + struct sk_buff *skb); +int atl_clean_rx(struct atl_desc_ring *ring, int budget, + rx_skb_handler_t rx_skb_func); +int atl_clean_hwts_rx(struct atl_desc_ring *ring, int budget); +void atl_clear_rx_bufs(struct atl_desc_ring *ring); + +#ifdef ATL_RINGS_IN_UC_MEM + +#define DECLARE_SCRATCH_DESC(_name) union atl_desc _name +#define DESC_PTR(_ring, _idx, _scratch) (&(_scratch)) +#define COMMIT_DESC(_ring, _idx, _scratch) \ + (_ring)->hw.descs[_idx] = (_scratch) +#define FETCH_DESC(_ring, _idx, _scratch) \ +do { \ + (_scratch) = READ_ONCE((_ring)->hw.descs[_idx]); \ + dma_rmb(); \ +} while(0) + +#define DESC_RMB() + +#else // ATL_RINGS_IN_UC_MEM + +#define DECLARE_SCRATCH_DESC(_name) +#define DESC_PTR(_ring, _idx, _scratch) (&(_ring)->hw.descs[_idx]) +#define COMMIT_DESC(_ring, _idx, _scratch) +#define FETCH_DESC(_ring, _idx, _scratch) +#define DESC_RMB() dma_rmb() + +#endif // ATL_RINGS_IN_UC_MEM + +#ifdef ATL_TX_HEAD_WB +#error Head ptr writeback not implemented +#elif !defined(ATL_TX_DESC_WB) +static inline uint32_t atl_get_tx_head(struct atl_desc_ring *ring) +{ + return atl_read(ring_hw(ring), ATL_TX_RING_HEAD(ring->idx)) & 0x1fff; +} +#endif + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring_desc.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring_desc.h new file mode 100644 index 000000000000..9fe1fe338f92 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_ring_desc.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_RING_DESC_H_ +#define _ATL_RING_DESC_H_ + +#include +#include + +#include "atl_desc.h" +#include "atl_hw.h" +#include "atl_stats.h" + +struct atl_nic; +struct atl_fwd_event; + +struct atl_desc_ring { + struct atl_hw_ring hw; + struct atl_nic *nic; + uint32_t head; + uint32_t tail; + union { + /* Rx ring only */ + uint32_t next_to_recycle; + /* Tx ring only, template desc for atl_map_tx_skb() */ + union atl_desc desc; + }; + union { + struct atl_rxbuf *rxbufs; + struct atl_txbuf *txbufs; + void *bufs; + }; + struct atl_queue_vec *qvec; + struct u64_stats_sync syncp; + struct atl_ring_stats stats; +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) + u32 tx_hw_head; + union { + struct atl_fwd_event *tx_evt; + struct atl_fwd_event *rx_evt; + }; + /* RX ring polling */ + struct timer_list *rx_poll_timer; +#endif +}; + +#endif /* _ATL_RING_DESC_H_ */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_stats.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_stats.h new file mode 100644 index 000000000000..0c6b73b7f494 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_stats.h @@ -0,0 +1,87 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATL_STATS_H_ +#define _ATL_STATS_H_ + +#include + +struct atl_rx_ring_stats { + uint64_t packets; + uint64_t bytes; + uint64_t linear_dropped; + uint64_t alloc_skb_failed; + uint64_t reused_head_page; + uint64_t reused_data_page; + uint64_t alloc_head_page; + uint64_t alloc_data_page; + uint64_t alloc_head_page_failed; + uint64_t alloc_data_page_failed; + uint64_t non_eop_descs; + uint64_t mac_err; + uint64_t csum_err; + uint64_t multicast; +}; + +struct atl_rx_fwd_ring_stats { + uint64_t packets; + uint64_t bytes; +}; + +struct atl_tx_ring_stats { + uint64_t packets; + uint64_t bytes; + uint64_t tx_busy; + uint64_t tx_restart; + uint64_t dma_map_failed; +}; + +struct atl_ring_stats { + union { + struct atl_rx_ring_stats rx; + struct atl_tx_ring_stats tx; + }; +}; + +struct atl_ether_stats { + uint64_t rx_pause; + uint64_t tx_pause; + uint64_t rx_ether_drops; + uint64_t rx_ether_octets; + uint64_t rx_ether_pkts; + uint64_t rx_ether_broacasts; + uint64_t rx_ether_multicasts; + uint64_t rx_ether_crc_align_errs; + uint64_t rx_filter_host; + uint64_t rx_filter_lost; +}; + +struct atl_global_stats { + struct atl_rx_ring_stats rx; + struct atl_tx_ring_stats tx; + struct atl_rx_fwd_ring_stats rx_fwd; + + /* MSM counters can't be reset without full HW reset, so + * store them in relative form: + * eth[i] == HW_counter - eth_base[i] + */ + struct atl_ether_stats eth; + struct atl_ether_stats eth_base; +}; + +struct atl_fwd_ring; + +#if IS_ENABLED(CONFIG_ATLFWD_FWD_NETLINK) +void atl_fwd_get_ring_stats(struct atl_fwd_ring *ring, + struct atl_ring_stats *stats); +#endif + +#endif /* _ATL_STATS_H_ */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.c new file mode 100644 index 000000000000..3d8b73cd8aa9 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include + +#define CREATE_TRACE_POINTS +#include "atl_trace.h" diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.h new file mode 100644 index 000000000000..f405f8e96c00 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atl_trace.h @@ -0,0 +1,274 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2017 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM atlnew + +#if !defined(_ATL_TRACE_H) || defined(TRACE_HEADER_MULTI_READ) +#define _ATL_TRACE_H + +#include +#include "atl_desc.h" + +DECLARE_EVENT_CLASS(atl_dma_map_class, + TP_PROTO(int frag_idx, int ring_idx, dma_addr_t daddr, size_t size, struct sk_buff *skb, + void *vaddr), + TP_ARGS(frag_idx, ring_idx, daddr, size, skb, vaddr), + TP_STRUCT__entry( + __field(int, frag_idx) + __field(int, ring_idx) + __field(dma_addr_t, daddr) + __field(size_t, size) + __field(struct sk_buff *, skb) + __field(void *, vaddr) + ), + TP_fast_assign( + __entry->frag_idx = frag_idx; + __entry->ring_idx = ring_idx; + __entry->daddr = daddr; + __entry->size = size; + __entry->skb = skb; + __entry->vaddr = vaddr; + ), + TP_printk("idx %d ring idx %d daddr %pad len %#zx skb %p vaddr %p", + __entry->frag_idx, __entry->ring_idx, &__entry->daddr, + __entry->size, __entry->skb, __entry->vaddr) +); + +#define DEFINE_MAP_EVENT(name) \ + DEFINE_EVENT(atl_dma_map_class, name, \ + TP_PROTO(int frag_idx, int ring_idx, \ + dma_addr_t daddr, size_t size, \ + struct sk_buff *skb, void *vaddr), \ + TP_ARGS(frag_idx, ring_idx, daddr, size, skb, vaddr)) + +DEFINE_MAP_EVENT(atl_dma_map_head); +DEFINE_MAP_EVENT(atl_dma_map_frag); +DEFINE_MAP_EVENT(atl_dma_map_rxbuf); + +DECLARE_EVENT_CLASS(atl_dma_unmap_class, + TP_PROTO(int frag_idx, int ring_idx, dma_addr_t daddr, size_t size, + struct sk_buff *skb), + TP_ARGS(frag_idx, ring_idx, daddr, size, skb), + TP_STRUCT__entry( + __field(int, frag_idx) + __field(int, ring_idx) + __field(dma_addr_t, daddr) + __field(size_t, size) + __field(struct sk_buff *, skb) + ), + TP_fast_assign( + __entry->frag_idx = frag_idx; + __entry->ring_idx = ring_idx; + __entry->daddr = daddr; + __entry->size = size; + __entry->skb = skb; + ), + TP_printk("idx %d ring idx %d daddr %pad len %#zx skb %p", + __entry->frag_idx, __entry->ring_idx, &__entry->daddr, + __entry->size, __entry->skb) +); + +#define DEFINE_UNMAP_EVENT(name) \ + DEFINE_EVENT(atl_dma_unmap_class, name, \ + TP_PROTO(int frag_idx, int ring_idx, dma_addr_t daddr, \ + size_t size, struct sk_buff *skb), \ + TP_ARGS(frag_idx, ring_idx, daddr, size, skb)) + +DEFINE_UNMAP_EVENT(atl_dma_unmap_head); +DEFINE_UNMAP_EVENT(atl_dma_unmap_frag); +DEFINE_UNMAP_EVENT(atl_dma_unmap_rxbuf); + +TRACE_EVENT(atl_fill_rx_desc, + TP_PROTO(int ring_idx, struct atl_rx_desc *desc), + TP_ARGS(ring_idx, desc), + TP_STRUCT__entry( + __field(int, ring_idx) + __field(dma_addr_t, daddr) + __field(dma_addr_t, haddr) + ), + TP_fast_assign( + __entry->ring_idx = ring_idx; + __entry->daddr = desc->daddr; + __entry->haddr = desc->haddr; + ), + TP_printk("[%d] daddr %pad", __entry->ring_idx, &__entry->daddr) +); + +TRACE_EVENT(atl_sync_rx_range, + TP_PROTO(int ring_idx, dma_addr_t daddr, unsigned long pg_off, + size_t size), + TP_ARGS(ring_idx, daddr, pg_off, size), + TP_STRUCT__entry( + __field(int, ring_idx) + __field(dma_addr_t, daddr) + __field(unsigned long, pg_off) + __field(size_t, size) + ), + TP_fast_assign( + __entry->ring_idx = ring_idx; + __entry->daddr = daddr; + __entry->pg_off = pg_off; + __entry->size = size; + ), + TP_printk("[%d] daddr %pad pg_off %#lx size %#zx", __entry->ring_idx, + &__entry->daddr, __entry->pg_off, __entry->size) +); + +#define DESCR_FIELD(DESCR, BIT_BEGIN, BIT_END) \ + ((DESCR >> BIT_END) &\ + (BIT_ULL(BIT_BEGIN - BIT_END + 1) - 1)) + +TRACE_EVENT(atl_rx_descr, + TP_PROTO(int ring_idx, unsigned int pointer, u64 *descr), + TP_ARGS(ring_idx, pointer, descr), + TP_STRUCT__entry( + __field(unsigned int, ring_idx) + __field(unsigned int, pointer) + __field(u8, dd) + __field(u8, eop) + __field(u8, rx_stat) + __field(u8, rx_estat) + __field(u8, rsc_cnt) + __field(u16, pkt_len) + __field(u16, next_desp) + __field(u16, vlan_tag) + + __field(u8, rss_type) + __field(u8, pkt_type) + __field(u8, rsvd) + __field(u8, rx_cntl) + __field(u8, sph) + __field(u16, hdr_len) + __field(u32, rss_hash) + + ), + TP_fast_assign( + __entry->ring_idx = ring_idx; + __entry->pointer = pointer; + __entry->rss_hash = DESCR_FIELD(descr[0], 63, 32); + __entry->hdr_len = DESCR_FIELD(descr[0], 31, 22); + __entry->sph = DESCR_FIELD(descr[0], 21, 21); + __entry->rx_cntl = DESCR_FIELD(descr[0], 20, 19); + __entry->rsvd = DESCR_FIELD(descr[0], 18, 12); + __entry->pkt_type = DESCR_FIELD(descr[0], 11, 4); + __entry->rss_type = DESCR_FIELD(descr[0], 3, 0); + + __entry->vlan_tag = DESCR_FIELD(descr[1], 63, 48); + __entry->next_desp = DESCR_FIELD(descr[1], 47, 32); + __entry->pkt_len = DESCR_FIELD(descr[1], 31, 16); + __entry->rsc_cnt = DESCR_FIELD(descr[1], 15, 12); + __entry->rx_estat = DESCR_FIELD(descr[1], 11, 6); + __entry->rx_stat = DESCR_FIELD(descr[1], 5, 2); + __entry->eop = DESCR_FIELD(descr[1], 1, 1); + __entry->dd = DESCR_FIELD(descr[1], 0, 0); + ), + TP_printk("ring=%d descr=%u rss_hash=0x%x hdr_len=%u sph=%u rx_cntl=%u rsvd=0x%x pkt_type=%u rss_type=%u vlan_tag=%u next_desp=%u pkt_len=%u rsc_cnt=%u rx_estat=0x%x rx_stat=0x%x eop=%u dd=%u", + __entry->ring_idx, __entry->pointer, __entry->rss_hash, + __entry->hdr_len, __entry->sph, __entry->rx_cntl, + __entry->rsvd, __entry->pkt_type, __entry->rss_type, + __entry->vlan_tag, __entry->next_desp, __entry->pkt_len, + __entry->rsc_cnt, __entry->rx_estat, __entry->rx_stat, + __entry->eop, __entry->dd) +); + +TRACE_EVENT(atl_tx_descr, + TP_PROTO(int ring_idx, unsigned int pointer, u64 *descr), + TP_ARGS(ring_idx, pointer, descr), + TP_STRUCT__entry( + __field(unsigned int, ring_idx) + __field(unsigned int, pointer) + /* Tx Descriptor */ + __field(u64, data_buf_addr) + __field(u32, pay_len) + __field(u8, ct_en) + __field(u8, ct_idx) + __field(u16, rsvd2) + __field(u8, tx_cmd) + __field(u8, eop) + __field(u8, dd) + __field(u16, buf_len) + __field(u8, rsvd1) + __field(u8, des_typ) + ), + TP_fast_assign( + __entry->ring_idx = ring_idx; + __entry->pointer = pointer; + __entry->data_buf_addr = descr[0]; + __entry->pay_len = DESCR_FIELD(descr[1], 63, 46); + __entry->ct_en = DESCR_FIELD(descr[1], 45, 45); + __entry->ct_idx = DESCR_FIELD(descr[1], 44, 44); + __entry->rsvd2 = DESCR_FIELD(descr[1], 43, 30); + __entry->tx_cmd = DESCR_FIELD(descr[1], 29, 22); + __entry->eop = DESCR_FIELD(descr[1], 21, 21); + __entry->dd = DESCR_FIELD(descr[1], 20, 20); + __entry->buf_len = DESCR_FIELD(descr[1], 19, 4); + __entry->rsvd1 = DESCR_FIELD(descr[1], 3, 3); + __entry->des_typ = DESCR_FIELD(descr[1], 2, 0); + + ), + TP_printk("ring=%d descr=%u pay_len=%u ct_en=%u ct_idx=%u rsvd2=0x%x tx_cmd=0x%x eop=%u dd=%u buf_len=%u rsvd1=%u des_typ=0x%x", + __entry->ring_idx, __entry->pointer, __entry->pay_len, + __entry->ct_en, __entry->ct_idx, __entry->rsvd2, + __entry->tx_cmd, __entry->eop, __entry->dd, __entry->buf_len, + __entry->rsvd1, __entry->des_typ) +); + + +TRACE_EVENT(atl_tx_context_descr, + TP_PROTO(int ring_idx, unsigned int pointer, u64 *descr), + TP_ARGS(ring_idx, pointer, descr), + TP_STRUCT__entry( + __field(unsigned int, ring_idx) + __field(unsigned int, pointer) + /* Tx Context Descriptor */ + __field(u8, out_len) + __field(u8, tun_len) + __field(u64, resvd3) + __field(u16, mss_len) + __field(u8, l4_len) + __field(u8, l3_len) + __field(u8, l2_len) + __field(u8, ct_cmd) + __field(u16, vlan_tag) + __field(u8, ct_idx) + __field(u8, des_typ) + ), + TP_fast_assign( + __entry->ring_idx = ring_idx; + __entry->pointer = pointer; + __entry->out_len = DESCR_FIELD(descr[0], 63, 56); + __entry->tun_len = DESCR_FIELD(descr[0], 55, 48); + __entry->resvd3 = DESCR_FIELD(descr[0], 47, 0); + __entry->mss_len = DESCR_FIELD(descr[1], 63, 48); + __entry->l4_len = DESCR_FIELD(descr[1], 47, 40); + __entry->l3_len = DESCR_FIELD(descr[1], 39, 31); + __entry->l2_len = DESCR_FIELD(descr[1], 30, 24); + __entry->ct_cmd = DESCR_FIELD(descr[1], 23, 20); + __entry->vlan_tag = DESCR_FIELD(descr[1], 19, 4); + __entry->ct_idx = DESCR_FIELD(descr[1], 3, 3); + __entry->des_typ = DESCR_FIELD(descr[1], 2, 0); + ), + TP_printk("ring=%d descr=%u out_len=%u tun_len=%u resvd3=%llu mss_len=%u l4_len=%u l3_len=%u l2_len=0x%x ct_cmd=%u vlan_tag=%u ct_idx=%u des_typ=0x%x", + __entry->ring_idx, __entry->pointer, __entry->out_len, + __entry->tun_len, __entry->resvd3, __entry->mss_len, + __entry->l4_len, __entry->l3_len, __entry->l2_len, + __entry->ct_cmd, __entry->vlan_tag, __entry->ct_idx, + __entry->des_typ) +); + +#endif /* _ATL_TRACE_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE atl_trace +#include diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/Makefile b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/Makefile new file mode 100644 index 000000000000..194a190b06fd --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/Makefile @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Atlantic Network Driver +# +# Copyright (C) 2019 aQuantia Corporation +# Copyright (C) 2019-2020 Marvell International Ltd. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. + +PKGCONFIG?=$(shell which pkg-config) +ifeq ($(PKGCONFIG),) +$(info Do you have pkg-config installed? If not, install it, e.g. on Ubuntu) +$(info $$ apt install pkg-config) +$(error pkg-config not found) +endif +LIBMNL_EXISTS=$(shell $(PKGCONFIG) --exists libmnl && echo "yes" || echo "no") +ifeq ($(LIBMNL_EXISTS),no) +$(info Do you have libmnl-dev installed? If not, install it, e.g. on Ubuntu) +$(info $$ apt install libmnl-dev) +$(error libmnl not found) +endif + +CFLAGS+=-Wall -Werror -Wfatal-errors $(shell $(PKGCONFIG) --cflags libmnl) -I.. +LDLIBS+=$(shell $(PKGCONFIG) --libs libmnl) + +.PHONY: all clean + +all: atlfwdtool + +atlfwdtool: atlfwdtool.c atlfwd_args.c atlfwd_msg.c atlfwd_reply.c + $(LINK.c) $^ $(LDLIBS) -o $@ + +clean: + $(RM) atlfwdtool + $(RM) *.o diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.c new file mode 100644 index 000000000000..2e62d91ac19c --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.c @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atlfwd_args.h" + +#include +#include +#include +#include +#include + +#include "libmnl/libmnl.h" + +static void print_usage(const char *binname) +{ + fprintf(stderr, + "Usage: %s [-d devname] [-v] command \n\n", + binname); + fprintf(stderr, "%s\n", "Options:"); + fprintf(stderr, "\t%s\n", + "-d devname\t- specify device name to work with"); + fprintf(stderr, "\t%s\n", "-v\t\t- verbose mode"); + fprintf(stderr, "%s\n", ""); + fprintf(stderr, "%s\n", "Supported commands:"); + fprintf(stderr, "\t%s\n", + "request_ring "); + fprintf(stderr, "\t%s\n", "release_ring "); + fprintf(stderr, "\t%s\n", "enable_ring "); + fprintf(stderr, "\t%s\n", "disable_ring "); + fprintf(stderr, "\t%s\n", "dump_ring "); + fprintf(stderr, "\t%s\n", "request_event "); + fprintf(stderr, "\t%s\n", "release_event "); + fprintf(stderr, "\t%s\n", "enable_event "); + fprintf(stderr, "\t%s\n", "disable_event "); + fprintf(stderr, "\t%s\n", "get_rx_queue_index "); + fprintf(stderr, "\t%s\n", "get_tx_queue_index "); + fprintf(stderr, "%s\n", ""); + fprintf(stderr, "\t%s\n", "force_icmp_tx_via "); + fprintf(stderr, "\t%s\n", "force_tx_via "); + fprintf(stderr, "\t%s\n", "disable_redirections"); + fprintf(stderr, "%s\n", ""); + fprintf(stderr, "\t%s\n", "ring_status []"); + fprintf(stderr, "\t%s\n", "set_tx_bunch "); +} + +#define STR_EQUAL(s1, s2) (strncmp((s1), (s2), MNL_ARRAY_SIZE((s2))) == 0) + +static enum atlfwd_nl_command get_command(const char *str) +{ + static const char cmd_req_ring[] = "request_ring"; + static const char cmd_rel_ring[] = "release_ring"; + static const char cmd_enable_ring[] = "enable_ring"; + static const char cmd_disable_ring[] = "disable_ring"; + static const char cmd_req_event[] = "request_event"; + static const char cmd_rel_event[] = "release_event"; + static const char cmd_enable_event[] = "enable_event"; + static const char cmd_disable_event[] = "disable_event"; + static const char cmd_dump_ring[] = "dump_ring"; + static const char cmd_get_rx_queue[] = "get_rx_queue_index"; + static const char cmd_get_tx_queue[] = "get_tx_queue_index"; + static const char cmd_disable_redirections[] = "disable_redirections"; + static const char cmd_force_icmp_tx_via[] = "force_icmp_tx_via"; + static const char cmd_force_tx_via[] = "force_tx_via"; + static const char cmd_ring_status[] = "ring_status"; + static const char cmd_set_tx_bunch[] = "set_tx_bunch"; + + if (STR_EQUAL(str, cmd_req_ring)) + return ATL_FWD_CMD_REQUEST_RING; + + if (STR_EQUAL(str, cmd_rel_ring)) + return ATL_FWD_CMD_RELEASE_RING; + + if (STR_EQUAL(str, cmd_enable_ring)) + return ATL_FWD_CMD_ENABLE_RING; + + if (STR_EQUAL(str, cmd_disable_ring)) + return ATL_FWD_CMD_DISABLE_RING; + + if (STR_EQUAL(str, cmd_dump_ring)) + return ATL_FWD_CMD_DUMP_RING; + + if (STR_EQUAL(str, cmd_get_rx_queue)) + return ATL_FWD_CMD_GET_RX_QUEUE; + + if (STR_EQUAL(str, cmd_get_tx_queue)) + return ATL_FWD_CMD_GET_TX_QUEUE; + + if (STR_EQUAL(str, cmd_req_event)) + return ATL_FWD_CMD_REQUEST_EVENT; + + if (STR_EQUAL(str, cmd_rel_event)) + return ATL_FWD_CMD_RELEASE_EVENT; + + if (STR_EQUAL(str, cmd_enable_event)) + return ATL_FWD_CMD_ENABLE_EVENT; + + if (STR_EQUAL(str, cmd_disable_event)) + return ATL_FWD_CMD_DISABLE_EVENT; + + if (STR_EQUAL(str, cmd_disable_redirections)) + return ATL_FWD_CMD_DISABLE_REDIRECTIONS; + + if (STR_EQUAL(str, cmd_force_icmp_tx_via)) + return ATL_FWD_CMD_FORCE_ICMP_TX_VIA; + + if (STR_EQUAL(str, cmd_force_tx_via)) + return ATL_FWD_CMD_FORCE_TX_VIA; + + if (STR_EQUAL(str, cmd_ring_status)) + return ATL_FWD_CMD_RING_STATUS; + + if (STR_EQUAL(str, cmd_set_tx_bunch)) + return ATL_FWD_CMD_SET_TX_BUNCH; + + return ATL_FWD_CMD_UNSPEC; +} + +static const char *get_arg(const int argc, char **argv) +{ + if (argc == optind) { + /* not enough arguments => exit immediately */ + print_usage(argv[0]); + exit(EINVAL); + } + + return argv[optind++]; +} + +static const char *get_last_arg(const int argc, char **argv) +{ + const char *result = get_arg(argc, argv); + + if (optind != argc) { + /* too many arguments => exit immediately */ + print_usage(argv[0]); + exit(EINVAL); + } + + return result; +} + +struct atlfwd_args *parse_args(const int argc, char **argv) +{ + static struct atlfwd_args parsed_args; + int opt; + + while ((opt = getopt(argc, argv, "d:v")) != -1) { + switch (opt) { + case 'd': + parsed_args.devname = optarg; + break; + case 'v': + parsed_args.verbose = true; + break; + default: + print_usage(argv[0]); + return NULL; + } + } + + parsed_args.cmd = get_command(get_arg(argc, argv)); + + switch (parsed_args.cmd) { + case ATL_FWD_CMD_REQUEST_RING: + parsed_args.flags = (uint32_t)atoi(get_arg(argc, argv)); + parsed_args.ring_size = (uint32_t)atoi(get_arg(argc, argv)); + parsed_args.buf_size = (uint32_t)atoi(get_arg(argc, argv)); + parsed_args.page_order = + (uint32_t)atoi(get_last_arg(argc, argv)); + break; + case ATL_FWD_CMD_RING_STATUS: + if (optind == argc) { + parsed_args.ring_index = -1; + break; + } + /* fall through */ + case ATL_FWD_CMD_RELEASE_RING: + /* fall through */ + case ATL_FWD_CMD_ENABLE_RING: + /* fall through */ + case ATL_FWD_CMD_DISABLE_RING: + /* fall through */ + case ATL_FWD_CMD_DUMP_RING: + /* fall through */ + case ATL_FWD_CMD_REQUEST_EVENT: + /* fall through */ + case ATL_FWD_CMD_RELEASE_EVENT: + /* fall through */ + case ATL_FWD_CMD_ENABLE_EVENT: + /* fall through */ + case ATL_FWD_CMD_DISABLE_EVENT: + /* fall through */ + case ATL_FWD_CMD_GET_RX_QUEUE: + /* fall through */ + case ATL_FWD_CMD_GET_TX_QUEUE: + /* fall through */ + case ATL_FWD_CMD_FORCE_ICMP_TX_VIA: + /* fall through */ + case ATL_FWD_CMD_FORCE_TX_VIA: + parsed_args.ring_index = + (int32_t)atoi(get_last_arg(argc, argv)); + break; + case ATL_FWD_CMD_SET_TX_BUNCH: + parsed_args.tx_bunch = (uint32_t)atoi(get_last_arg(argc, argv)); + break; + case ATL_FWD_CMD_DISABLE_REDIRECTIONS: + break; + default: + print_usage(argv[0]); + return NULL; + } + + return &parsed_args; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.h new file mode 100644 index 000000000000..a49030d0f810 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_args.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATLFWD_ARGS_H_ +#define _ATLFWD_ARGS_H_ + +#include +#include + +#include "atl_fwdnl.h" + +struct atlfwd_args { + char *devname; + bool verbose; + enum atlfwd_nl_command cmd; + union { + struct { + uint32_t flags; + uint32_t ring_size; + uint32_t buf_size; + uint32_t page_order; + }; + int32_t ring_index; + uint32_t tx_bunch; + }; +}; + +struct atlfwd_args *parse_args(const int argc, char **argv); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_ctx.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_ctx.h new file mode 100644 index 000000000000..d215040e9e61 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_ctx.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATLFWD_CTX_H_ +#define _ATLFWD_CTX_H_ + +#include "libmnl/libmnl.h" +#include "linux/genetlink.h" + +#include "atl_fwdnl.h" + +struct nl_context { + struct mnl_socket *sock; + unsigned int port; + unsigned int seq; + char *msgbuf; + unsigned int msgbufsize; + struct nlmsghdr *nlhdr; + struct genlmsghdr *gnlhdr; + int family_id; + char *devname; + int verbose; +}; + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.c new file mode 100644 index 000000000000..37e6507828a7 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atlfwd_msg.h" + +#include +#include +#include + +int atlnl_msg_alloc(struct nl_context *ctx, const int msg_type, const int cmd, + const unsigned int flags, const int version) +{ + ctx->msgbuf = malloc(MNL_SOCKET_BUFFER_SIZE); + if (!ctx->msgbuf) + return -ENOMEM; + ctx->msgbufsize = MNL_SOCKET_BUFFER_SIZE; + memset(ctx->msgbuf, 0, MNL_SOCKET_BUFFER_SIZE); + + struct nlmsghdr *nlhdr = mnl_nlmsg_put_header(ctx->msgbuf); + + nlhdr->nlmsg_type = msg_type; + nlhdr->nlmsg_flags = flags; + nlhdr->nlmsg_seq = ++ctx->seq; + ctx->nlhdr = nlhdr; + + struct genlmsghdr *gnlhdr = + mnl_nlmsg_put_extra_header(nlhdr, sizeof(*gnlhdr)); + + gnlhdr->cmd = cmd; + gnlhdr->version = version; + ctx->gnlhdr = gnlhdr; + + return 0; +} + +int atlnl_msg_realloc(struct nl_context *ctx, const unsigned int new_size) +{ + const unsigned int old_size = ctx->msgbufsize; + unsigned int nlhdr_offset = (char *)ctx->nlhdr - ctx->msgbuf; + unsigned int gnlhdr_offset = (char *)ctx->gnlhdr - ctx->msgbuf; + char *new_buff = realloc(ctx->msgbuf, new_size); + + if (!new_buff) + return -ENOMEM; + + memset(new_buff + old_size, 0, new_size - old_size); + ctx->msgbuf = new_buff; + ctx->msgbufsize = new_size; + ctx->nlhdr = (struct nlmsghdr *)(new_buff + nlhdr_offset); + ctx->gnlhdr = (struct genlmsghdr *)(new_buff + gnlhdr_offset); + + return 0; +} + +void atlnl_msg_free(struct nl_context *ctx) +{ + if (ctx->msgbuf) + free(ctx->msgbuf); + + ctx->msgbuf = NULL; + ctx->msgbufsize = 0; + ctx->nlhdr = NULL; + ctx->gnlhdr = NULL; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.h new file mode 100644 index 000000000000..af3bbcbe55c1 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_msg.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATLFWD_MSG_H_ +#define _ATLFWD_MSG_H_ + +#include "atlfwd_ctx.h" + +int atlnl_msg_alloc(struct nl_context *ctx, const int msg_type, const int cmd, + const unsigned int flags, const int version); +int atlnl_msg_realloc(struct nl_context *ctx, const unsigned int new_size); +void atlnl_msg_free(struct nl_context *ctx); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.c new file mode 100644 index 000000000000..7c3606e41133 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.c @@ -0,0 +1,82 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "atlfwd_reply.h" + +#include + +#include "atlfwd_msg.h" + +int atlnl_process_ack(struct nl_context *ctx, const ssize_t len) +{ + struct nlmsgerr *nlerr = mnl_nlmsg_get_payload(ctx->nlhdr); + unsigned int tlv_offset = sizeof(*nlerr); + + if (len < NLMSG_HDRLEN + tlv_offset) + return -EFAULT; + +#ifdef NLM_F_ACK_TLVS + if (ctx->nlhdr->nlmsg_flags & NLM_F_ACK_TLVS) { + if (!(ctx->nlhdr->nlmsg_flags & NLM_F_CAPPED)) + tlv_offset += MNL_ALIGN( + mnl_nlmsg_get_payload_len(&nlerr->msg)); + + const struct nlattr *attr; + + mnl_attr_for_each(attr, ctx->nlhdr, tlv_offset) + { + if (mnl_attr_get_type(attr) == NLMSGERR_ATTR_MSG && + (ctx->verbose || nlerr->error)) { + const char *msg = mnl_attr_get_str(attr); + + fprintf(stderr, "netlink %s: %s\n", + nlerr->error ? "error" : "warning", + msg); + } + } + } +#endif + + if (nlerr->error) { + errno = -nlerr->error; + perror("netlink error"); + } else if (ctx->verbose) { + printf("%s\n", "got ack"); + } + + return nlerr->error; +} + +int atlnl_process_reply(struct nl_context *ctx, mnl_cb_t reply_cb) +{ + int ret = 0; + + atlnl_msg_realloc(ctx, 4096); + do { + ssize_t len = mnl_socket_recvfrom(ctx->sock, ctx->msgbuf, + ctx->msgbufsize); + + if (len == 0) + return 0; + if (len < NLMSG_HDRLEN) + return -EFAULT; + + ctx->nlhdr = (struct nlmsghdr *)ctx->msgbuf; + if (ctx->nlhdr->nlmsg_type == NLMSG_ERROR) + return atlnl_process_ack(ctx, len); + + ctx->gnlhdr = mnl_nlmsg_get_payload(ctx->nlhdr); + ret = mnl_cb_run(ctx->msgbuf, len, ctx->seq, ctx->port, + reply_cb, ctx); + } while (ret > 0); + + return ret; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.h b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.h new file mode 100644 index 000000000000..5d3f9cf88de2 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwd_reply.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _ATLFWD_REPLY_H_ +#define _ATLFWD_REPLY_H_ + +#include "atlfwd_ctx.h" + +int atlnl_process_ack(struct nl_context *ctx, const ssize_t len); +int atlnl_process_reply(struct nl_context *ctx, mnl_cb_t reply_cb); + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwdtool.c b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwdtool.c new file mode 100644 index 000000000000..1226e8470559 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/atlfwdtool/atlfwdtool.c @@ -0,0 +1,418 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include +#include + +#include "atlfwd_args.h" +#include "atlfwd_ctx.h" +#include "atlfwd_msg.h" +#include "atlfwd_reply.h" + +#define ATL_FWD_CMD_STR(cmd)\ +[cmd] = #cmd +static const char *cmd_str[NUM_ATL_FWD_CMD] = { + ATL_FWD_CMD_STR(ATL_FWD_CMD_REQUEST_RING), + ATL_FWD_CMD_STR(ATL_FWD_CMD_RELEASE_RING), + ATL_FWD_CMD_STR(ATL_FWD_CMD_ENABLE_RING), + ATL_FWD_CMD_STR(ATL_FWD_CMD_DISABLE_RING), + ATL_FWD_CMD_STR(ATL_FWD_CMD_DISABLE_REDIRECTIONS), + ATL_FWD_CMD_STR(ATL_FWD_CMD_FORCE_ICMP_TX_VIA), + ATL_FWD_CMD_STR(ATL_FWD_CMD_FORCE_TX_VIA), + ATL_FWD_CMD_STR(ATL_FWD_CMD_RING_STATUS), + ATL_FWD_CMD_STR(ATL_FWD_CMD_DUMP_RING), + ATL_FWD_CMD_STR(ATL_FWD_CMD_SET_TX_BUNCH), + ATL_FWD_CMD_STR(ATL_FWD_CMD_REQUEST_EVENT), + ATL_FWD_CMD_STR(ATL_FWD_CMD_RELEASE_EVENT), + ATL_FWD_CMD_STR(ATL_FWD_CMD_ENABLE_EVENT), + ATL_FWD_CMD_STR(ATL_FWD_CMD_DISABLE_EVENT), + ATL_FWD_CMD_STR(ATL_FWD_CMD_GET_RX_QUEUE), + ATL_FWD_CMD_STR(ATL_FWD_CMD_GET_TX_QUEUE), +}; +#define ATL_FWD_ATTR_STR(attr)\ +[attr] = #attr +static const char *attr_str[NUM_ATL_FWD_ATTR] = { + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_FLAGS), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_RING_SIZE), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_BUF_SIZE), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_PAGE_ORDER), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_RING_INDEX), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_RING_STATUS), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_RING_IS_TX), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_RING_FLAGS), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_TX_BUNCH_SIZE), + ATL_FWD_ATTR_STR(ATL_FWD_ATTR_QUEUE_INDEX), +}; + +/* get atl_fwd family id */ +static int atlnl_family_cb(const struct nlmsghdr *nlhdr, void *data) +{ + struct nl_context *ctx = (struct nl_context *)data; + const struct nlattr *attr; + + ctx->family_id = 0; + mnl_attr_for_each(attr, nlhdr, GENL_HDRLEN) + { + if (mnl_attr_get_type(attr) == CTRL_ATTR_FAMILY_ID) { + ctx->family_id = mnl_attr_get_u16(attr); + break; + } + } + + return (ctx->family_id ? MNL_CB_OK : MNL_CB_ERROR); +} + +static int atlnl_get_family_id(struct nl_context *ctx) +{ + int ret = atlnl_msg_alloc(ctx, GENL_ID_CTRL, CTRL_CMD_GETFAMILY, + NLM_F_REQUEST | NLM_F_ACK, 1); + if (ctx->verbose) + printf("calling %s\n", "CTRL_CMD_GETFAMILY"); + if (ret != 0) + goto err_msgfree; + + mnl_attr_put_strz(ctx->nlhdr, CTRL_ATTR_FAMILY_NAME, ATL_FWD_GENL_NAME); + + if (mnl_socket_sendto(ctx->sock, ctx->nlhdr, ctx->nlhdr->nlmsg_len) < 0) + goto err_msgfree; + + ret = atlnl_process_reply(ctx, atlnl_family_cb); + if (ret < 0) + goto err_msgfree; + + ret = ctx->family_id ? 0 : -EADDRNOTAVAIL; + + if (ctx->verbose) + printf("family id: %d\n", ctx->family_id); + +err_msgfree: + atlnl_msg_free(ctx); + return ret; +} + +static int atlnl_reqring_cb(const struct nlmsghdr *nlhdr, void *data) +{ + const struct nlattr *attr; + int ring_index = -1; + + mnl_attr_for_each(attr, nlhdr, GENL_HDRLEN) + { + if (mnl_attr_get_type(attr) == ATL_FWD_ATTR_RING_INDEX) { + ring_index = (int)mnl_attr_get_u32(attr); + break; + } + } + + if (ring_index == -1) { + fprintf(stderr, "Error: %s attribute is missing in reply\n", + attr_str[ATL_FWD_ATTR_RING_INDEX]); + return MNL_CB_ERROR; + } + + printf("Ring index: %d\n", ring_index); + return MNL_CB_OK; +} + +static int atlnl_getqueue_cb(const struct nlmsghdr *nlhdr, void *data) +{ + const struct nlattr *attr; + int queue = -1; + + mnl_attr_for_each(attr, nlhdr, GENL_HDRLEN) + { + if (mnl_attr_get_type(attr) == ATL_FWD_ATTR_QUEUE_INDEX) { + queue = (int)mnl_attr_get_u32(attr); + break; + } + } + + if (queue == -1) { + fprintf(stderr, "Error: %s attribute is missing in reply\n", + attr_str[ATL_FWD_ATTR_QUEUE_INDEX]); + return MNL_CB_ERROR; + } + + printf("%d\n", queue); + return MNL_CB_OK; +} + +static bool is_ring_created(const enum atlfwd_nl_ring_status ring_status) +{ + switch (ring_status) { + case ATL_FWD_RING_STATUS_CREATED_DISABLED: + case ATL_FWD_RING_STATUS_ENABLED: + return true; + default: + return false; + } + + return false; +} + +static const char * +ring_status_string(const enum atlfwd_nl_ring_status ring_status) +{ + static const char *status_str[NUM_ATL_FWD_RING_STATUS] = { + [ATL_FWD_RING_STATUS_INVALID] = "invalid status", + [ATL_FWD_RING_STATUS_RELEASED] = "released", + [ATL_FWD_RING_STATUS_CREATED_DISABLED] = "disabled", + [ATL_FWD_RING_STATUS_ENABLED] = "enabled", + }; + + return (ring_status < NUM_ATL_FWD_RING_STATUS ? + status_str[ring_status] : + status_str[ATL_FWD_RING_STATUS_INVALID]); +} + +struct ring_status { + enum atlfwd_nl_ring_status status; + bool is_tx; + int size; + unsigned int flags; +}; + +static void print_ring_status(const int ring_index, + const struct ring_status *status) +{ + char prefix[16]; + + snprintf(prefix, MNL_ARRAY_SIZE(prefix), "fwd_ring_%d_", ring_index); + + printf("%sstatus:\t%s\n", prefix, ring_status_string(status->status)); + printf("%sdirection:\t%s\n", prefix, status->is_tx ? "tx" : "rx"); + if (is_ring_created(status->status)) { + printf("%sflags:\t%d\n", prefix, status->flags); + printf("%ssize:\t%d\n", prefix, status->size); + } else { + printf("%sflags:\t%s\n", prefix, "n/a"); + printf("%ssize:\t%s\n", prefix, "n/a"); + } +} + +static int atlnl_ringstatus_cb(const struct nlmsghdr *nlhdr, void *data) +{ + const struct nlattr *attr; + int ring_index = -1; + struct ring_status status = { + .status = ATL_FWD_RING_STATUS_INVALID, + .is_tx = false, + .size = 0, + .flags = 0, + }; + + mnl_attr_for_each(attr, nlhdr, GENL_HDRLEN) + { + switch (mnl_attr_get_type(attr)) { + case ATL_FWD_ATTR_RING_INDEX: + if (ring_index >= 0) + print_ring_status(ring_index, &status); + ring_index = (int)mnl_attr_get_u32(attr); + break; + case ATL_FWD_ATTR_RING_STATUS: + status.status = (int)mnl_attr_get_u32(attr); + break; + case ATL_FWD_ATTR_RING_IS_TX: + status.is_tx = (bool)mnl_attr_get_u32(attr); + break; + case ATL_FWD_ATTR_RING_SIZE: + status.size = (int)mnl_attr_get_u32(attr); + break; + case ATL_FWD_ATTR_RING_FLAGS: + status.flags = (unsigned int)mnl_attr_get_u32(attr); + break; + default: + fprintf(stderr, "Error: %s\n", + "Unexpected attribute in reply"); + break; + } + } + + if (ring_index == -1) { + fprintf(stderr, "Error: %s\n", "Incorrect reply"); + return MNL_CB_ERROR; + } + + print_ring_status(ring_index, &status); + + return MNL_CB_OK; +} + +static int atlnl_cmd_generic_u32_args(struct nl_context *ctx, + const enum atlfwd_nl_command cmd, + mnl_cb_t reply_cb, const int attr_count, + ...) +{ + int ret = atlnl_msg_alloc(ctx, ctx->family_id, cmd, + NLM_F_REQUEST | NLM_F_ACK, 1); + va_list args; + int i; + + if (ret != 0) { + fprintf(stderr, "Error: %s\n", "message allocation failed."); + goto err_msgfree; + } + + if (ctx->verbose) + printf("calling %s\n", + cmd_str[cmd] ? cmd_str[cmd] : "unknown command"); + + if (ctx->devname != NULL) + mnl_attr_put_strz(ctx->nlhdr, ATL_FWD_ATTR_IFNAME, + ctx->devname); + + va_start(args, attr_count); + for (i = 0; i != attr_count; i++) { + const enum atlfwd_nl_attribute attr = + va_arg(args, enum atlfwd_nl_attribute); + const uint32_t value = va_arg(args, uint32_t); + + mnl_attr_put_u32(ctx->nlhdr, attr, value); + + if (ctx->verbose) + printf("\t(%s: %u)\n", + attr_str[attr] ? attr_str[attr] : + "unknown attribute", + value); + } + va_end(args); + + if (mnl_socket_sendto(ctx->sock, ctx->nlhdr, ctx->nlhdr->nlmsg_len) < 0) + goto err_msgfree; + + ret = atlnl_process_reply(ctx, reply_cb); + +err_msgfree: + atlnl_msg_free(ctx); + return ret; +} + +int main(int argc, char **argv) +{ + struct atlfwd_args *args = parse_args(argc, argv); + static struct nl_context nlctx; + int result = EINVAL; + int ret = 0; + + if (!args) + return result; + + nlctx.devname = args->devname; + nlctx.verbose = args->verbose; + + result = ECONNREFUSED; + + /* open socket */ + nlctx.sock = mnl_socket_open(NETLINK_GENERIC); + if (!nlctx.sock) + return result; + +#ifdef NETLINK_EXT_ACK + /* request extended acknowledgment */ + unsigned int true_val = 1; + ret = mnl_socket_setsockopt(nlctx.sock, NETLINK_EXT_ACK, &true_val, + sizeof(true_val)); + if (ret < 0) + goto err_sockclose; +#endif + + /* bind and get the port id */ + ret = mnl_socket_bind(nlctx.sock, 0, MNL_SOCKET_AUTOPID); + if (ret < 0) { + result = -ret; + goto err_sockclose; + } + nlctx.port = mnl_socket_get_portid(nlctx.sock); + + /* parse command line options */ + result = EINVAL; + + /* obtain family id by name */ + ret = atlnl_get_family_id(&nlctx); + if (ret < 0) + goto err_sockclose; + + /* process command(s) */ + switch (args->cmd) { + case ATL_FWD_CMD_REQUEST_RING: + ret = atlnl_cmd_generic_u32_args( + &nlctx, args->cmd, atlnl_reqring_cb, 4, + ATL_FWD_ATTR_FLAGS, args->flags, ATL_FWD_ATTR_RING_SIZE, + args->ring_size, ATL_FWD_ATTR_BUF_SIZE, args->buf_size, + ATL_FWD_ATTR_PAGE_ORDER, args->page_order); + break; + case ATL_FWD_CMD_RELEASE_RING: + /* fall through */ + case ATL_FWD_CMD_ENABLE_RING: + /* fall through */ + case ATL_FWD_CMD_DISABLE_RING: + /* fall through */ + case ATL_FWD_CMD_DUMP_RING: + /* fall through */ + case ATL_FWD_CMD_REQUEST_EVENT: + /* fall through */ + case ATL_FWD_CMD_RELEASE_EVENT: + /* fall through */ + case ATL_FWD_CMD_ENABLE_EVENT: + /* fall through */ + case ATL_FWD_CMD_DISABLE_EVENT: + /* fall through */ + case ATL_FWD_CMD_FORCE_ICMP_TX_VIA: + /* fall through */ + case ATL_FWD_CMD_FORCE_TX_VIA: + ret = atlnl_cmd_generic_u32_args(&nlctx, args->cmd, NULL, 1, + ATL_FWD_ATTR_RING_INDEX, + (uint32_t)args->ring_index); + break; + case ATL_FWD_CMD_GET_RX_QUEUE: + /* fall through */ + case ATL_FWD_CMD_GET_TX_QUEUE: + ret = atlnl_cmd_generic_u32_args(&nlctx, args->cmd, + atlnl_getqueue_cb, 1, + ATL_FWD_ATTR_RING_INDEX, + (uint32_t)args->ring_index); + break; + case ATL_FWD_CMD_RING_STATUS: + if (args->ring_index < 0) + ret = atlnl_cmd_generic_u32_args( + &nlctx, args->cmd, atlnl_ringstatus_cb, 0); + else + ret = atlnl_cmd_generic_u32_args( + &nlctx, args->cmd, atlnl_ringstatus_cb, 1, + ATL_FWD_ATTR_RING_INDEX, + (uint32_t)args->ring_index); + break; + case ATL_FWD_CMD_DISABLE_REDIRECTIONS: + ret = atlnl_cmd_generic_u32_args(&nlctx, args->cmd, NULL, 0); + break; + case ATL_FWD_CMD_SET_TX_BUNCH: + ret = atlnl_cmd_generic_u32_args(&nlctx, args->cmd, NULL, 1, + ATL_FWD_ATTR_TX_BUNCH_SIZE, + args->tx_bunch); + break; + default: + fprintf(stderr, "Unknown command %d\n", args->cmd); + goto err_sockclose; + } + + result = 0; + if (ret < 0) { + result = -ret; + perror("cmd error"); + } + +err_sockclose: + mnl_socket_close(nlctx.sock); + return result; +} diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Egress_registers.h b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Egress_registers.h new file mode 100644 index 000000000000..1aa4ce4abf9d --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Egress_registers.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef MSS_EGRESS_REGS_HEADER +#define MSS_EGRESS_REGS_HEADER + +#define MSS_EGRESS_CTL_REGISTER_ADDR 0x00005002 +#define MSS_EGRESS_SA_EXPIRED_STATUS_REGISTER_ADDR 0x00005060 +#define MSS_EGRESS_SA_THRESHOLD_EXPIRED_STATUS_REGISTER_ADDR 0x00005062 +#define MSS_EGRESS_LUT_ADDR_CTL_REGISTER_ADDR 0x00005080 +#define MSS_EGRESS_LUT_CTL_REGISTER_ADDR 0x00005081 +#define MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR 0x000050A0 + +struct mss_egress_ctl_register { + union { + struct { + unsigned int soft_reset : 1; + unsigned int drop_kay_packet : 1; + unsigned int drop_egprc_lut_miss : 1; + unsigned int gcm_start : 1; + unsigned int gcm_test_mode : 1; + unsigned int unmatched_use_sc_0 : 1; + unsigned int drop_invalid_sa_sc_packets : 1; + unsigned int reserved0 : 1; + /* Should always be set to 0. */ + unsigned int external_classification_enable : 1; + unsigned int icv_lsb_8bytes_enable : 1; + unsigned int high_prio : 1; + unsigned int clear_counter : 1; + unsigned int clear_global_time : 1; + unsigned int ethertype_explicit_sectag_lsb : 3; + } bits_0; + unsigned short word_0; + }; + union { + struct { + unsigned int ethertype_explicit_sectag_msb : 13; + unsigned int reserved0 : 3; + } bits_1; + unsigned short word_1; + }; +}; + +struct mss_egress_lut_addr_ctl_register { + union { + struct { + unsigned int lut_addr : 9; + unsigned int reserved0 : 3; + /* 0x0 : Egress MAC Control FIlter (CTLF) LUT + * 0x1 : Egress Classification LUT + * 0x2 : Egress SC/SA LUT + * 0x3 : Egress SMIB + */ + unsigned int lut_select : 4; + } bits_0; + unsigned short word_0; + }; +}; + +struct mss_egress_lut_ctl_register { + union { + struct { + unsigned int reserved0 : 14; + unsigned int lut_read : 1; + unsigned int lut_write : 1; + } bits_0; + unsigned short word_0; + }; +}; + +#endif /* MSS_EGRESS_REGS_HEADER */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Ingress_registers.h b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Ingress_registers.h new file mode 100644 index 000000000000..8422d4595dd2 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/MSS_Ingress_registers.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef MSS_INGRESS_REGS_HEADER +#define MSS_INGRESS_REGS_HEADER + +#define SEC_INGRESS_PACKET_EDIT_CTL_REGISTER_ADDR 0x00007035 +#define MSS_INGRESS_CTL_REGISTER_ADDR 0x0000800E +#define MSS_INGRESS_LUT_ADDR_CTL_REGISTER_ADDR 0x00008080 +#define MSS_INGRESS_LUT_CTL_REGISTER_ADDR 0x00008081 +#define MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR 0x000080A0 + +struct mss_ingress_ctl_register { + union { + struct { + unsigned int soft_reset : 1; + unsigned int operation_point_to_point : 1; + unsigned int create_sci : 1; + /* Unused */ + unsigned int mask_short_length_error : 1; + unsigned int drop_kay_packet : 1; + unsigned int drop_igprc_miss : 1; + /* Unused */ + unsigned int check_icv : 1; + unsigned int clear_global_time : 1; + unsigned int clear_count : 1; + unsigned int high_prio : 1; + unsigned int remove_sectag : 1; + unsigned int global_validate_frames : 2; + unsigned int icv_lsb_8bytes_enabled : 1; + unsigned int reserved0 : 2; + } bits_0; + unsigned short word_0; + }; + union { + struct { + unsigned int reserved0 : 16; + } bits_1; + unsigned short word_1; + }; +}; + +struct mss_ingress_lut_addr_ctl_register { + union { + struct { + unsigned int lut_addr : 9; + unsigned int reserved0 : 3; + /* 0x0 : Ingress Pre-Security MAC Control FIlter + * (IGPRCTLF) LUT + * 0x1 : Ingress Pre-Security Classification LUT (IGPRC) + * 0x2 : Ingress Packet Format (IGPFMT) SAKey LUT + * 0x3 : Ingress Packet Format (IGPFMT) SC/SA LUT + * 0x4 : Ingress Post-Security Classification LUT + * (IGPOC) + * 0x5 : Ingress Post-Security MAC Control Filter + * (IGPOCTLF) LUT + * 0x6 : Ingress MIB (IGMIB) + */ + unsigned int lut_select : 4; + } bits_0; + unsigned short word_0; + }; +}; + +struct mss_ingress_lut_ctl_register { + union { + struct { + unsigned int reserved0 : 14; + unsigned int lut_read : 1; + unsigned int lut_write : 1; + } bits_0; + unsigned short word_0; + }; +}; + +#endif /* MSS_INGRESS_REGS_HEADER */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.c b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.c new file mode 100644 index 000000000000..364ca11ec031 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.c @@ -0,0 +1,2976 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include "macsec_api.h" +#include +#include "MSS_Ingress_registers.h" +#include "MSS_Egress_registers.h" +#include "atl_mdio.h" + +#define AQ_API_CALL_SAFE(func, ...) \ +({ \ + int ret; \ + do { \ + ret = atl_mdio_hwsem_get(hw); \ + if (unlikely(ret)) \ + break; \ + \ + ret = func(__VA_ARGS__); \ + \ + atl_mdio_hwsem_put(hw); \ + } while (0); \ + ret; \ +}) + +/******************************************************************************* + * MACSEC config and status + ******************************************************************************/ + +static int set_raw_ingress_record(struct atl_hw *hw, u16 *packed_record, + u8 num_words, u8 table_id, + u16 table_index) +{ + struct mss_ingress_lut_addr_ctl_register lut_sel_reg; + struct mss_ingress_lut_ctl_register lut_op_reg; + + unsigned int i; + + /* NOTE: MSS registers must always be read/written as adjacent pairs. + * For instance, to write either or both 1E.80A0 and 80A1, we have to: + * 1. Write 1E.80A0 first + * 2. Then write 1E.80A1 + * + * For HHD devices: These writes need to be performed consecutively, and + * to ensure this we use the PIF mailbox to delegate the reads/writes to + * the FW. + * + * For EUR devices: Not need to use the PIF mailbox; it is safe to + * write to the registers directly. + */ + + /* Write the packed record words to the data buffer registers. */ + for (i = 0; i < num_words; i += 2) { + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR + i, + packed_record[i]); + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR + i + 1, + packed_record[i + 1]); + } + + /* Clear out the unused data buffer registers. */ + for (i = num_words; i < 24; i += 2) { + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR + i, 0); + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR + i + 1, + 0); + } + + /* Select the table and row index to write to */ + lut_sel_reg.bits_0.lut_select = table_id; + lut_sel_reg.bits_0.lut_addr = table_index; + + lut_op_reg.bits_0.lut_read = 0; + lut_op_reg.bits_0.lut_write = 1; + + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_ADDR_CTL_REGISTER_ADDR, + lut_sel_reg.word_0); + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_CTL_REGISTER_ADDR, + lut_op_reg.word_0); + + return 0; +} + +/*! Read the specified Ingress LUT table row. + * packed_record - [OUT] The table row data (raw). + */ +static int get_raw_ingress_record(struct atl_hw *hw, u16 *packed_record, + u8 num_words, u8 table_id, + u16 table_index) +{ + struct mss_ingress_lut_addr_ctl_register lut_sel_reg; + struct mss_ingress_lut_ctl_register lut_op_reg; + int ret; + + unsigned int i; + + /* Select the table and row index to read */ + lut_sel_reg.bits_0.lut_select = table_id; + lut_sel_reg.bits_0.lut_addr = table_index; + + lut_op_reg.bits_0.lut_read = 1; + lut_op_reg.bits_0.lut_write = 0; + + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_ADDR_CTL_REGISTER_ADDR, + lut_sel_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_CTL_REGISTER_ADDR, + lut_op_reg.word_0); + if (unlikely(ret)) + return ret; + + memset(packed_record, 0, sizeof(u16) * num_words); + + for (i = 0; i < num_words; i += 2) { + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR + + i, + &packed_record[i]); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_LUT_DATA_CTL_REGISTER_ADDR + + i + 1, + &packed_record[i + 1]); + if (unlikely(ret)) + return ret; + } + + return 0; +} + +/*! Write packed_record to the specified Egress LUT table row. */ +static int set_raw_egress_record(struct atl_hw *hw, u16 *packed_record, + u8 num_words, u8 table_id, + u16 table_index) +{ + struct mss_egress_lut_addr_ctl_register lut_sel_reg; + struct mss_egress_lut_ctl_register lut_op_reg; + + unsigned int i; + + /* Write the packed record words to the data buffer registers. */ + for (i = 0; i < num_words; i += 2) { + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR + i, + packed_record[i]); + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR + i + 1, + packed_record[i + 1]); + } + + /* Clear out the unused data buffer registers. */ + for (i = num_words; i < 28; i += 2) { + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR + i, 0); + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR + i + 1, + 0); + } + + /* Select the table and row index to write to */ + lut_sel_reg.bits_0.lut_select = table_id; + lut_sel_reg.bits_0.lut_addr = table_index; + + lut_op_reg.bits_0.lut_read = 0; + lut_op_reg.bits_0.lut_write = 1; + + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_ADDR_CTL_REGISTER_ADDR, + lut_sel_reg.word_0); + __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_CTL_REGISTER_ADDR, + lut_op_reg.word_0); + + return 0; +} + +static int get_raw_egress_record(struct atl_hw *hw, u16 *packed_record, + u8 num_words, u8 table_id, + u16 table_index) +{ + struct mss_egress_lut_addr_ctl_register lut_sel_reg; + struct mss_egress_lut_ctl_register lut_op_reg; + int ret; + + unsigned int i; + + /* Select the table and row index to read */ + lut_sel_reg.bits_0.lut_select = table_id; + lut_sel_reg.bits_0.lut_addr = table_index; + + lut_op_reg.bits_0.lut_read = 1; + lut_op_reg.bits_0.lut_write = 0; + + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_ADDR_CTL_REGISTER_ADDR, + lut_sel_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_CTL_REGISTER_ADDR, + lut_op_reg.word_0); + if (unlikely(ret)) + return ret; + + memset(packed_record, 0, sizeof(u16) * num_words); + + for (i = 0; i < num_words; i += 2) { + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR + i, + &packed_record[i]); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_LUT_DATA_CTL_REGISTER_ADDR + + i + 1, + &packed_record[i + 1]); + if (unlikely(ret)) + return ret; + } + + return 0; +} + +static int +set_ingress_prectlf_record(struct atl_hw *hw, + const struct aq_mss_ingress_prectlf_record *rec, + u16 table_index) +{ + u16 packed_record[6]; + + if (table_index >= NUMROWS_INGRESSPRECTLFRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 6); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->sa_da[0] >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->sa_da[0] >> 16) & 0xFFFF) << 0); + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->sa_da[1] >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->eth_type >> 0) & 0xFFFF) << 0); + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->match_mask >> 0) & 0xFFFF) << 0); + packed_record[5] = (packed_record[5] & 0xFFF0) | + (((rec->match_type >> 0) & 0xF) << 0); + packed_record[5] = + (packed_record[5] & 0xFFEF) | (((rec->action >> 0) & 0x1) << 4); + + return set_raw_ingress_record(hw, packed_record, 6, 0, + ROWOFFSET_INGRESSPRECTLFRECORD + + table_index); +} + +int aq_mss_set_ingress_prectlf_record( + struct atl_hw *hw, const struct aq_mss_ingress_prectlf_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_prectlf_record, hw, rec, + table_index); +} + +static int get_ingress_prectlf_record(struct atl_hw *hw, + struct aq_mss_ingress_prectlf_record *rec, + u16 table_index) +{ + u16 packed_record[6]; + int ret; + + if (table_index >= NUMROWS_INGRESSPRECTLFRECORD) + return -EINVAL; + + /* If the row that we want to read is odd, first read the previous even + * row, throw that value away, and finally read the desired row. + * This is a workaround for EUR devices that allows us to read + * odd-numbered rows. For HHD devices: this workaround will not work, + * so don't bother; odd-numbered rows are not readable. + */ + if ((table_index % 2) > 0) { + ret = get_raw_ingress_record(hw, packed_record, 6, 0, + ROWOFFSET_INGRESSPRECTLFRECORD + + table_index - 1); + if (unlikely(ret)) + return ret; + } + + ret = get_raw_ingress_record(hw, packed_record, 6, 0, + ROWOFFSET_INGRESSPRECTLFRECORD + + table_index); + if (unlikely(ret)) + return ret; + + rec->sa_da[0] = (rec->sa_da[0] & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->sa_da[0] = (rec->sa_da[0] & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->sa_da[1] = (rec->sa_da[1] & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + + rec->eth_type = (rec->eth_type & 0xFFFF0000) | + (((packed_record[3] >> 0) & 0xFFFF) << 0); + + rec->match_mask = (rec->match_mask & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + + rec->match_type = (rec->match_type & 0xFFFFFFF0) | + (((packed_record[5] >> 0) & 0xF) << 0); + + rec->action = (rec->action & 0xFFFFFFFE) | + (((packed_record[5] >> 4) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_ingress_prectlf_record(struct atl_hw *hw, + struct aq_mss_ingress_prectlf_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_prectlf_record, hw, rec, + table_index); +} + +static int +set_ingress_preclass_record(struct atl_hw *hw, + const struct aq_mss_ingress_preclass_record *rec, + u16 table_index) +{ + u16 packed_record[20]; + + if (table_index >= NUMROWS_INGRESSPRECLASSRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 20); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->sci[0] >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->sci[0] >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->sci[1] >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->sci[1] >> 16) & 0xFFFF) << 0); + + packed_record[4] = + (packed_record[4] & 0xFF00) | (((rec->tci >> 0) & 0xFF) << 0); + + packed_record[4] = (packed_record[4] & 0x00FF) | + (((rec->encr_offset >> 0) & 0xFF) << 8); + + packed_record[5] = (packed_record[5] & 0x0000) | + (((rec->eth_type >> 0) & 0xFFFF) << 0); + + packed_record[6] = (packed_record[6] & 0x0000) | + (((rec->snap[0] >> 0) & 0xFFFF) << 0); + packed_record[7] = (packed_record[7] & 0x0000) | + (((rec->snap[0] >> 16) & 0xFFFF) << 0); + + packed_record[8] = (packed_record[8] & 0xFF00) | + (((rec->snap[1] >> 0) & 0xFF) << 0); + + packed_record[8] = + (packed_record[8] & 0x00FF) | (((rec->llc >> 0) & 0xFF) << 8); + packed_record[9] = + (packed_record[9] & 0x0000) | (((rec->llc >> 8) & 0xFFFF) << 0); + + packed_record[10] = (packed_record[10] & 0x0000) | + (((rec->mac_sa[0] >> 0) & 0xFFFF) << 0); + packed_record[11] = (packed_record[11] & 0x0000) | + (((rec->mac_sa[0] >> 16) & 0xFFFF) << 0); + + packed_record[12] = (packed_record[12] & 0x0000) | + (((rec->mac_sa[1] >> 0) & 0xFFFF) << 0); + + packed_record[13] = (packed_record[13] & 0x0000) | + (((rec->mac_da[0] >> 0) & 0xFFFF) << 0); + packed_record[14] = (packed_record[14] & 0x0000) | + (((rec->mac_da[0] >> 16) & 0xFFFF) << 0); + + packed_record[15] = (packed_record[15] & 0x0000) | + (((rec->mac_da[1] >> 0) & 0xFFFF) << 0); + + packed_record[16] = (packed_record[16] & 0xFFFE) | + (((rec->lpbk_packet >> 0) & 0x1) << 0); + + packed_record[16] = (packed_record[16] & 0xFFF9) | + (((rec->an_mask >> 0) & 0x3) << 1); + + packed_record[16] = (packed_record[16] & 0xFE07) | + (((rec->tci_mask >> 0) & 0x3F) << 3); + + packed_record[16] = (packed_record[16] & 0x01FF) | + (((rec->sci_mask >> 0) & 0x7F) << 9); + packed_record[17] = (packed_record[17] & 0xFFFE) | + (((rec->sci_mask >> 7) & 0x1) << 0); + + packed_record[17] = (packed_record[17] & 0xFFF9) | + (((rec->eth_type_mask >> 0) & 0x3) << 1); + + packed_record[17] = (packed_record[17] & 0xFF07) | + (((rec->snap_mask >> 0) & 0x1F) << 3); + + packed_record[17] = (packed_record[17] & 0xF8FF) | + (((rec->llc_mask >> 0) & 0x7) << 8); + + packed_record[17] = (packed_record[17] & 0xF7FF) | + (((rec->_802_2_encapsulate >> 0) & 0x1) << 11); + + packed_record[17] = (packed_record[17] & 0x0FFF) | + (((rec->sa_mask >> 0) & 0xF) << 12); + packed_record[18] = (packed_record[18] & 0xFFFC) | + (((rec->sa_mask >> 4) & 0x3) << 0); + + packed_record[18] = (packed_record[18] & 0xFF03) | + (((rec->da_mask >> 0) & 0x3F) << 2); + + packed_record[18] = (packed_record[18] & 0xFEFF) | + (((rec->lpbk_mask >> 0) & 0x1) << 8); + + packed_record[18] = (packed_record[18] & 0xC1FF) | + (((rec->sc_idx >> 0) & 0x1F) << 9); + + packed_record[18] = (packed_record[18] & 0xBFFF) | + (((rec->proc_dest >> 0) & 0x1) << 14); + + packed_record[18] = (packed_record[18] & 0x7FFF) | + (((rec->action >> 0) & 0x1) << 15); + packed_record[19] = (packed_record[19] & 0xFFFE) | + (((rec->action >> 1) & 0x1) << 0); + + packed_record[19] = (packed_record[19] & 0xFFFD) | + (((rec->ctrl_unctrl >> 0) & 0x1) << 1); + + packed_record[19] = (packed_record[19] & 0xFFFB) | + (((rec->sci_from_table >> 0) & 0x1) << 2); + + packed_record[19] = (packed_record[19] & 0xFF87) | + (((rec->reserved >> 0) & 0xF) << 3); + + packed_record[19] = + (packed_record[19] & 0xFF7F) | (((rec->valid >> 0) & 0x1) << 7); + + return set_raw_ingress_record(hw, packed_record, 20, 1, + ROWOFFSET_INGRESSPRECLASSRECORD + + table_index); +} + +int aq_mss_set_ingress_preclass_record( + struct atl_hw *hw, const struct aq_mss_ingress_preclass_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_preclass_record, hw, rec, + table_index); +} + +static int +get_ingress_preclass_record(struct atl_hw *hw, + struct aq_mss_ingress_preclass_record *rec, + u16 table_index) +{ + u16 packed_record[20]; + int ret; + + if (table_index >= NUMROWS_INGRESSPRECLASSRECORD) + return -EINVAL; + + /* If the row that we want to read is odd, first read the previous even + * row, throw that value away, and finally read the desired row. + */ + if ((table_index % 2) > 0) { + ret = get_raw_ingress_record(hw, packed_record, 20, 1, + ROWOFFSET_INGRESSPRECLASSRECORD + + table_index - 1); + if (unlikely(ret)) + return ret; + } + + ret = get_raw_ingress_record(hw, packed_record, 20, 1, + ROWOFFSET_INGRESSPRECLASSRECORD + + table_index); + if (unlikely(ret)) + return ret; + + rec->sci[0] = (rec->sci[0] & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->sci[0] = (rec->sci[0] & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->sci[1] = (rec->sci[1] & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->sci[1] = (rec->sci[1] & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->tci = (rec->tci & 0xFFFFFF00) | + (((packed_record[4] >> 0) & 0xFF) << 0); + + rec->encr_offset = (rec->encr_offset & 0xFFFFFF00) | + (((packed_record[4] >> 8) & 0xFF) << 0); + + rec->eth_type = (rec->eth_type & 0xFFFF0000) | + (((packed_record[5] >> 0) & 0xFFFF) << 0); + + rec->snap[0] = (rec->snap[0] & 0xFFFF0000) | + (((packed_record[6] >> 0) & 0xFFFF) << 0); + rec->snap[0] = (rec->snap[0] & 0x0000FFFF) | + (((packed_record[7] >> 0) & 0xFFFF) << 16); + + rec->snap[1] = (rec->snap[1] & 0xFFFFFF00) | + (((packed_record[8] >> 0) & 0xFF) << 0); + + rec->llc = (rec->llc & 0xFFFFFF00) | + (((packed_record[8] >> 8) & 0xFF) << 0); + rec->llc = (rec->llc & 0xFF0000FF) | + (((packed_record[9] >> 0) & 0xFFFF) << 8); + + rec->mac_sa[0] = (rec->mac_sa[0] & 0xFFFF0000) | + (((packed_record[10] >> 0) & 0xFFFF) << 0); + rec->mac_sa[0] = (rec->mac_sa[0] & 0x0000FFFF) | + (((packed_record[11] >> 0) & 0xFFFF) << 16); + + rec->mac_sa[1] = (rec->mac_sa[1] & 0xFFFF0000) | + (((packed_record[12] >> 0) & 0xFFFF) << 0); + + rec->mac_da[0] = (rec->mac_da[0] & 0xFFFF0000) | + (((packed_record[13] >> 0) & 0xFFFF) << 0); + rec->mac_da[0] = (rec->mac_da[0] & 0x0000FFFF) | + (((packed_record[14] >> 0) & 0xFFFF) << 16); + + rec->mac_da[1] = (rec->mac_da[1] & 0xFFFF0000) | + (((packed_record[15] >> 0) & 0xFFFF) << 0); + + rec->lpbk_packet = (rec->lpbk_packet & 0xFFFFFFFE) | + (((packed_record[16] >> 0) & 0x1) << 0); + + rec->an_mask = (rec->an_mask & 0xFFFFFFFC) | + (((packed_record[16] >> 1) & 0x3) << 0); + + rec->tci_mask = (rec->tci_mask & 0xFFFFFFC0) | + (((packed_record[16] >> 3) & 0x3F) << 0); + + rec->sci_mask = (rec->sci_mask & 0xFFFFFF80) | + (((packed_record[16] >> 9) & 0x7F) << 0); + rec->sci_mask = (rec->sci_mask & 0xFFFFFF7F) | + (((packed_record[17] >> 0) & 0x1) << 7); + + rec->eth_type_mask = (rec->eth_type_mask & 0xFFFFFFFC) | + (((packed_record[17] >> 1) & 0x3) << 0); + + rec->snap_mask = (rec->snap_mask & 0xFFFFFFE0) | + (((packed_record[17] >> 3) & 0x1F) << 0); + + rec->llc_mask = (rec->llc_mask & 0xFFFFFFF8) | + (((packed_record[17] >> 8) & 0x7) << 0); + + rec->_802_2_encapsulate = (rec->_802_2_encapsulate & 0xFFFFFFFE) | + (((packed_record[17] >> 11) & 0x1) << 0); + + rec->sa_mask = (rec->sa_mask & 0xFFFFFFF0) | + (((packed_record[17] >> 12) & 0xF) << 0); + rec->sa_mask = (rec->sa_mask & 0xFFFFFFCF) | + (((packed_record[18] >> 0) & 0x3) << 4); + + rec->da_mask = (rec->da_mask & 0xFFFFFFC0) | + (((packed_record[18] >> 2) & 0x3F) << 0); + + rec->lpbk_mask = (rec->lpbk_mask & 0xFFFFFFFE) | + (((packed_record[18] >> 8) & 0x1) << 0); + + rec->sc_idx = (rec->sc_idx & 0xFFFFFFE0) | + (((packed_record[18] >> 9) & 0x1F) << 0); + + rec->proc_dest = (rec->proc_dest & 0xFFFFFFFE) | + (((packed_record[18] >> 14) & 0x1) << 0); + + rec->action = (rec->action & 0xFFFFFFFE) | + (((packed_record[18] >> 15) & 0x1) << 0); + rec->action = (rec->action & 0xFFFFFFFD) | + (((packed_record[19] >> 0) & 0x1) << 1); + + rec->ctrl_unctrl = (rec->ctrl_unctrl & 0xFFFFFFFE) | + (((packed_record[19] >> 1) & 0x1) << 0); + + rec->sci_from_table = (rec->sci_from_table & 0xFFFFFFFE) | + (((packed_record[19] >> 2) & 0x1) << 0); + + rec->reserved = (rec->reserved & 0xFFFFFFF0) | + (((packed_record[19] >> 3) & 0xF) << 0); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[19] >> 7) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_ingress_preclass_record( + struct atl_hw *hw, struct aq_mss_ingress_preclass_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_preclass_record, hw, rec, + table_index); +} + +static int set_ingress_sc_record(struct atl_hw *hw, + const struct aq_mss_ingress_sc_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + + if (table_index >= NUMROWS_INGRESSSCRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 8); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->stop_time >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->stop_time >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->start_time >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->start_time >> 16) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0xFFFC) | + (((rec->validate_frames >> 0) & 0x3) << 0); + + packed_record[4] = (packed_record[4] & 0xFFFB) | + (((rec->replay_protect >> 0) & 0x1) << 2); + + packed_record[4] = (packed_record[4] & 0x0007) | + (((rec->anti_replay_window >> 0) & 0x1FFF) << 3); + packed_record[5] = (packed_record[5] & 0x0000) | + (((rec->anti_replay_window >> 13) & 0xFFFF) << 0); + packed_record[6] = (packed_record[6] & 0xFFF8) | + (((rec->anti_replay_window >> 29) & 0x7) << 0); + + packed_record[6] = (packed_record[6] & 0xFFF7) | + (((rec->receiving >> 0) & 0x1) << 3); + + packed_record[6] = + (packed_record[6] & 0xFFEF) | (((rec->fresh >> 0) & 0x1) << 4); + + packed_record[6] = + (packed_record[6] & 0xFFDF) | (((rec->an_rol >> 0) & 0x1) << 5); + + packed_record[6] = (packed_record[6] & 0x003F) | + (((rec->reserved >> 0) & 0x3FF) << 6); + packed_record[7] = (packed_record[7] & 0x8000) | + (((rec->reserved >> 10) & 0x7FFF) << 0); + + packed_record[7] = + (packed_record[7] & 0x7FFF) | (((rec->valid >> 0) & 0x1) << 15); + + return set_raw_ingress_record(hw, packed_record, 8, 3, + ROWOFFSET_INGRESSSCRECORD + table_index); +} + +int aq_mss_set_ingress_sc_record(struct atl_hw *hw, + const struct aq_mss_ingress_sc_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_sc_record, hw, rec, table_index); +} + +static int get_ingress_sc_record(struct atl_hw *hw, + struct aq_mss_ingress_sc_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + int ret; + + if (table_index >= NUMROWS_INGRESSSCRECORD) + return -EINVAL; + + ret = get_raw_ingress_record(hw, packed_record, 8, 3, + ROWOFFSET_INGRESSSCRECORD + table_index); + if (unlikely(ret)) + return ret; + + rec->stop_time = (rec->stop_time & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->stop_time = (rec->stop_time & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->start_time = (rec->start_time & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->start_time = (rec->start_time & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->validate_frames = (rec->validate_frames & 0xFFFFFFFC) | + (((packed_record[4] >> 0) & 0x3) << 0); + + rec->replay_protect = (rec->replay_protect & 0xFFFFFFFE) | + (((packed_record[4] >> 2) & 0x1) << 0); + + rec->anti_replay_window = (rec->anti_replay_window & 0xFFFFE000) | + (((packed_record[4] >> 3) & 0x1FFF) << 0); + rec->anti_replay_window = (rec->anti_replay_window & 0xE0001FFF) | + (((packed_record[5] >> 0) & 0xFFFF) << 13); + rec->anti_replay_window = (rec->anti_replay_window & 0x1FFFFFFF) | + (((packed_record[6] >> 0) & 0x7) << 29); + + rec->receiving = (rec->receiving & 0xFFFFFFFE) | + (((packed_record[6] >> 3) & 0x1) << 0); + + rec->fresh = (rec->fresh & 0xFFFFFFFE) | + (((packed_record[6] >> 4) & 0x1) << 0); + + rec->an_rol = (rec->an_rol & 0xFFFFFFFE) | + (((packed_record[6] >> 5) & 0x1) << 0); + + rec->reserved = (rec->reserved & 0xFFFFFC00) | + (((packed_record[6] >> 6) & 0x3FF) << 0); + rec->reserved = (rec->reserved & 0xFE0003FF) | + (((packed_record[7] >> 0) & 0x7FFF) << 10); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[7] >> 15) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_ingress_sc_record(struct atl_hw *hw, + struct aq_mss_ingress_sc_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_sc_record, hw, rec, table_index); +} + +static int set_ingress_sa_record(struct atl_hw *hw, + const struct aq_mss_ingress_sa_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + + if (table_index >= NUMROWS_INGRESSSARECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 8); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->stop_time >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->stop_time >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->start_time >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->start_time >> 16) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->next_pn >> 0) & 0xFFFF) << 0); + packed_record[5] = (packed_record[5] & 0x0000) | + (((rec->next_pn >> 16) & 0xFFFF) << 0); + + packed_record[6] = (packed_record[6] & 0xFFFE) | + (((rec->sat_nextpn >> 0) & 0x1) << 0); + + packed_record[6] = + (packed_record[6] & 0xFFFD) | (((rec->in_use >> 0) & 0x1) << 1); + + packed_record[6] = + (packed_record[6] & 0xFFFB) | (((rec->fresh >> 0) & 0x1) << 2); + + packed_record[6] = (packed_record[6] & 0x0007) | + (((rec->reserved >> 0) & 0x1FFF) << 3); + packed_record[7] = (packed_record[7] & 0x8000) | + (((rec->reserved >> 13) & 0x7FFF) << 0); + + packed_record[7] = + (packed_record[7] & 0x7FFF) | (((rec->valid >> 0) & 0x1) << 15); + + return set_raw_ingress_record(hw, packed_record, 8, 3, + ROWOFFSET_INGRESSSARECORD + table_index); +} + +int aq_mss_set_ingress_sa_record(struct atl_hw *hw, + const struct aq_mss_ingress_sa_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_sa_record, hw, rec, table_index); +} + +static int get_ingress_sa_record(struct atl_hw *hw, + struct aq_mss_ingress_sa_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + int ret; + + if (table_index >= NUMROWS_INGRESSSARECORD) + return -EINVAL; + + ret = get_raw_ingress_record(hw, packed_record, 8, 3, + ROWOFFSET_INGRESSSARECORD + table_index); + if (unlikely(ret)) + return ret; + + rec->stop_time = (rec->stop_time & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->stop_time = (rec->stop_time & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->start_time = (rec->start_time & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->start_time = (rec->start_time & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->next_pn = (rec->next_pn & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + rec->next_pn = (rec->next_pn & 0x0000FFFF) | + (((packed_record[5] >> 0) & 0xFFFF) << 16); + + rec->sat_nextpn = (rec->sat_nextpn & 0xFFFFFFFE) | + (((packed_record[6] >> 0) & 0x1) << 0); + + rec->in_use = (rec->in_use & 0xFFFFFFFE) | + (((packed_record[6] >> 1) & 0x1) << 0); + + rec->fresh = (rec->fresh & 0xFFFFFFFE) | + (((packed_record[6] >> 2) & 0x1) << 0); + + rec->reserved = (rec->reserved & 0xFFFFE000) | + (((packed_record[6] >> 3) & 0x1FFF) << 0); + rec->reserved = (rec->reserved & 0xF0001FFF) | + (((packed_record[7] >> 0) & 0x7FFF) << 13); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[7] >> 15) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_ingress_sa_record(struct atl_hw *hw, + struct aq_mss_ingress_sa_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_sa_record, hw, rec, table_index); +} + +static int +set_ingress_sakey_record(struct atl_hw *hw, + const struct aq_mss_ingress_sakey_record *rec, + u16 table_index) +{ + u16 packed_record[18]; + + if (table_index >= NUMROWS_INGRESSSAKEYRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 18); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->key[0] >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->key[0] >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->key[1] >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->key[1] >> 16) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->key[2] >> 0) & 0xFFFF) << 0); + packed_record[5] = (packed_record[5] & 0x0000) | + (((rec->key[2] >> 16) & 0xFFFF) << 0); + + packed_record[6] = (packed_record[6] & 0x0000) | + (((rec->key[3] >> 0) & 0xFFFF) << 0); + packed_record[7] = (packed_record[7] & 0x0000) | + (((rec->key[3] >> 16) & 0xFFFF) << 0); + + packed_record[8] = (packed_record[8] & 0x0000) | + (((rec->key[4] >> 0) & 0xFFFF) << 0); + packed_record[9] = (packed_record[9] & 0x0000) | + (((rec->key[4] >> 16) & 0xFFFF) << 0); + + packed_record[10] = (packed_record[10] & 0x0000) | + (((rec->key[5] >> 0) & 0xFFFF) << 0); + packed_record[11] = (packed_record[11] & 0x0000) | + (((rec->key[5] >> 16) & 0xFFFF) << 0); + + packed_record[12] = (packed_record[12] & 0x0000) | + (((rec->key[6] >> 0) & 0xFFFF) << 0); + packed_record[13] = (packed_record[13] & 0x0000) | + (((rec->key[6] >> 16) & 0xFFFF) << 0); + + packed_record[14] = (packed_record[14] & 0x0000) | + (((rec->key[7] >> 0) & 0xFFFF) << 0); + packed_record[15] = (packed_record[15] & 0x0000) | + (((rec->key[7] >> 16) & 0xFFFF) << 0); + + packed_record[16] = (packed_record[16] & 0xFFFC) | + (((rec->key_len >> 0) & 0x3) << 0); + + return set_raw_ingress_record(hw, packed_record, 18, 2, + ROWOFFSET_INGRESSSAKEYRECORD + + table_index); +} + +int aq_mss_set_ingress_sakey_record( + struct atl_hw *hw, const struct aq_mss_ingress_sakey_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_sakey_record, hw, rec, table_index); +} + +static int get_ingress_sakey_record(struct atl_hw *hw, + struct aq_mss_ingress_sakey_record *rec, + u16 table_index) +{ + u16 packed_record[18]; + int ret; + + if (table_index >= NUMROWS_INGRESSSAKEYRECORD) + return -EINVAL; + + ret = get_raw_ingress_record(hw, packed_record, 18, 2, + ROWOFFSET_INGRESSSAKEYRECORD + + table_index); + if (unlikely(ret)) + return ret; + + rec->key[0] = (rec->key[0] & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->key[0] = (rec->key[0] & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->key[1] = (rec->key[1] & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->key[1] = (rec->key[1] & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->key[2] = (rec->key[2] & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + rec->key[2] = (rec->key[2] & 0x0000FFFF) | + (((packed_record[5] >> 0) & 0xFFFF) << 16); + + rec->key[3] = (rec->key[3] & 0xFFFF0000) | + (((packed_record[6] >> 0) & 0xFFFF) << 0); + rec->key[3] = (rec->key[3] & 0x0000FFFF) | + (((packed_record[7] >> 0) & 0xFFFF) << 16); + + rec->key[4] = (rec->key[4] & 0xFFFF0000) | + (((packed_record[8] >> 0) & 0xFFFF) << 0); + rec->key[4] = (rec->key[4] & 0x0000FFFF) | + (((packed_record[9] >> 0) & 0xFFFF) << 16); + + rec->key[5] = (rec->key[5] & 0xFFFF0000) | + (((packed_record[10] >> 0) & 0xFFFF) << 0); + rec->key[5] = (rec->key[5] & 0x0000FFFF) | + (((packed_record[11] >> 0) & 0xFFFF) << 16); + + rec->key[6] = (rec->key[6] & 0xFFFF0000) | + (((packed_record[12] >> 0) & 0xFFFF) << 0); + rec->key[6] = (rec->key[6] & 0x0000FFFF) | + (((packed_record[13] >> 0) & 0xFFFF) << 16); + + rec->key[7] = (rec->key[7] & 0xFFFF0000) | + (((packed_record[14] >> 0) & 0xFFFF) << 0); + rec->key[7] = (rec->key[7] & 0x0000FFFF) | + (((packed_record[15] >> 0) & 0xFFFF) << 16); + + rec->key_len = (rec->key_len & 0xFFFFFFFC) | + (((packed_record[16] >> 0) & 0x3) << 0); + + return 0; +} + +int aq_mss_get_ingress_sakey_record(struct atl_hw *hw, + struct aq_mss_ingress_sakey_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_sakey_record, hw, rec, table_index); +} + +static int +set_ingress_postclass_record(struct atl_hw *hw, + const struct aq_mss_ingress_postclass_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + + if (table_index >= NUMROWS_INGRESSPOSTCLASSRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 8); + + packed_record[0] = + (packed_record[0] & 0xFF00) | (((rec->byte0 >> 0) & 0xFF) << 0); + + packed_record[0] = + (packed_record[0] & 0x00FF) | (((rec->byte1 >> 0) & 0xFF) << 8); + + packed_record[1] = + (packed_record[1] & 0xFF00) | (((rec->byte2 >> 0) & 0xFF) << 0); + + packed_record[1] = + (packed_record[1] & 0x00FF) | (((rec->byte3 >> 0) & 0xFF) << 8); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->eth_type >> 0) & 0xFFFF) << 0); + + packed_record[3] = (packed_record[3] & 0xFFFE) | + (((rec->eth_type_valid >> 0) & 0x1) << 0); + + packed_record[3] = (packed_record[3] & 0xE001) | + (((rec->vlan_id >> 0) & 0xFFF) << 1); + + packed_record[3] = (packed_record[3] & 0x1FFF) | + (((rec->vlan_up >> 0) & 0x7) << 13); + + packed_record[4] = (packed_record[4] & 0xFFFE) | + (((rec->vlan_valid >> 0) & 0x1) << 0); + + packed_record[4] = + (packed_record[4] & 0xFFC1) | (((rec->sai >> 0) & 0x1F) << 1); + + packed_record[4] = (packed_record[4] & 0xFFBF) | + (((rec->sai_hit >> 0) & 0x1) << 6); + + packed_record[4] = (packed_record[4] & 0xF87F) | + (((rec->eth_type_mask >> 0) & 0xF) << 7); + + packed_record[4] = (packed_record[4] & 0x07FF) | + (((rec->byte3_location >> 0) & 0x1F) << 11); + packed_record[5] = (packed_record[5] & 0xFFFE) | + (((rec->byte3_location >> 5) & 0x1) << 0); + + packed_record[5] = (packed_record[5] & 0xFFF9) | + (((rec->byte3_mask >> 0) & 0x3) << 1); + + packed_record[5] = (packed_record[5] & 0xFE07) | + (((rec->byte2_location >> 0) & 0x3F) << 3); + + packed_record[5] = (packed_record[5] & 0xF9FF) | + (((rec->byte2_mask >> 0) & 0x3) << 9); + + packed_record[5] = (packed_record[5] & 0x07FF) | + (((rec->byte1_location >> 0) & 0x1F) << 11); + packed_record[6] = (packed_record[6] & 0xFFFE) | + (((rec->byte1_location >> 5) & 0x1) << 0); + + packed_record[6] = (packed_record[6] & 0xFFF9) | + (((rec->byte1_mask >> 0) & 0x3) << 1); + + packed_record[6] = (packed_record[6] & 0xFE07) | + (((rec->byte0_location >> 0) & 0x3F) << 3); + + packed_record[6] = (packed_record[6] & 0xF9FF) | + (((rec->byte0_mask >> 0) & 0x3) << 9); + + packed_record[6] = (packed_record[6] & 0xE7FF) | + (((rec->eth_type_valid_mask >> 0) & 0x3) << 11); + + packed_record[6] = (packed_record[6] & 0x1FFF) | + (((rec->vlan_id_mask >> 0) & 0x7) << 13); + packed_record[7] = (packed_record[7] & 0xFFFE) | + (((rec->vlan_id_mask >> 3) & 0x1) << 0); + + packed_record[7] = (packed_record[7] & 0xFFF9) | + (((rec->vlan_up_mask >> 0) & 0x3) << 1); + + packed_record[7] = (packed_record[7] & 0xFFE7) | + (((rec->vlan_valid_mask >> 0) & 0x3) << 3); + + packed_record[7] = (packed_record[7] & 0xFF9F) | + (((rec->sai_mask >> 0) & 0x3) << 5); + + packed_record[7] = (packed_record[7] & 0xFE7F) | + (((rec->sai_hit_mask >> 0) & 0x3) << 7); + + packed_record[7] = (packed_record[7] & 0xFDFF) | + (((rec->firstlevel_actions >> 0) & 0x1) << 9); + + packed_record[7] = (packed_record[7] & 0xFBFF) | + (((rec->secondlevel_actions >> 0) & 0x1) << 10); + + packed_record[7] = (packed_record[7] & 0x87FF) | + (((rec->reserved >> 0) & 0xF) << 11); + + packed_record[7] = + (packed_record[7] & 0x7FFF) | (((rec->valid >> 0) & 0x1) << 15); + + return set_raw_ingress_record(hw, packed_record, 8, 4, + ROWOFFSET_INGRESSPOSTCLASSRECORD + + table_index); +} + +int aq_mss_set_ingress_postclass_record( + struct atl_hw *hw, const struct aq_mss_ingress_postclass_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_postclass_record, hw, rec, + table_index); +} + +static int +get_ingress_postclass_record(struct atl_hw *hw, + struct aq_mss_ingress_postclass_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + int ret; + + if (table_index >= NUMROWS_INGRESSPOSTCLASSRECORD) + return -EINVAL; + + /* If the row that we want to read is odd, first read the previous even + * row, throw that value away, and finally read the desired row. + */ + if ((table_index % 2) > 0) { + ret = get_raw_ingress_record(hw, packed_record, 8, 4, + ROWOFFSET_INGRESSPOSTCLASSRECORD + + table_index - 1); + if (unlikely(ret)) + return ret; + } + + ret = get_raw_ingress_record(hw, packed_record, 8, 4, + ROWOFFSET_INGRESSPOSTCLASSRECORD + + table_index); + if (unlikely(ret)) + return ret; + + rec->byte0 = (rec->byte0 & 0xFFFFFF00) | + (((packed_record[0] >> 0) & 0xFF) << 0); + + rec->byte1 = (rec->byte1 & 0xFFFFFF00) | + (((packed_record[0] >> 8) & 0xFF) << 0); + + rec->byte2 = (rec->byte2 & 0xFFFFFF00) | + (((packed_record[1] >> 0) & 0xFF) << 0); + + rec->byte3 = (rec->byte3 & 0xFFFFFF00) | + (((packed_record[1] >> 8) & 0xFF) << 0); + + rec->eth_type = (rec->eth_type & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + + rec->eth_type_valid = (rec->eth_type_valid & 0xFFFFFFFE) | + (((packed_record[3] >> 0) & 0x1) << 0); + + rec->vlan_id = (rec->vlan_id & 0xFFFFF000) | + (((packed_record[3] >> 1) & 0xFFF) << 0); + + rec->vlan_up = (rec->vlan_up & 0xFFFFFFF8) | + (((packed_record[3] >> 13) & 0x7) << 0); + + rec->vlan_valid = (rec->vlan_valid & 0xFFFFFFFE) | + (((packed_record[4] >> 0) & 0x1) << 0); + + rec->sai = (rec->sai & 0xFFFFFFE0) | + (((packed_record[4] >> 1) & 0x1F) << 0); + + rec->sai_hit = (rec->sai_hit & 0xFFFFFFFE) | + (((packed_record[4] >> 6) & 0x1) << 0); + + rec->eth_type_mask = (rec->eth_type_mask & 0xFFFFFFF0) | + (((packed_record[4] >> 7) & 0xF) << 0); + + rec->byte3_location = (rec->byte3_location & 0xFFFFFFE0) | + (((packed_record[4] >> 11) & 0x1F) << 0); + rec->byte3_location = (rec->byte3_location & 0xFFFFFFDF) | + (((packed_record[5] >> 0) & 0x1) << 5); + + rec->byte3_mask = (rec->byte3_mask & 0xFFFFFFFC) | + (((packed_record[5] >> 1) & 0x3) << 0); + + rec->byte2_location = (rec->byte2_location & 0xFFFFFFC0) | + (((packed_record[5] >> 3) & 0x3F) << 0); + + rec->byte2_mask = (rec->byte2_mask & 0xFFFFFFFC) | + (((packed_record[5] >> 9) & 0x3) << 0); + + rec->byte1_location = (rec->byte1_location & 0xFFFFFFE0) | + (((packed_record[5] >> 11) & 0x1F) << 0); + rec->byte1_location = (rec->byte1_location & 0xFFFFFFDF) | + (((packed_record[6] >> 0) & 0x1) << 5); + + rec->byte1_mask = (rec->byte1_mask & 0xFFFFFFFC) | + (((packed_record[6] >> 1) & 0x3) << 0); + + rec->byte0_location = (rec->byte0_location & 0xFFFFFFC0) | + (((packed_record[6] >> 3) & 0x3F) << 0); + + rec->byte0_mask = (rec->byte0_mask & 0xFFFFFFFC) | + (((packed_record[6] >> 9) & 0x3) << 0); + + rec->eth_type_valid_mask = (rec->eth_type_valid_mask & 0xFFFFFFFC) | + (((packed_record[6] >> 11) & 0x3) << 0); + + rec->vlan_id_mask = (rec->vlan_id_mask & 0xFFFFFFF8) | + (((packed_record[6] >> 13) & 0x7) << 0); + rec->vlan_id_mask = (rec->vlan_id_mask & 0xFFFFFFF7) | + (((packed_record[7] >> 0) & 0x1) << 3); + + rec->vlan_up_mask = (rec->vlan_up_mask & 0xFFFFFFFC) | + (((packed_record[7] >> 1) & 0x3) << 0); + + rec->vlan_valid_mask = (rec->vlan_valid_mask & 0xFFFFFFFC) | + (((packed_record[7] >> 3) & 0x3) << 0); + + rec->sai_mask = (rec->sai_mask & 0xFFFFFFFC) | + (((packed_record[7] >> 5) & 0x3) << 0); + + rec->sai_hit_mask = (rec->sai_hit_mask & 0xFFFFFFFC) | + (((packed_record[7] >> 7) & 0x3) << 0); + + rec->firstlevel_actions = (rec->firstlevel_actions & 0xFFFFFFFE) | + (((packed_record[7] >> 9) & 0x1) << 0); + + rec->secondlevel_actions = (rec->secondlevel_actions & 0xFFFFFFFE) | + (((packed_record[7] >> 10) & 0x1) << 0); + + rec->reserved = (rec->reserved & 0xFFFFFFF0) | + (((packed_record[7] >> 11) & 0xF) << 0); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[7] >> 15) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_ingress_postclass_record( + struct atl_hw *hw, struct aq_mss_ingress_postclass_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_postclass_record, hw, rec, + table_index); +} + +static int +set_ingress_postctlf_record(struct atl_hw *hw, + const struct aq_mss_ingress_postctlf_record *rec, + u16 table_index) +{ + u16 packed_record[6]; + + if (table_index >= NUMROWS_INGRESSPOSTCTLFRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 6); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->sa_da[0] >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->sa_da[0] >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->sa_da[1] >> 0) & 0xFFFF) << 0); + + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->eth_type >> 0) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->match_mask >> 0) & 0xFFFF) << 0); + + packed_record[5] = (packed_record[5] & 0xFFF0) | + (((rec->match_type >> 0) & 0xF) << 0); + + packed_record[5] = + (packed_record[5] & 0xFFEF) | (((rec->action >> 0) & 0x1) << 4); + + return set_raw_ingress_record(hw, packed_record, 6, 5, + ROWOFFSET_INGRESSPOSTCTLFRECORD + + table_index); +} + +int aq_mss_set_ingress_postctlf_record( + struct atl_hw *hw, const struct aq_mss_ingress_postctlf_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_ingress_postctlf_record, hw, rec, + table_index); +} + +static int +get_ingress_postctlf_record(struct atl_hw *hw, + struct aq_mss_ingress_postctlf_record *rec, + u16 table_index) +{ + u16 packed_record[6]; + int ret; + + if (table_index >= NUMROWS_INGRESSPOSTCTLFRECORD) + return -EINVAL; + + /* If the row that we want to read is odd, first read the previous even + * row, throw that value away, and finally read the desired row. + */ + if ((table_index % 2) > 0) { + ret = get_raw_ingress_record(hw, packed_record, 6, 5, + ROWOFFSET_INGRESSPOSTCTLFRECORD + + table_index - 1); + if (unlikely(ret)) + return ret; + } + + ret = get_raw_ingress_record(hw, packed_record, 6, 5, + ROWOFFSET_INGRESSPOSTCTLFRECORD + + table_index); + if (unlikely(ret)) + return ret; + + rec->sa_da[0] = (rec->sa_da[0] & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->sa_da[0] = (rec->sa_da[0] & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->sa_da[1] = (rec->sa_da[1] & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + + rec->eth_type = (rec->eth_type & 0xFFFF0000) | + (((packed_record[3] >> 0) & 0xFFFF) << 0); + + rec->match_mask = (rec->match_mask & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + + rec->match_type = (rec->match_type & 0xFFFFFFF0) | + (((packed_record[5] >> 0) & 0xF) << 0); + + rec->action = (rec->action & 0xFFFFFFFE) | + (((packed_record[5] >> 4) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_ingress_postctlf_record( + struct atl_hw *hw, struct aq_mss_ingress_postctlf_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_ingress_postctlf_record, hw, rec, + table_index); +} + +static int set_egress_ctlf_record(struct atl_hw *hw, + const struct aq_mss_egress_ctlf_record *rec, + u16 table_index) +{ + u16 packed_record[6]; + + if (table_index >= NUMROWS_EGRESSCTLFRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 6); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->sa_da[0] >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->sa_da[0] >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->sa_da[1] >> 0) & 0xFFFF) << 0); + + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->eth_type >> 0) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->match_mask >> 0) & 0xFFFF) << 0); + + packed_record[5] = (packed_record[5] & 0xFFF0) | + (((rec->match_type >> 0) & 0xF) << 0); + + packed_record[5] = + (packed_record[5] & 0xFFEF) | (((rec->action >> 0) & 0x1) << 4); + + return set_raw_egress_record(hw, packed_record, 6, 0, + ROWOFFSET_EGRESSCTLFRECORD + table_index); +} + +int aq_mss_set_egress_ctlf_record(struct atl_hw *hw, + const struct aq_mss_egress_ctlf_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_egress_ctlf_record, hw, rec, table_index); +} + +static int get_egress_ctlf_record(struct atl_hw *hw, + struct aq_mss_egress_ctlf_record *rec, + u16 table_index) +{ + u16 packed_record[6]; + int ret; + + if (table_index >= NUMROWS_EGRESSCTLFRECORD) + return -EINVAL; + + /* If the row that we want to read is odd, first read the previous even + * row, throw that value away, and finally read the desired row. + */ + if ((table_index % 2) > 0) { + ret = get_raw_egress_record(hw, packed_record, 6, 0, + ROWOFFSET_EGRESSCTLFRECORD + + table_index - 1); + if (unlikely(ret)) + return ret; + } + + ret = get_raw_egress_record(hw, packed_record, 6, 0, + ROWOFFSET_EGRESSCTLFRECORD + table_index); + if (unlikely(ret)) + return ret; + + rec->sa_da[0] = (rec->sa_da[0] & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->sa_da[0] = (rec->sa_da[0] & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->sa_da[1] = (rec->sa_da[1] & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + + rec->eth_type = (rec->eth_type & 0xFFFF0000) | + (((packed_record[3] >> 0) & 0xFFFF) << 0); + + rec->match_mask = (rec->match_mask & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + + rec->match_type = (rec->match_type & 0xFFFFFFF0) | + (((packed_record[5] >> 0) & 0xF) << 0); + + rec->action = (rec->action & 0xFFFFFFFE) | + (((packed_record[5] >> 4) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_egress_ctlf_record(struct atl_hw *hw, + struct aq_mss_egress_ctlf_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_egress_ctlf_record, hw, rec, table_index); +} + +static int set_egress_class_record(struct atl_hw *hw, + const struct aq_mss_egress_class_record *rec, + u16 table_index) +{ + u16 packed_record[28]; + + if (table_index >= NUMROWS_EGRESSCLASSRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 28); + + packed_record[0] = (packed_record[0] & 0xF000) | + (((rec->vlan_id >> 0) & 0xFFF) << 0); + + packed_record[0] = (packed_record[0] & 0x8FFF) | + (((rec->vlan_up >> 0) & 0x7) << 12); + + packed_record[0] = (packed_record[0] & 0x7FFF) | + (((rec->vlan_valid >> 0) & 0x1) << 15); + + packed_record[1] = + (packed_record[1] & 0xFF00) | (((rec->byte3 >> 0) & 0xFF) << 0); + + packed_record[1] = + (packed_record[1] & 0x00FF) | (((rec->byte2 >> 0) & 0xFF) << 8); + + packed_record[2] = + (packed_record[2] & 0xFF00) | (((rec->byte1 >> 0) & 0xFF) << 0); + + packed_record[2] = + (packed_record[2] & 0x00FF) | (((rec->byte0 >> 0) & 0xFF) << 8); + + packed_record[3] = + (packed_record[3] & 0xFF00) | (((rec->tci >> 0) & 0xFF) << 0); + + packed_record[3] = (packed_record[3] & 0x00FF) | + (((rec->sci[0] >> 0) & 0xFF) << 8); + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->sci[0] >> 8) & 0xFFFF) << 0); + packed_record[5] = (packed_record[5] & 0xFF00) | + (((rec->sci[0] >> 24) & 0xFF) << 0); + + packed_record[5] = (packed_record[5] & 0x00FF) | + (((rec->sci[1] >> 0) & 0xFF) << 8); + packed_record[6] = (packed_record[6] & 0x0000) | + (((rec->sci[1] >> 8) & 0xFFFF) << 0); + packed_record[7] = (packed_record[7] & 0xFF00) | + (((rec->sci[1] >> 24) & 0xFF) << 0); + + packed_record[7] = (packed_record[7] & 0x00FF) | + (((rec->eth_type >> 0) & 0xFF) << 8); + packed_record[8] = (packed_record[8] & 0xFF00) | + (((rec->eth_type >> 8) & 0xFF) << 0); + + packed_record[8] = (packed_record[8] & 0x00FF) | + (((rec->snap[0] >> 0) & 0xFF) << 8); + packed_record[9] = (packed_record[9] & 0x0000) | + (((rec->snap[0] >> 8) & 0xFFFF) << 0); + packed_record[10] = (packed_record[10] & 0xFF00) | + (((rec->snap[0] >> 24) & 0xFF) << 0); + + packed_record[10] = (packed_record[10] & 0x00FF) | + (((rec->snap[1] >> 0) & 0xFF) << 8); + + packed_record[11] = (packed_record[11] & 0x0000) | + (((rec->llc >> 0) & 0xFFFF) << 0); + packed_record[12] = + (packed_record[12] & 0xFF00) | (((rec->llc >> 16) & 0xFF) << 0); + + packed_record[12] = (packed_record[12] & 0x00FF) | + (((rec->mac_sa[0] >> 0) & 0xFF) << 8); + packed_record[13] = (packed_record[13] & 0x0000) | + (((rec->mac_sa[0] >> 8) & 0xFFFF) << 0); + packed_record[14] = (packed_record[14] & 0xFF00) | + (((rec->mac_sa[0] >> 24) & 0xFF) << 0); + + packed_record[14] = (packed_record[14] & 0x00FF) | + (((rec->mac_sa[1] >> 0) & 0xFF) << 8); + packed_record[15] = (packed_record[15] & 0xFF00) | + (((rec->mac_sa[1] >> 8) & 0xFF) << 0); + + packed_record[15] = (packed_record[15] & 0x00FF) | + (((rec->mac_da[0] >> 0) & 0xFF) << 8); + packed_record[16] = (packed_record[16] & 0x0000) | + (((rec->mac_da[0] >> 8) & 0xFFFF) << 0); + packed_record[17] = (packed_record[17] & 0xFF00) | + (((rec->mac_da[0] >> 24) & 0xFF) << 0); + + packed_record[17] = (packed_record[17] & 0x00FF) | + (((rec->mac_da[1] >> 0) & 0xFF) << 8); + packed_record[18] = (packed_record[18] & 0xFF00) | + (((rec->mac_da[1] >> 8) & 0xFF) << 0); + + packed_record[18] = + (packed_record[18] & 0x00FF) | (((rec->pn >> 0) & 0xFF) << 8); + packed_record[19] = + (packed_record[19] & 0x0000) | (((rec->pn >> 8) & 0xFFFF) << 0); + packed_record[20] = + (packed_record[20] & 0xFF00) | (((rec->pn >> 24) & 0xFF) << 0); + + packed_record[20] = (packed_record[20] & 0xC0FF) | + (((rec->byte3_location >> 0) & 0x3F) << 8); + + packed_record[20] = (packed_record[20] & 0xBFFF) | + (((rec->byte3_mask >> 0) & 0x1) << 14); + + packed_record[20] = (packed_record[20] & 0x7FFF) | + (((rec->byte2_location >> 0) & 0x1) << 15); + packed_record[21] = (packed_record[21] & 0xFFE0) | + (((rec->byte2_location >> 1) & 0x1F) << 0); + + packed_record[21] = (packed_record[21] & 0xFFDF) | + (((rec->byte2_mask >> 0) & 0x1) << 5); + + packed_record[21] = (packed_record[21] & 0xF03F) | + (((rec->byte1_location >> 0) & 0x3F) << 6); + + packed_record[21] = (packed_record[21] & 0xEFFF) | + (((rec->byte1_mask >> 0) & 0x1) << 12); + + packed_record[21] = (packed_record[21] & 0x1FFF) | + (((rec->byte0_location >> 0) & 0x7) << 13); + packed_record[22] = (packed_record[22] & 0xFFF8) | + (((rec->byte0_location >> 3) & 0x7) << 0); + + packed_record[22] = (packed_record[22] & 0xFFF7) | + (((rec->byte0_mask >> 0) & 0x1) << 3); + + packed_record[22] = (packed_record[22] & 0xFFCF) | + (((rec->vlan_id_mask >> 0) & 0x3) << 4); + + packed_record[22] = (packed_record[22] & 0xFFBF) | + (((rec->vlan_up_mask >> 0) & 0x1) << 6); + + packed_record[22] = (packed_record[22] & 0xFF7F) | + (((rec->vlan_valid_mask >> 0) & 0x1) << 7); + + packed_record[22] = (packed_record[22] & 0x00FF) | + (((rec->tci_mask >> 0) & 0xFF) << 8); + + packed_record[23] = (packed_record[23] & 0xFF00) | + (((rec->sci_mask >> 0) & 0xFF) << 0); + + packed_record[23] = (packed_record[23] & 0xFCFF) | + (((rec->eth_type_mask >> 0) & 0x3) << 8); + + packed_record[23] = (packed_record[23] & 0x83FF) | + (((rec->snap_mask >> 0) & 0x1F) << 10); + + packed_record[23] = (packed_record[23] & 0x7FFF) | + (((rec->llc_mask >> 0) & 0x1) << 15); + packed_record[24] = (packed_record[24] & 0xFFFC) | + (((rec->llc_mask >> 1) & 0x3) << 0); + + packed_record[24] = (packed_record[24] & 0xFF03) | + (((rec->sa_mask >> 0) & 0x3F) << 2); + + packed_record[24] = (packed_record[24] & 0xC0FF) | + (((rec->da_mask >> 0) & 0x3F) << 8); + + packed_record[24] = (packed_record[24] & 0x3FFF) | + (((rec->pn_mask >> 0) & 0x3) << 14); + packed_record[25] = (packed_record[25] & 0xFFFC) | + (((rec->pn_mask >> 2) & 0x3) << 0); + + packed_record[25] = (packed_record[25] & 0xFFFB) | + (((rec->eight02dot2 >> 0) & 0x1) << 2); + + packed_record[25] = (packed_record[25] & 0xFFF7) | + (((rec->tci_sc >> 0) & 0x1) << 3); + + packed_record[25] = (packed_record[25] & 0xFFEF) | + (((rec->tci_87543 >> 0) & 0x1) << 4); + + packed_record[25] = (packed_record[25] & 0xFFDF) | + (((rec->exp_sectag_en >> 0) & 0x1) << 5); + + packed_record[25] = (packed_record[25] & 0xF83F) | + (((rec->sc_idx >> 0) & 0x1F) << 6); + + packed_record[25] = (packed_record[25] & 0xE7FF) | + (((rec->sc_sa >> 0) & 0x3) << 11); + + packed_record[25] = (packed_record[25] & 0xDFFF) | + (((rec->debug >> 0) & 0x1) << 13); + + packed_record[25] = (packed_record[25] & 0x3FFF) | + (((rec->action >> 0) & 0x3) << 14); + + packed_record[26] = + (packed_record[26] & 0xFFF7) | (((rec->valid >> 0) & 0x1) << 3); + + return set_raw_egress_record(hw, packed_record, 28, 1, + ROWOFFSET_EGRESSCLASSRECORD + table_index); +} + +int aq_mss_set_egress_class_record(struct atl_hw *hw, + const struct aq_mss_egress_class_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_egress_class_record, hw, rec, table_index); +} + +static int get_egress_class_record(struct atl_hw *hw, + struct aq_mss_egress_class_record *rec, + u16 table_index) +{ + u16 packed_record[28]; + int ret; + + if (table_index >= NUMROWS_EGRESSCLASSRECORD) + return -EINVAL; + + /* If the row that we want to read is odd, first read the previous even + * row, throw that value away, and finally read the desired row. + */ + if ((table_index % 2) > 0) { + ret = get_raw_egress_record(hw, packed_record, 28, 1, + ROWOFFSET_EGRESSCLASSRECORD + + table_index - 1); + if (unlikely(ret)) + return ret; + } + + ret = get_raw_egress_record(hw, packed_record, 28, 1, + ROWOFFSET_EGRESSCLASSRECORD + table_index); + if (unlikely(ret)) + return ret; + + rec->vlan_id = (rec->vlan_id & 0xFFFFF000) | + (((packed_record[0] >> 0) & 0xFFF) << 0); + + rec->vlan_up = (rec->vlan_up & 0xFFFFFFF8) | + (((packed_record[0] >> 12) & 0x7) << 0); + + rec->vlan_valid = (rec->vlan_valid & 0xFFFFFFFE) | + (((packed_record[0] >> 15) & 0x1) << 0); + + rec->byte3 = (rec->byte3 & 0xFFFFFF00) | + (((packed_record[1] >> 0) & 0xFF) << 0); + + rec->byte2 = (rec->byte2 & 0xFFFFFF00) | + (((packed_record[1] >> 8) & 0xFF) << 0); + + rec->byte1 = (rec->byte1 & 0xFFFFFF00) | + (((packed_record[2] >> 0) & 0xFF) << 0); + + rec->byte0 = (rec->byte0 & 0xFFFFFF00) | + (((packed_record[2] >> 8) & 0xFF) << 0); + + rec->tci = (rec->tci & 0xFFFFFF00) | + (((packed_record[3] >> 0) & 0xFF) << 0); + + rec->sci[0] = (rec->sci[0] & 0xFFFFFF00) | + (((packed_record[3] >> 8) & 0xFF) << 0); + rec->sci[0] = (rec->sci[0] & 0xFF0000FF) | + (((packed_record[4] >> 0) & 0xFFFF) << 8); + rec->sci[0] = (rec->sci[0] & 0x00FFFFFF) | + (((packed_record[5] >> 0) & 0xFF) << 24); + + rec->sci[1] = (rec->sci[1] & 0xFFFFFF00) | + (((packed_record[5] >> 8) & 0xFF) << 0); + rec->sci[1] = (rec->sci[1] & 0xFF0000FF) | + (((packed_record[6] >> 0) & 0xFFFF) << 8); + rec->sci[1] = (rec->sci[1] & 0x00FFFFFF) | + (((packed_record[7] >> 0) & 0xFF) << 24); + + rec->eth_type = (rec->eth_type & 0xFFFFFF00) | + (((packed_record[7] >> 8) & 0xFF) << 0); + rec->eth_type = (rec->eth_type & 0xFFFF00FF) | + (((packed_record[8] >> 0) & 0xFF) << 8); + + rec->snap[0] = (rec->snap[0] & 0xFFFFFF00) | + (((packed_record[8] >> 8) & 0xFF) << 0); + rec->snap[0] = (rec->snap[0] & 0xFF0000FF) | + (((packed_record[9] >> 0) & 0xFFFF) << 8); + rec->snap[0] = (rec->snap[0] & 0x00FFFFFF) | + (((packed_record[10] >> 0) & 0xFF) << 24); + + rec->snap[1] = (rec->snap[1] & 0xFFFFFF00) | + (((packed_record[10] >> 8) & 0xFF) << 0); + + rec->llc = (rec->llc & 0xFFFF0000) | + (((packed_record[11] >> 0) & 0xFFFF) << 0); + rec->llc = (rec->llc & 0xFF00FFFF) | + (((packed_record[12] >> 0) & 0xFF) << 16); + + rec->mac_sa[0] = (rec->mac_sa[0] & 0xFFFFFF00) | + (((packed_record[12] >> 8) & 0xFF) << 0); + rec->mac_sa[0] = (rec->mac_sa[0] & 0xFF0000FF) | + (((packed_record[13] >> 0) & 0xFFFF) << 8); + rec->mac_sa[0] = (rec->mac_sa[0] & 0x00FFFFFF) | + (((packed_record[14] >> 0) & 0xFF) << 24); + + rec->mac_sa[1] = (rec->mac_sa[1] & 0xFFFFFF00) | + (((packed_record[14] >> 8) & 0xFF) << 0); + rec->mac_sa[1] = (rec->mac_sa[1] & 0xFFFF00FF) | + (((packed_record[15] >> 0) & 0xFF) << 8); + + rec->mac_da[0] = (rec->mac_da[0] & 0xFFFFFF00) | + (((packed_record[15] >> 8) & 0xFF) << 0); + rec->mac_da[0] = (rec->mac_da[0] & 0xFF0000FF) | + (((packed_record[16] >> 0) & 0xFFFF) << 8); + rec->mac_da[0] = (rec->mac_da[0] & 0x00FFFFFF) | + (((packed_record[17] >> 0) & 0xFF) << 24); + + rec->mac_da[1] = (rec->mac_da[1] & 0xFFFFFF00) | + (((packed_record[17] >> 8) & 0xFF) << 0); + rec->mac_da[1] = (rec->mac_da[1] & 0xFFFF00FF) | + (((packed_record[18] >> 0) & 0xFF) << 8); + + rec->pn = (rec->pn & 0xFFFFFF00) | + (((packed_record[18] >> 8) & 0xFF) << 0); + rec->pn = (rec->pn & 0xFF0000FF) | + (((packed_record[19] >> 0) & 0xFFFF) << 8); + rec->pn = (rec->pn & 0x00FFFFFF) | + (((packed_record[20] >> 0) & 0xFF) << 24); + + rec->byte3_location = (rec->byte3_location & 0xFFFFFFC0) | + (((packed_record[20] >> 8) & 0x3F) << 0); + + rec->byte3_mask = (rec->byte3_mask & 0xFFFFFFFE) | + (((packed_record[20] >> 14) & 0x1) << 0); + + rec->byte2_location = (rec->byte2_location & 0xFFFFFFFE) | + (((packed_record[20] >> 15) & 0x1) << 0); + rec->byte2_location = (rec->byte2_location & 0xFFFFFFC1) | + (((packed_record[21] >> 0) & 0x1F) << 1); + + rec->byte2_mask = (rec->byte2_mask & 0xFFFFFFFE) | + (((packed_record[21] >> 5) & 0x1) << 0); + + rec->byte1_location = (rec->byte1_location & 0xFFFFFFC0) | + (((packed_record[21] >> 6) & 0x3F) << 0); + + rec->byte1_mask = (rec->byte1_mask & 0xFFFFFFFE) | + (((packed_record[21] >> 12) & 0x1) << 0); + + rec->byte0_location = (rec->byte0_location & 0xFFFFFFF8) | + (((packed_record[21] >> 13) & 0x7) << 0); + rec->byte0_location = (rec->byte0_location & 0xFFFFFFC7) | + (((packed_record[22] >> 0) & 0x7) << 3); + + rec->byte0_mask = (rec->byte0_mask & 0xFFFFFFFE) | + (((packed_record[22] >> 3) & 0x1) << 0); + + rec->vlan_id_mask = (rec->vlan_id_mask & 0xFFFFFFFC) | + (((packed_record[22] >> 4) & 0x3) << 0); + + rec->vlan_up_mask = (rec->vlan_up_mask & 0xFFFFFFFE) | + (((packed_record[22] >> 6) & 0x1) << 0); + + rec->vlan_valid_mask = (rec->vlan_valid_mask & 0xFFFFFFFE) | + (((packed_record[22] >> 7) & 0x1) << 0); + + rec->tci_mask = (rec->tci_mask & 0xFFFFFF00) | + (((packed_record[22] >> 8) & 0xFF) << 0); + + rec->sci_mask = (rec->sci_mask & 0xFFFFFF00) | + (((packed_record[23] >> 0) & 0xFF) << 0); + + rec->eth_type_mask = (rec->eth_type_mask & 0xFFFFFFFC) | + (((packed_record[23] >> 8) & 0x3) << 0); + + rec->snap_mask = (rec->snap_mask & 0xFFFFFFE0) | + (((packed_record[23] >> 10) & 0x1F) << 0); + + rec->llc_mask = (rec->llc_mask & 0xFFFFFFFE) | + (((packed_record[23] >> 15) & 0x1) << 0); + rec->llc_mask = (rec->llc_mask & 0xFFFFFFF9) | + (((packed_record[24] >> 0) & 0x3) << 1); + + rec->sa_mask = (rec->sa_mask & 0xFFFFFFC0) | + (((packed_record[24] >> 2) & 0x3F) << 0); + + rec->da_mask = (rec->da_mask & 0xFFFFFFC0) | + (((packed_record[24] >> 8) & 0x3F) << 0); + + rec->pn_mask = (rec->pn_mask & 0xFFFFFFFC) | + (((packed_record[24] >> 14) & 0x3) << 0); + rec->pn_mask = (rec->pn_mask & 0xFFFFFFF3) | + (((packed_record[25] >> 0) & 0x3) << 2); + + rec->eight02dot2 = (rec->eight02dot2 & 0xFFFFFFFE) | + (((packed_record[25] >> 2) & 0x1) << 0); + + rec->tci_sc = (rec->tci_sc & 0xFFFFFFFE) | + (((packed_record[25] >> 3) & 0x1) << 0); + + rec->tci_87543 = (rec->tci_87543 & 0xFFFFFFFE) | + (((packed_record[25] >> 4) & 0x1) << 0); + + rec->exp_sectag_en = (rec->exp_sectag_en & 0xFFFFFFFE) | + (((packed_record[25] >> 5) & 0x1) << 0); + + rec->sc_idx = (rec->sc_idx & 0xFFFFFFE0) | + (((packed_record[25] >> 6) & 0x1F) << 0); + + rec->sc_sa = (rec->sc_sa & 0xFFFFFFFC) | + (((packed_record[25] >> 11) & 0x3) << 0); + + rec->debug = (rec->debug & 0xFFFFFFFE) | + (((packed_record[25] >> 13) & 0x1) << 0); + + rec->action = (rec->action & 0xFFFFFFFC) | + (((packed_record[25] >> 14) & 0x3) << 0); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[26] >> 3) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_egress_class_record(struct atl_hw *hw, + struct aq_mss_egress_class_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_egress_class_record, hw, rec, table_index); +} + +static int set_egress_sc_record(struct atl_hw *hw, + const struct aq_mss_egress_sc_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + + if (table_index >= NUMROWS_EGRESSSCRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 8); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->start_time >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->start_time >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->stop_time >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->stop_time >> 16) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0xFFFC) | + (((rec->curr_an >> 0) & 0x3) << 0); + + packed_record[4] = (packed_record[4] & 0xFFFB) | + (((rec->an_roll >> 0) & 0x1) << 2); + + packed_record[4] = + (packed_record[4] & 0xFE07) | (((rec->tci >> 0) & 0x3F) << 3); + + packed_record[4] = (packed_record[4] & 0x01FF) | + (((rec->enc_off >> 0) & 0x7F) << 9); + packed_record[5] = (packed_record[5] & 0xFFFE) | + (((rec->enc_off >> 7) & 0x1) << 0); + + packed_record[5] = (packed_record[5] & 0xFFFD) | + (((rec->protect >> 0) & 0x1) << 1); + + packed_record[5] = + (packed_record[5] & 0xFFFB) | (((rec->recv >> 0) & 0x1) << 2); + + packed_record[5] = + (packed_record[5] & 0xFFF7) | (((rec->fresh >> 0) & 0x1) << 3); + + packed_record[5] = (packed_record[5] & 0xFFCF) | + (((rec->sak_len >> 0) & 0x3) << 4); + + packed_record[7] = + (packed_record[7] & 0x7FFF) | (((rec->valid >> 0) & 0x1) << 15); + + return set_raw_egress_record(hw, packed_record, 8, 2, + ROWOFFSET_EGRESSSCRECORD + table_index); +} + +int aq_mss_set_egress_sc_record(struct atl_hw *hw, + const struct aq_mss_egress_sc_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_egress_sc_record, hw, rec, table_index); +} + +static int get_egress_sc_record(struct atl_hw *hw, + struct aq_mss_egress_sc_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + int ret; + + if (table_index >= NUMROWS_EGRESSSCRECORD) + return -EINVAL; + + ret = get_raw_egress_record(hw, packed_record, 8, 2, + ROWOFFSET_EGRESSSCRECORD + table_index); + if (unlikely(ret)) + return ret; + + rec->start_time = (rec->start_time & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->start_time = (rec->start_time & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->stop_time = (rec->stop_time & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->stop_time = (rec->stop_time & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->curr_an = (rec->curr_an & 0xFFFFFFFC) | + (((packed_record[4] >> 0) & 0x3) << 0); + + rec->an_roll = (rec->an_roll & 0xFFFFFFFE) | + (((packed_record[4] >> 2) & 0x1) << 0); + + rec->tci = (rec->tci & 0xFFFFFFC0) | + (((packed_record[4] >> 3) & 0x3F) << 0); + + rec->enc_off = (rec->enc_off & 0xFFFFFF80) | + (((packed_record[4] >> 9) & 0x7F) << 0); + rec->enc_off = (rec->enc_off & 0xFFFFFF7F) | + (((packed_record[5] >> 0) & 0x1) << 7); + + rec->protect = (rec->protect & 0xFFFFFFFE) | + (((packed_record[5] >> 1) & 0x1) << 0); + + rec->recv = (rec->recv & 0xFFFFFFFE) | + (((packed_record[5] >> 2) & 0x1) << 0); + + rec->fresh = (rec->fresh & 0xFFFFFFFE) | + (((packed_record[5] >> 3) & 0x1) << 0); + + rec->sak_len = (rec->sak_len & 0xFFFFFFFC) | + (((packed_record[5] >> 4) & 0x3) << 0); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[7] >> 15) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_egress_sc_record(struct atl_hw *hw, + struct aq_mss_egress_sc_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_egress_sc_record, hw, rec, table_index); +} + +static int set_egress_sa_record(struct atl_hw *hw, + const struct aq_mss_egress_sa_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + + if (table_index >= NUMROWS_EGRESSSARECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 8); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->start_time >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->start_time >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->stop_time >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->stop_time >> 16) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->next_pn >> 0) & 0xFFFF) << 0); + packed_record[5] = (packed_record[5] & 0x0000) | + (((rec->next_pn >> 16) & 0xFFFF) << 0); + + packed_record[6] = + (packed_record[6] & 0xFFFE) | (((rec->sat_pn >> 0) & 0x1) << 0); + + packed_record[6] = + (packed_record[6] & 0xFFFD) | (((rec->fresh >> 0) & 0x1) << 1); + + packed_record[7] = + (packed_record[7] & 0x7FFF) | (((rec->valid >> 0) & 0x1) << 15); + + return set_raw_egress_record(hw, packed_record, 8, 2, + ROWOFFSET_EGRESSSARECORD + table_index); +} + +int aq_mss_set_egress_sa_record(struct atl_hw *hw, + const struct aq_mss_egress_sa_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_egress_sa_record, hw, rec, table_index); +} + +static int get_egress_sa_record(struct atl_hw *hw, + struct aq_mss_egress_sa_record *rec, + u16 table_index) +{ + u16 packed_record[8]; + int ret; + + if (table_index >= NUMROWS_EGRESSSARECORD) + return -EINVAL; + + ret = get_raw_egress_record(hw, packed_record, 8, 2, + ROWOFFSET_EGRESSSARECORD + table_index); + if (unlikely(ret)) + return ret; + + rec->start_time = (rec->start_time & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->start_time = (rec->start_time & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->stop_time = (rec->stop_time & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->stop_time = (rec->stop_time & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->next_pn = (rec->next_pn & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + rec->next_pn = (rec->next_pn & 0x0000FFFF) | + (((packed_record[5] >> 0) & 0xFFFF) << 16); + + rec->sat_pn = (rec->sat_pn & 0xFFFFFFFE) | + (((packed_record[6] >> 0) & 0x1) << 0); + + rec->fresh = (rec->fresh & 0xFFFFFFFE) | + (((packed_record[6] >> 1) & 0x1) << 0); + + rec->valid = (rec->valid & 0xFFFFFFFE) | + (((packed_record[7] >> 15) & 0x1) << 0); + + return 0; +} + +int aq_mss_get_egress_sa_record(struct atl_hw *hw, + struct aq_mss_egress_sa_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_egress_sa_record, hw, rec, table_index); +} + +static int set_egress_sakey_record(struct atl_hw *hw, + const struct aq_mss_egress_sakey_record *rec, + u16 table_index) +{ + u16 packed_record[16]; + int ret; + + if (table_index >= NUMROWS_EGRESSSAKEYRECORD) + return -EINVAL; + + memset(packed_record, 0, sizeof(u16) * 16); + + packed_record[0] = (packed_record[0] & 0x0000) | + (((rec->key[0] >> 0) & 0xFFFF) << 0); + packed_record[1] = (packed_record[1] & 0x0000) | + (((rec->key[0] >> 16) & 0xFFFF) << 0); + + packed_record[2] = (packed_record[2] & 0x0000) | + (((rec->key[1] >> 0) & 0xFFFF) << 0); + packed_record[3] = (packed_record[3] & 0x0000) | + (((rec->key[1] >> 16) & 0xFFFF) << 0); + + packed_record[4] = (packed_record[4] & 0x0000) | + (((rec->key[2] >> 0) & 0xFFFF) << 0); + packed_record[5] = (packed_record[5] & 0x0000) | + (((rec->key[2] >> 16) & 0xFFFF) << 0); + + packed_record[6] = (packed_record[6] & 0x0000) | + (((rec->key[3] >> 0) & 0xFFFF) << 0); + packed_record[7] = (packed_record[7] & 0x0000) | + (((rec->key[3] >> 16) & 0xFFFF) << 0); + + packed_record[8] = (packed_record[8] & 0x0000) | + (((rec->key[4] >> 0) & 0xFFFF) << 0); + packed_record[9] = (packed_record[9] & 0x0000) | + (((rec->key[4] >> 16) & 0xFFFF) << 0); + + packed_record[10] = (packed_record[10] & 0x0000) | + (((rec->key[5] >> 0) & 0xFFFF) << 0); + packed_record[11] = (packed_record[11] & 0x0000) | + (((rec->key[5] >> 16) & 0xFFFF) << 0); + + packed_record[12] = (packed_record[12] & 0x0000) | + (((rec->key[6] >> 0) & 0xFFFF) << 0); + packed_record[13] = (packed_record[13] & 0x0000) | + (((rec->key[6] >> 16) & 0xFFFF) << 0); + + packed_record[14] = (packed_record[14] & 0x0000) | + (((rec->key[7] >> 0) & 0xFFFF) << 0); + packed_record[15] = (packed_record[15] & 0x0000) | + (((rec->key[7] >> 16) & 0xFFFF) << 0); + + ret = set_raw_egress_record(hw, packed_record, 8, 2, + ROWOFFSET_EGRESSSAKEYRECORD + table_index); + if (unlikely(ret)) + return ret; + ret = set_raw_egress_record(hw, packed_record + 8, 8, 2, + ROWOFFSET_EGRESSSAKEYRECORD + table_index - + 32); + if (unlikely(ret)) + return ret; + + return 0; +} + +int aq_mss_set_egress_sakey_record(struct atl_hw *hw, + const struct aq_mss_egress_sakey_record *rec, + u16 table_index) +{ + return AQ_API_CALL_SAFE(set_egress_sakey_record, hw, rec, table_index); +} + +static int get_egress_sakey_record(struct atl_hw *hw, + struct aq_mss_egress_sakey_record *rec, + u16 table_index) +{ + u16 packed_record[16]; + int ret; + + if (table_index >= NUMROWS_EGRESSSAKEYRECORD) + return -EINVAL; + + ret = get_raw_egress_record(hw, packed_record, 8, 2, + ROWOFFSET_EGRESSSAKEYRECORD + table_index); + if (unlikely(ret)) + return ret; + ret = get_raw_egress_record(hw, packed_record + 8, 8, 2, + ROWOFFSET_EGRESSSAKEYRECORD + table_index - + 32); + if (unlikely(ret)) + return ret; + + rec->key[0] = (rec->key[0] & 0xFFFF0000) | + (((packed_record[0] >> 0) & 0xFFFF) << 0); + rec->key[0] = (rec->key[0] & 0x0000FFFF) | + (((packed_record[1] >> 0) & 0xFFFF) << 16); + + rec->key[1] = (rec->key[1] & 0xFFFF0000) | + (((packed_record[2] >> 0) & 0xFFFF) << 0); + rec->key[1] = (rec->key[1] & 0x0000FFFF) | + (((packed_record[3] >> 0) & 0xFFFF) << 16); + + rec->key[2] = (rec->key[2] & 0xFFFF0000) | + (((packed_record[4] >> 0) & 0xFFFF) << 0); + rec->key[2] = (rec->key[2] & 0x0000FFFF) | + (((packed_record[5] >> 0) & 0xFFFF) << 16); + + rec->key[3] = (rec->key[3] & 0xFFFF0000) | + (((packed_record[6] >> 0) & 0xFFFF) << 0); + rec->key[3] = (rec->key[3] & 0x0000FFFF) | + (((packed_record[7] >> 0) & 0xFFFF) << 16); + + rec->key[4] = (rec->key[4] & 0xFFFF0000) | + (((packed_record[8] >> 0) & 0xFFFF) << 0); + rec->key[4] = (rec->key[4] & 0x0000FFFF) | + (((packed_record[9] >> 0) & 0xFFFF) << 16); + + rec->key[5] = (rec->key[5] & 0xFFFF0000) | + (((packed_record[10] >> 0) & 0xFFFF) << 0); + rec->key[5] = (rec->key[5] & 0x0000FFFF) | + (((packed_record[11] >> 0) & 0xFFFF) << 16); + + rec->key[6] = (rec->key[6] & 0xFFFF0000) | + (((packed_record[12] >> 0) & 0xFFFF) << 0); + rec->key[6] = (rec->key[6] & 0x0000FFFF) | + (((packed_record[13] >> 0) & 0xFFFF) << 16); + + rec->key[7] = (rec->key[7] & 0xFFFF0000) | + (((packed_record[14] >> 0) & 0xFFFF) << 0); + rec->key[7] = (rec->key[7] & 0x0000FFFF) | + (((packed_record[15] >> 0) & 0xFFFF) << 16); + + return 0; +} + +int aq_mss_get_egress_sakey_record(struct atl_hw *hw, + struct aq_mss_egress_sakey_record *rec, + u16 table_index) +{ + memset(rec, 0, sizeof(*rec)); + + return AQ_API_CALL_SAFE(get_egress_sakey_record, hw, rec, table_index); +} + +static int get_egress_sc_counters(struct atl_hw *hw, + struct aq_mss_egress_sc_counters *counters, + u16 sc_index) +{ + u16 packed_record[4]; + int ret; + + if (sc_index >= NUMROWS_EGRESSSCRECORD) + return -EINVAL; + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sc_index * 8 + 4); + if (unlikely(ret)) + return ret; + counters->sc_protected_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sc_protected_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sc_index * 8 + 5); + if (unlikely(ret)) + return ret; + counters->sc_encrypted_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sc_encrypted_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sc_index * 8 + 6); + if (unlikely(ret)) + return ret; + counters->sc_protected_octets[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sc_protected_octets[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sc_index * 8 + 7); + if (unlikely(ret)) + return ret; + counters->sc_encrypted_octets[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sc_encrypted_octets[1] = + packed_record[2] | (packed_record[3] << 16); + + return 0; +} + +int aq_mss_get_egress_sc_counters(struct atl_hw *hw, + struct aq_mss_egress_sc_counters *counters, + u16 sc_index) +{ + memset(counters, 0, sizeof(*counters)); + + return AQ_API_CALL_SAFE(get_egress_sc_counters, hw, counters, sc_index); +} + +static int get_egress_sa_counters(struct atl_hw *hw, + struct aq_mss_egress_sa_counters *counters, + u16 sa_index) +{ + u16 packed_record[4]; + int ret; + + if (sa_index >= NUMROWS_EGRESSSARECORD) + return -EINVAL; + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sa_index * 8 + 0); + if (unlikely(ret)) + return ret; + counters->sa_hit_drop_redirect[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sa_hit_drop_redirect[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sa_index * 8 + 1); + if (unlikely(ret)) + return ret; + counters->sa_protected2_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sa_protected2_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sa_index * 8 + 2); + if (unlikely(ret)) + return ret; + counters->sa_protected_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sa_protected_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, sa_index * 8 + 3); + if (unlikely(ret)) + return ret; + counters->sa_encrypted_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->sa_encrypted_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + return 0; +} + +int aq_mss_get_egress_sa_counters(struct atl_hw *hw, + struct aq_mss_egress_sa_counters *counters, + u16 sa_index) +{ + memset(counters, 0, sizeof(*counters)); + + return AQ_API_CALL_SAFE(get_egress_sa_counters, hw, counters, sa_index); +} + +static int +get_egress_common_counters(struct atl_hw *hw, + struct aq_mss_egress_common_counters *counters) +{ + u16 packed_record[4]; + int ret; + + ret = get_raw_egress_record(hw, packed_record, 4, 3, 256 + 0); + if (unlikely(ret)) + return ret; + counters->ctl_pkt[0] = packed_record[0] | (packed_record[1] << 16); + counters->ctl_pkt[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, 256 + 1); + if (unlikely(ret)) + return ret; + counters->unknown_sa_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unknown_sa_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, 256 + 2); + if (unlikely(ret)) + return ret; + counters->untagged_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->untagged_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, 256 + 3); + if (unlikely(ret)) + return ret; + counters->too_long[0] = packed_record[0] | (packed_record[1] << 16); + counters->too_long[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, 256 + 4); + if (unlikely(ret)) + return ret; + counters->ecc_error_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->ecc_error_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_egress_record(hw, packed_record, 4, 3, 256 + 5); + if (unlikely(ret)) + return ret; + counters->unctrl_hit_drop_redir[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unctrl_hit_drop_redir[1] = + packed_record[2] | (packed_record[3] << 16); + + return 0; +} + +int aq_mss_get_egress_common_counters( + struct atl_hw *hw, struct aq_mss_egress_common_counters *counters) +{ + memset(counters, 0, sizeof(*counters)); + + return AQ_API_CALL_SAFE(get_egress_common_counters, hw, counters); +} + +static int clear_egress_counters(struct atl_hw *hw) +{ + struct mss_egress_ctl_register ctl_reg; + int ret; + + memset(&ctl_reg, 0, sizeof(ctl_reg)); + + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR, + &ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR + 4, + &ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + /* Toggle the Egress MIB clear bit 0->1->0 */ + ctl_reg.bits_0.clear_counter = 0; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR + 4, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + ctl_reg.bits_0.clear_counter = 1; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR + 4, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + ctl_reg.bits_0.clear_counter = 0; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_CTL_REGISTER_ADDR + 4, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + return 0; +} + +int aq_mss_clear_egress_counters(struct atl_hw *hw) +{ + return AQ_API_CALL_SAFE(clear_egress_counters, hw); +} + +static int get_ingress_sa_counters(struct atl_hw *hw, + struct aq_mss_ingress_sa_counters *counters, + u16 sa_index) +{ + u16 packed_record[4]; + int ret; + + if (sa_index >= NUMROWS_INGRESSSARECORD) + return -EINVAL; + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 0); + if (unlikely(ret)) + return ret; + counters->untagged_hit_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->untagged_hit_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 1); + if (unlikely(ret)) + return ret; + counters->ctrl_hit_drop_redir_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->ctrl_hit_drop_redir_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 2); + if (unlikely(ret)) + return ret; + counters->not_using_sa[0] = packed_record[0] | (packed_record[1] << 16); + counters->not_using_sa[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 3); + if (unlikely(ret)) + return ret; + counters->unused_sa[0] = packed_record[0] | (packed_record[1] << 16); + counters->unused_sa[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 4); + if (unlikely(ret)) + return ret; + counters->not_valid_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->not_valid_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 5); + if (unlikely(ret)) + return ret; + counters->invalid_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->invalid_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 6); + if (unlikely(ret)) + return ret; + counters->ok_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->ok_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 7); + if (unlikely(ret)) + return ret; + counters->late_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->late_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 8); + if (unlikely(ret)) + return ret; + counters->delayed_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->delayed_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 9); + if (unlikely(ret)) + return ret; + counters->unchecked_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unchecked_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 10); + if (unlikely(ret)) + return ret; + counters->validated_octets[0] = + packed_record[0] | (packed_record[1] << 16); + counters->validated_octets[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, + sa_index * 12 + 11); + if (unlikely(ret)) + return ret; + counters->decrypted_octets[0] = + packed_record[0] | (packed_record[1] << 16); + counters->decrypted_octets[1] = + packed_record[2] | (packed_record[3] << 16); + + return 0; +} + +int aq_mss_get_ingress_sa_counters(struct atl_hw *hw, + struct aq_mss_ingress_sa_counters *counters, + u16 sa_index) +{ + memset(counters, 0, sizeof(*counters)); + + return AQ_API_CALL_SAFE(get_ingress_sa_counters, hw, counters, + sa_index); +} + +static int +get_ingress_common_counters(struct atl_hw *hw, + struct aq_mss_ingress_common_counters *counters) +{ + u16 packed_record[4]; + int ret; + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 0); + if (unlikely(ret)) + return ret; + counters->ctl_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->ctl_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 1); + if (unlikely(ret)) + return ret; + counters->tagged_miss_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->tagged_miss_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 2); + if (unlikely(ret)) + return ret; + counters->untagged_miss_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->untagged_miss_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 3); + if (unlikely(ret)) + return ret; + counters->notag_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->notag_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 4); + if (unlikely(ret)) + return ret; + counters->untagged_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->untagged_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 5); + if (unlikely(ret)) + return ret; + counters->bad_tag_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->bad_tag_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 6); + if (unlikely(ret)) + return ret; + counters->no_sci_pkts[0] = packed_record[0] | (packed_record[1] << 16); + counters->no_sci_pkts[1] = packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 7); + if (unlikely(ret)) + return ret; + counters->unknown_sci_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unknown_sci_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 8); + if (unlikely(ret)) + return ret; + counters->ctrl_prt_pass_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->ctrl_prt_pass_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 9); + if (unlikely(ret)) + return ret; + counters->unctrl_prt_pass_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unctrl_prt_pass_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 10); + if (unlikely(ret)) + return ret; + counters->ctrl_prt_fail_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->ctrl_prt_fail_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 11); + if (unlikely(ret)) + return ret; + counters->unctrl_prt_fail_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unctrl_prt_fail_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 12); + if (unlikely(ret)) + return ret; + counters->too_long_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->too_long_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 13); + if (unlikely(ret)) + return ret; + counters->igpoc_ctl_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->igpoc_ctl_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 14); + if (unlikely(ret)) + return ret; + counters->ecc_error_pkts[0] = + packed_record[0] | (packed_record[1] << 16); + counters->ecc_error_pkts[1] = + packed_record[2] | (packed_record[3] << 16); + + ret = get_raw_ingress_record(hw, packed_record, 4, 6, 385 + 15); + if (unlikely(ret)) + return ret; + counters->unctrl_hit_drop_redir[0] = + packed_record[0] | (packed_record[1] << 16); + counters->unctrl_hit_drop_redir[1] = + packed_record[2] | (packed_record[3] << 16); + + return 0; +} + +int aq_mss_get_ingress_common_counters( + struct atl_hw *hw, struct aq_mss_ingress_common_counters *counters) +{ + memset(counters, 0, sizeof(*counters)); + + return AQ_API_CALL_SAFE(get_ingress_common_counters, hw, counters); +} + +static int clear_ingress_counters(struct atl_hw *hw) +{ + struct mss_ingress_ctl_register ctl_reg; + int ret; + + memset(&ctl_reg, 0, sizeof(ctl_reg)); + + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR, + &ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR + 4, + &ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + /* Toggle the Ingress MIB clear bit 0->1->0 */ + ctl_reg.bits_0.clear_count = 0; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR + 4, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + ctl_reg.bits_0.clear_count = 1; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR + 4, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + ctl_reg.bits_0.clear_count = 0; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR + 4, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + return 0; +} + +int aq_mss_clear_ingress_counters(struct atl_hw *hw) +{ + return AQ_API_CALL_SAFE(clear_ingress_counters, hw); +} + +static int get_egress_sa_expired(struct atl_hw *hw, u32 *expired) +{ + u16 val; + int ret; + + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_EXPIRED_STATUS_REGISTER_ADDR, &val); + if (unlikely(ret)) + return ret; + + *expired = val; + + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_EXPIRED_STATUS_REGISTER_ADDR + 1, + &val); + if (unlikely(ret)) + return ret; + + *expired |= val << 16; + + return 0; +} + +int aq_mss_get_egress_sa_expired(struct atl_hw *hw, u32 *expired) +{ + *expired = 0; + + return AQ_API_CALL_SAFE(get_egress_sa_expired, hw, expired); +} + +static int get_egress_sa_threshold_expired(struct atl_hw *hw, u32 *expired) +{ + u16 val; + int ret; + + ret = __atl_mdio_read( + hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_THRESHOLD_EXPIRED_STATUS_REGISTER_ADDR, &val); + if (unlikely(ret)) + return ret; + + *expired = val; + + ret = __atl_mdio_read( + hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_THRESHOLD_EXPIRED_STATUS_REGISTER_ADDR + 1, &val); + if (unlikely(ret)) + return ret; + + *expired |= val << 16; + + return 0; +} + +int aq_mss_get_egress_sa_threshold_expired(struct atl_hw *hw, u32 *expired) +{ + *expired = 0; + + return AQ_API_CALL_SAFE(get_egress_sa_threshold_expired, hw, expired); +} + +static int set_egress_sa_expired(struct atl_hw *hw, u32 expired) +{ + int ret; + + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_EXPIRED_STATUS_REGISTER_ADDR, + expired & 0xFFFF); + if (unlikely(ret)) + return ret; + + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_EXPIRED_STATUS_REGISTER_ADDR + 1, + expired >> 16); + if (unlikely(ret)) + return ret; + + return 0; +} + +int aq_mss_set_egress_sa_expired(struct atl_hw *hw, u32 expired) +{ + return AQ_API_CALL_SAFE(set_egress_sa_expired, hw, expired); +} + +static int set_egress_sa_threshold_expired(struct atl_hw *hw, u32 expired) +{ + int ret; + + ret = __atl_mdio_write( + hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_THRESHOLD_EXPIRED_STATUS_REGISTER_ADDR, + expired & 0xFFFF); + if (unlikely(ret)) + return ret; + + ret = __atl_mdio_write( + hw, 0, MDIO_MMD_VEND1, + MSS_EGRESS_SA_THRESHOLD_EXPIRED_STATUS_REGISTER_ADDR + 1, + expired >> 16); + if (unlikely(ret)) + return ret; + + return 0; +} + +int aq_mss_set_egress_sa_threshold_expired(struct atl_hw *hw, u32 expired) +{ + return AQ_API_CALL_SAFE(set_egress_sa_threshold_expired, hw, expired); +} + + +static int set_drop_igprc_miss_packets(struct atl_hw *hw, bool drop) +{ + struct mss_ingress_ctl_register ctl_reg; + int ret; + + memset(&ctl_reg, 0, sizeof(ctl_reg)); + + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR, + &ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_read(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR + 1, + &ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + ctl_reg.bits_0.drop_igprc_miss = (drop) ? (1) : (0); + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR, + ctl_reg.word_0); + if (unlikely(ret)) + return ret; + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + MSS_INGRESS_CTL_REGISTER_ADDR + 1, + ctl_reg.word_1); + if (unlikely(ret)) + return ret; + + return 0; +} + +int aq_mss_set_drop_igprc_miss_packets(struct atl_hw *hw, bool drop) +{ + return AQ_API_CALL_SAFE(set_drop_igprc_miss_packets, hw, drop); +} + +static int set_packet_edit_control(struct atl_hw *hw, u32 control) +{ + int ret; + + ret = __atl_mdio_write(hw, 0, MDIO_MMD_VEND1, + SEC_INGRESS_PACKET_EDIT_CTL_REGISTER_ADDR, + control); + + return ret; +} + +int aq_mss_set_packet_edit_control(struct atl_hw *hw, u32 control) +{ + return AQ_API_CALL_SAFE(set_packet_edit_control, hw, control); +} + diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.h b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.h new file mode 100644 index 000000000000..bb9eab84b101 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_api.h @@ -0,0 +1,334 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef __MACSEC_API_H__ +#define __MACSEC_API_H__ + +#include "atl_hw.h" +#include "macsec_struct.h" + +#define NUMROWS_INGRESSPRECTLFRECORD 24 +#define ROWOFFSET_INGRESSPRECTLFRECORD 0 + +#define NUMROWS_INGRESSPRECLASSRECORD 48 +#define ROWOFFSET_INGRESSPRECLASSRECORD 0 + +#define NUMROWS_INGRESSPOSTCLASSRECORD 48 +#define ROWOFFSET_INGRESSPOSTCLASSRECORD 0 + +#define NUMROWS_INGRESSSCRECORD 32 +#define ROWOFFSET_INGRESSSCRECORD 0 + +#define NUMROWS_INGRESSSARECORD 32 +#define ROWOFFSET_INGRESSSARECORD 32 + +#define NUMROWS_INGRESSSAKEYRECORD 32 +#define ROWOFFSET_INGRESSSAKEYRECORD 0 + +#define NUMROWS_INGRESSPOSTCTLFRECORD 24 +#define ROWOFFSET_INGRESSPOSTCTLFRECORD 0 + +#define NUMROWS_EGRESSCTLFRECORD 24 +#define ROWOFFSET_EGRESSCTLFRECORD 0 + +#define NUMROWS_EGRESSCLASSRECORD 48 +#define ROWOFFSET_EGRESSCLASSRECORD 0 + +#define NUMROWS_EGRESSSCRECORD 32 +#define ROWOFFSET_EGRESSSCRECORD 0 + +#define NUMROWS_EGRESSSARECORD 32 +#define ROWOFFSET_EGRESSSARECORD 32 + +#define NUMROWS_EGRESSSAKEYRECORD 32 +#define ROWOFFSET_EGRESSSAKEYRECORD 96 + +/*! Read the raw table data from the specified row of the Egress CTL + * Filter table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 23). + */ +int aq_mss_get_egress_ctlf_record(struct atl_hw *hw, + struct aq_mss_egress_ctlf_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Egress CTL Filter table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 23). + */ +int aq_mss_set_egress_ctlf_record(struct atl_hw *hw, + const struct aq_mss_egress_ctlf_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Egress + * Packet Classifier table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 47). + */ +int aq_mss_get_egress_class_record(struct atl_hw *hw, + struct aq_mss_egress_class_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Egress Packet Classifier table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write (max 47). + */ +int aq_mss_set_egress_class_record(struct atl_hw *hw, + const struct aq_mss_egress_class_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Egress SC + * Lookup table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 31). + */ +int aq_mss_get_egress_sc_record(struct atl_hw *hw, + struct aq_mss_egress_sc_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Egress SC Lookup table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write (max 31). + */ +int aq_mss_set_egress_sc_record(struct atl_hw *hw, + const struct aq_mss_egress_sc_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Egress SA + * Lookup table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 31). + */ +int aq_mss_get_egress_sa_record(struct atl_hw *hw, + struct aq_mss_egress_sa_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Egress SA Lookup table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write (max 31). + */ +int aq_mss_set_egress_sa_record(struct atl_hw *hw, + const struct aq_mss_egress_sa_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Egress SA + * Key Lookup table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 31). + */ +int aq_mss_get_egress_sakey_record(struct atl_hw *hw, + struct aq_mss_egress_sakey_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Egress SA Key Lookup table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write (max 31). + */ +int aq_mss_set_egress_sakey_record(struct atl_hw *hw, + const struct aq_mss_egress_sakey_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress + * Pre-MACSec CTL Filter table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 23). + */ +int aq_mss_get_ingress_prectlf_record(struct atl_hw *hw, + struct aq_mss_ingress_prectlf_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress Pre-MACSec CTL Filter table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 23). + */ +int aq_mss_set_ingress_prectlf_record( + struct atl_hw *hw, const struct aq_mss_ingress_prectlf_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress + * Pre-MACSec Packet Classifier table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 47). + */ +int aq_mss_get_ingress_preclass_record( + struct atl_hw *hw, struct aq_mss_ingress_preclass_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress Pre-MACSec Packet Classifier table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 47). + */ +int aq_mss_set_ingress_preclass_record( + struct atl_hw *hw, const struct aq_mss_ingress_preclass_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress SC + * Lookup table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 31). + */ +int aq_mss_get_ingress_sc_record(struct atl_hw *hw, + struct aq_mss_ingress_sc_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress SC Lookup table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 31). + */ +int aq_mss_set_ingress_sc_record(struct atl_hw *hw, + const struct aq_mss_ingress_sc_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress SA + * Lookup table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 31). + */ +int aq_mss_get_ingress_sa_record(struct atl_hw *hw, + struct aq_mss_ingress_sa_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress SA Lookup table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 31). + */ +int aq_mss_set_ingress_sa_record(struct atl_hw *hw, + const struct aq_mss_ingress_sa_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress SA + * Key Lookup table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 31). + */ +int aq_mss_get_ingress_sakey_record(struct atl_hw *hw, + struct aq_mss_ingress_sakey_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress SA Key Lookup table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 31). + */ +int aq_mss_set_ingress_sakey_record( + struct atl_hw *hw, const struct aq_mss_ingress_sakey_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress + * Post-MACSec Packet Classifier table, and unpack it into the + * fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 48). + */ +int aq_mss_get_ingress_postclass_record( + struct atl_hw *hw, struct aq_mss_ingress_postclass_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress Post-MACSec Packet Classifier table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 48). + */ +int aq_mss_set_ingress_postclass_record( + struct atl_hw *hw, const struct aq_mss_ingress_postclass_record *rec, + u16 table_index); + +/*! Read the raw table data from the specified row of the Ingress + * Post-MACSec CTL Filter table, and unpack it into the fields of rec. + * rec - [OUT] The raw table row data will be unpacked into the fields of rec. + * table_index - The table row to read (max 23). + */ +int aq_mss_get_ingress_postctlf_record( + struct atl_hw *hw, struct aq_mss_ingress_postctlf_record *rec, + u16 table_index); + +/*! Pack the fields of rec, and write the packed data into the + * specified row of the Ingress Post-MACSec CTL Filter table. + * rec - [IN] The bitfield values to write to the table row. + * table_index - The table row to write(max 23). + */ +int aq_mss_set_ingress_postctlf_record( + struct atl_hw *hw, const struct aq_mss_ingress_postctlf_record *rec, + u16 table_index); + +/*! Read the counters for the specified SC, and unpack them into the + * fields of counters. + * counters - [OUT] The raw table row data will be unpacked here. + * sc_index - The table row to read (max 31). + */ +int aq_mss_get_egress_sc_counters(struct atl_hw *hw, + struct aq_mss_egress_sc_counters *counters, + u16 sc_index); + +/*! Read the counters for the specified SA, and unpack them into the + * fields of counters. + * counters - [OUT] The raw table row data will be unpacked here. + * sa_index - The table row to read (max 31). + */ +int aq_mss_get_egress_sa_counters(struct atl_hw *hw, + struct aq_mss_egress_sa_counters *counters, + u16 sa_index); + +/*! Read the counters for the common egress counters, and unpack them + * into the fields of counters. + * counters - [OUT] The raw table row data will be unpacked here. + */ +int aq_mss_get_egress_common_counters( + struct atl_hw *hw, struct aq_mss_egress_common_counters *counters); + +/*! Clear all Egress counters to 0.*/ +int aq_mss_clear_egress_counters(struct atl_hw *hw); + +/*! Read the counters for the specified SA, and unpack them into the + * fields of counters. + * counters - [OUT] The raw table row data will be unpacked here. + * sa_index - The table row to read (max 31). + */ +int aq_mss_get_ingress_sa_counters(struct atl_hw *hw, + struct aq_mss_ingress_sa_counters *counters, + u16 sa_index); + +/*! Read the counters for the common ingress counters, and unpack them + * into the fields of counters. + * counters - [OUT] The raw table row data will be unpacked here. + */ +int aq_mss_get_ingress_common_counters( + struct atl_hw *hw, struct aq_mss_ingress_common_counters *counters); + +/*! Clear all Ingress counters to 0. */ +int aq_mss_clear_ingress_counters(struct atl_hw *hw); + +/*! Get Egress SA expired. */ +int aq_mss_get_egress_sa_expired(struct atl_hw *hw, u32 *expired); +/*! Get Egress SA threshold expired. */ +int aq_mss_get_egress_sa_threshold_expired(struct atl_hw *hw, + u32 *expired); +/*! Set Egress SA expired. */ +int aq_mss_set_egress_sa_expired(struct atl_hw *hw, u32 expired); +/*! Set Egress SA threshold expired. */ +int aq_mss_set_egress_sa_threshold_expired(struct atl_hw *hw, u32 expired); + +/*! Set Drop IGPRC miss packets */ +int aq_mss_set_drop_igprc_miss_packets(struct atl_hw *hw, bool drop); + +/*! Set Packet Edit Control register */ +int aq_mss_set_packet_edit_control(struct atl_hw *hw, u32 control); + +#endif /* __MACSEC_API_H__ */ diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_struct.h b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_struct.h new file mode 100644 index 000000000000..f211c074ec2d --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/macsec/macsec_struct.h @@ -0,0 +1,920 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Atlantic Network Driver + * + * Copyright (C) 2019 aQuantia Corporation + * Copyright (C) 2019-2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _MACSEC_STRUCT_H_ +#define _MACSEC_STRUCT_H_ + +/*! Represents the bitfields of a single row in the Egress CTL Filter + * table. + */ +struct aq_mss_egress_ctlf_record { + /*! This is used to store the 48 bit value used to compare SA, DA or + * halfDA+half SA value. + */ + u32 sa_da[2]; + /*! This is used to store the 16 bit ethertype value used for + * comparison. + */ + u32 eth_type; + /*! The match mask is per-nibble. 0 means don't care, i.e. every value + * will match successfully. The total data is 64 bit, i.e. 16 nibbles + * masks. + */ + u32 match_mask; + /*! 0: No compare, i.e. This entry is not used + * 1: compare DA only + * 2: compare SA only + * 3: compare half DA + half SA + * 4: compare ether type only + * 5: compare DA + ethertype + * 6: compare SA + ethertype + * 7: compare DA+ range. + */ + u32 match_type; + /*! 0: Bypass the remaining modules if matched. + * 1: Forward to next module for more classifications. + */ + u32 action; +}; + +/*! Represents the bitfields of a single row in the Egress Packet + * Classifier table. + */ +struct aq_mss_egress_class_record { + /*! VLAN ID field. */ + u32 vlan_id; + /*! VLAN UP field. */ + u32 vlan_up; + /*! VLAN Present in the Packet. */ + u32 vlan_valid; + /*! The 8 bit value used to compare with extracted value for byte 3. */ + u32 byte3; + /*! The 8 bit value used to compare with extracted value for byte 2. */ + u32 byte2; + /*! The 8 bit value used to compare with extracted value for byte 1. */ + u32 byte1; + /*! The 8 bit value used to compare with extracted value for byte 0. */ + u32 byte0; + /*! The 8 bit TCI field used to compare with extracted value. */ + u32 tci; + /*! The 64 bit SCI field in the SecTAG. */ + u32 sci[2]; + /*! The 16 bit Ethertype (in the clear) field used to compare with + * extracted value. + */ + u32 eth_type; + /*! This is to specify the 40bit SNAP header if the SNAP header's mask + * is enabled. + */ + u32 snap[2]; + /*! This is to specify the 24bit LLC header if the LLC header's mask is + * enabled. + */ + u32 llc; + /*! The 48 bit MAC_SA field used to compare with extracted value. */ + u32 mac_sa[2]; + /*! The 48 bit MAC_DA field used to compare with extracted value. */ + u32 mac_da[2]; + /*! The 32 bit Packet number used to compare with extracted value. */ + u32 pn; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte3_location; + /*! 0: don't care + * 1: enable comparison of extracted byte pointed by byte 3 location. + */ + u32 byte3_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte2_location; + /*! 0: don't care + * 1: enable comparison of extracted byte pointed by byte 2 location. + */ + u32 byte2_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte1_location; + /*! 0: don't care + * 1: enable comparison of extracted byte pointed by byte 1 location. + */ + u32 byte1_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte0_location; + /*! 0: don't care + * 1: enable comparison of extracted byte pointed by byte 0 location. + */ + u32 byte0_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of extracted VLAN ID field. + */ + u32 vlan_id_mask; + /*! 0: don't care + * 1: enable comparison of extracted VLAN UP field. + */ + u32 vlan_up_mask; + /*! 0: don't care + * 1: enable comparison of extracted VLAN Valid field. + */ + u32 vlan_valid_mask; + /*! This is bit mask to enable comparison the 8 bit TCI field, + * including the AN field. + * For explicit SECTAG, AN is hardware controlled. For sending + * packet w/ explicit SECTAG, rest of the TCI fields are directly + * from the SECTAG. + */ + u32 tci_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of SCI + * Note: If this field is not 0, this means the input packet's + * SECTAG is explicitly tagged and MACSEC module will only update + * the MSDU. + * PN number is hardware controlled. + */ + u32 sci_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of Ethertype. + */ + u32 eth_type_mask; + /*! Mask is per-byte. + * 0: don't care and no SNAP header exist. + * 1: compare the SNAP header. + * If this bit is set to 1, the extracted filed will assume the + * SNAP header exist as encapsulated in 802.3 (RFC 1042). I.E. the + * next 5 bytes after the the LLC header is SNAP header. + */ + u32 snap_mask; + /*! 0: don't care and no LLC header exist. + * 1: compare the LLC header. + * If this bit is set to 1, the extracted filed will assume the + * LLC header exist as encapsulated in 802.3 (RFC 1042). I.E. the + * next three bytes after the 802.3MAC header is LLC header. + */ + u32 llc_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of MAC_SA. + */ + u32 sa_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of MAC_DA. + */ + u32 da_mask; + /*! Mask is per-byte. */ + u32 pn_mask; + /*! Reserved. This bit should be always 0. */ + u32 eight02dot2; + /*! 1: For explicit sectag case use TCI_SC from table + * 0: use TCI_SC from explicit sectag. + */ + u32 tci_sc; + /*! 1: For explicit sectag case,use TCI_V,ES,SCB,E,C from table + * 0: use TCI_V,ES,SCB,E,C from explicit sectag. + */ + u32 tci_87543; + /*! 1: indicates that incoming packet has explicit sectag. */ + u32 exp_sectag_en; + /*! If packet matches and tagged as controlled-packet, this SC/SA + * index is used for later SC and SA table lookup. + */ + u32 sc_idx; + /*! This field is used to specify how many SA entries are + * associated with 1 SC entry. + * 2'b00: 1 SC has 4 SA. + * SC index is equivalent to {SC_Index[4:2], 1'b0}. + * SA index is equivalent to {SC_Index[4:2], SC entry's current AN[1:0] + * 2'b10: 1 SC has 2 SA. + * SC index is equivalent to SC_Index[4:1] + * SA index is equivalent to {SC_Index[4:1], SC entry's current AN[0]} + * 2'b11: 1 SC has 1 SA. No SC entry exists for the specific SA. + * SA index is equivalent to SC_Index[4:0] + * Note: if specified as 2'b11, hardware AN roll over is not + * supported. + */ + u32 sc_sa; + /*! 0: the packets will be sent to MAC FIFO + * 1: The packets will be sent to Debug/Loopback FIFO. + * If the above's action is drop, this bit has no meaning. + */ + u32 debug; + /*! 0: forward to remaining modules + * 1: bypass the next encryption modules. This packet is considered + * un-control packet. + * 2: drop + * 3: Reserved. + */ + u32 action; + /*! 0: Not valid entry. This entry is not used + * 1: valid entry. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Egress SC Lookup table. */ +struct aq_mss_egress_sc_record { + /*! This is to specify when the SC was first used. Set by HW. */ + u32 start_time; + /*! This is to specify when the SC was last used. Set by HW. */ + u32 stop_time; + /*! This is to specify which of the SA entries are used by current HW. + * Note: This value need to be set by SW after reset. It will be + * automatically updated by HW, if AN roll over is enabled. + */ + u32 curr_an; + /*! 0: Clear the SA Valid Bit after PN expiry. + * 1: Do not Clear the SA Valid bit after PN expiry of the current SA. + * When the Enable AN roll over is set, S/W does not need to + * program the new SA's and the H/W will automatically roll over + * between the SA's without session expiry. + * For normal operation, Enable AN Roll over will be set to '0' + * and in which case, the SW needs to program the new SA values + * after the current PN expires. + */ + u32 an_roll; + /*! This is the TCI field used if packet is not explicitly tagged. */ + u32 tci; + /*! This value indicates the offset where the decryption will start. + * [[Values of 0, 4, 8-50]. + */ + u32 enc_off; + /*! 0: Do not protect frames, all the packets will be forwarded + * unchanged. MIB counter (OutPktsUntagged) will be updated. + * 1: Protect. + */ + u32 protect; + /*! 0: when none of the SA related to SC has inUse set. + * 1: when either of the SA related to the SC has inUse set. + * This bit is set by HW. + */ + u32 recv; + /*! 0: H/W Clears this bit on the first use. + * 1: SW updates this entry, when programming the SC Table. + */ + u32 fresh; + /*! AES Key size + * 00 - 128bits + * 01 - 192bits + * 10 - 256bits + * 11 - Reserved. + */ + u32 sak_len; + /*! 0: Invalid SC + * 1: Valid SC. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Egress SA Lookup table. */ +struct aq_mss_egress_sa_record { + /*! This is to specify when the SC was first used. Set by HW. */ + u32 start_time; + /*! This is to specify when the SC was last used. Set by HW. */ + u32 stop_time; + /*! This is set by SW and updated by HW to store the Next PN number + * used for encryption. + */ + u32 next_pn; + /*! The Next_PN number is going to wrapped around from 0xFFFF_FFFF + * to 0. set by HW. + */ + u32 sat_pn; + /*! 0: This SA is in use. + * 1: This SA is Fresh and set by SW. + */ + u32 fresh; + /*! 0: Invalid SA + * 1: Valid SA. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Egress SA Key + * Lookup table. + */ +struct aq_mss_egress_sakey_record { + /*! Key for AES-GCM processing. */ + u32 key[8]; +}; + +/*! Represents the bitfields of a single row in the Ingress Pre-MACSec + * CTL Filter table. + */ +struct aq_mss_ingress_prectlf_record { + /*! This is used to store the 48 bit value used to compare SA, DA + * or halfDA+half SA value. + */ + u32 sa_da[2]; + /*! This is used to store the 16 bit ethertype value used for + * comparison. + */ + u32 eth_type; + /*! The match mask is per-nibble. 0 means don't care, i.e. every + * value will match successfully. The total data is 64 bit, i.e. + * 16 nibbles masks. + */ + u32 match_mask; + /*! 0: No compare, i.e. This entry is not used + * 1: compare DA only + * 2: compare SA only + * 3: compare half DA + half SA + * 4: compare ether type only + * 5: compare DA + ethertype + * 6: compare SA + ethertype + * 7: compare DA+ range. + */ + u32 match_type; + /*! 0: Bypass the remaining modules if matched. + * 1: Forward to next module for more classifications. + */ + u32 action; +}; + +/*! Represents the bitfields of a single row in the Ingress Pre-MACSec + * Packet Classifier table. + */ +struct aq_mss_ingress_preclass_record { + /*! The 64 bit SCI field used to compare with extracted value. + * Should have SCI value in case TCI[SCI_SEND] == 0. This will be + * used for ICV calculation. + */ + u32 sci[2]; + /*! The 8 bit TCI field used to compare with extracted value. */ + u32 tci; + /*! 8 bit encryption offset. */ + u32 encr_offset; + /*! The 16 bit Ethertype (in the clear) field used to compare with + * extracted value. + */ + u32 eth_type; + /*! This is to specify the 40bit SNAP header if the SNAP header's + * mask is enabled. + */ + u32 snap[2]; + /*! This is to specify the 24bit LLC header if the LLC header's + * mask is enabled. + */ + u32 llc; + /*! The 48 bit MAC_SA field used to compare with extracted value. */ + u32 mac_sa[2]; + /*! The 48 bit MAC_DA field used to compare with extracted value. */ + u32 mac_da[2]; + /*! 0: this is to compare with non-LPBK packet + * 1: this is to compare with LPBK packet. + * This value is used to compare with a controlled-tag which goes + * with the packet when looped back from Egress port. + */ + u32 lpbk_packet; + /*! The value of this bit mask will affects how the SC index and SA + * index created. + * 2'b00: 1 SC has 4 SA. + * SC index is equivalent to {SC_Index[4:2], 1'b0}. + * SA index is equivalent to {SC_Index[4:2], SECTAG's AN[1:0]} + * Here AN bits are not compared. + * 2'b10: 1 SC has 2 SA. + * SC index is equivalent to SC_Index[4:1] + * SA index is equivalent to {SC_Index[4:1], SECTAG's AN[0]} + * Compare AN[1] field only + * 2'b11: 1 SC has 1 SA. No SC entry exists for the specific SA. + * SA index is equivalent to SC_Index[4:0] + * AN[1:0] bits are compared. + * NOTE: This design is to supports different usage of AN. User + * can either ping-pong buffer 2 SA by using only the AN[0] bit. + * Or use 4 SA per SC by use AN[1:0] bits. Or even treat each SA + * as independent. i.e. AN[1:0] is just another matching pointer + * to select SA. + */ + u32 an_mask; + /*! This is bit mask to enable comparison the upper 6 bits TCI + * field, which does not include the AN field. + * 0: don't compare + * 1: enable comparison of the bits. + */ + u32 tci_mask; + /*! 0: don't care + * 1: enable comparison of SCI. + */ + u32 sci_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of Ethertype. + */ + u32 eth_type_mask; + /*! Mask is per-byte. + * 0: don't care and no SNAP header exist. + * 1: compare the SNAP header. + * If this bit is set to 1, the extracted filed will assume the + * SNAP header exist as encapsulated in 802.3 (RFC 1042). I.E. the + * next 5 bytes after the the LLC header is SNAP header. + */ + u32 snap_mask; + /*! Mask is per-byte. + * 0: don't care and no LLC header exist. + * 1: compare the LLC header. + * If this bit is set to 1, the extracted filed will assume the + * LLC header exist as encapsulated in 802.3 (RFC 1042). I.E. the + * next three bytes after the 802.3MAC header is LLC header. + */ + u32 llc_mask; + /*! Reserved. This bit should be always 0. */ + u32 _802_2_encapsulate; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of MAC_SA. + */ + u32 sa_mask; + /*! Mask is per-byte. + * 0: don't care + * 1: enable comparison of MAC_DA. + */ + u32 da_mask; + /*! 0: don't care + * 1: enable checking if this is loopback packet or not. + */ + u32 lpbk_mask; + /*! If packet matches and tagged as controlled-packet. This SC/SA + * index is used for later SC and SA table lookup. + */ + u32 sc_idx; + /*! 0: the packets will be sent to MAC FIFO + * 1: The packets will be sent to Debug/Loopback FIFO. + * If the above's action is drop. This bit has no meaning. + */ + u32 proc_dest; + /*! 0: Process: Forward to next two modules for 802.1AE decryption. + * 1: Process but keep SECTAG: Forward to next two modules for + * 802.1AE decryption but keep the MACSEC header with added error + * code information. ICV will be stripped for all control packets. + * 2: Bypass: Bypass the next two decryption modules but processed + * by post-classification. + * 3: Drop: drop this packet and update counts accordingly. + */ + u32 action; + /*! 0: This is a controlled-port packet if matched. + * 1: This is an uncontrolled-port packet if matched. + */ + u32 ctrl_unctrl; + /*! Use the SCI value from the Table if 'SC' bit of the input + * packet is not present. + */ + u32 sci_from_table; + /*! Reserved. */ + u32 reserved; + /*! 0: Not valid entry. This entry is not used + * 1: valid entry. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Ingress SC Lookup table. */ +struct aq_mss_ingress_sc_record { + /*! This is to specify when the SC was first used. Set by HW. */ + u32 stop_time; + /*! This is to specify when the SC was first used. Set by HW. */ + u32 start_time; + /*! 0: Strict + * 1: Check + * 2: Disabled. + */ + u32 validate_frames; + /*! 1: Replay control enabled. + * 0: replay control disabled. + */ + u32 replay_protect; + /*! This is to specify the window range for anti-replay. Default is 0. + * 0: is strict order enforcement. + */ + u32 anti_replay_window; + /*! 0: when none of the SA related to SC has inUse set. + * 1: when either of the SA related to the SC has inUse set. + * This bit is set by HW. + */ + u32 receiving; + /*! 0: when hardware processed the SC for the first time, it clears + * this bit + * 1: This bit is set by SW, when it sets up the SC. + */ + u32 fresh; + /*! 0: The AN number will not automatically roll over if Next_PN is + * saturated. + * 1: The AN number will automatically roll over if Next_PN is + * saturated. + * Rollover is valid only after expiry. Normal roll over between + * SA's should be normal process. + */ + u32 an_rol; + /*! Reserved. */ + u32 reserved; + /*! 0: Invalid SC + * 1: Valid SC. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Ingress SA Lookup table. */ +struct aq_mss_ingress_sa_record { + /*! This is to specify when the SC was first used. Set by HW. */ + u32 stop_time; + /*! This is to specify when the SC was first used. Set by HW. */ + u32 start_time; + /*! This is updated by HW to store the expected NextPN number for + * anti-replay. + */ + u32 next_pn; + /*! The Next_PN number is going to wrapped around from 0XFFFF_FFFF + * to 0. set by HW. + */ + u32 sat_nextpn; + /*! 0: This SA is not yet used. + * 1: This SA is inUse. + */ + u32 in_use; + /*! 0: when hardware processed the SC for the first time, it clears + * this timer + * 1: This bit is set by SW, when it sets up the SC. + */ + u32 fresh; + /*! Reserved. */ + u32 reserved; + /*! 0: Invalid SA. + * 1: Valid SA. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Ingress SA Key + * Lookup table. + */ +struct aq_mss_ingress_sakey_record { + /*! Key for AES-GCM processing. */ + u32 key[8]; + /*! AES key size + * 00 - 128bits + * 01 - 192bits + * 10 - 256bits + * 11 - reserved. + */ + u32 key_len; +}; + +/*! Represents the bitfields of a single row in the Ingress Post- + * MACSec Packet Classifier table. + */ +struct aq_mss_ingress_postclass_record { + /*! The 8 bit value used to compare with extracted value for byte 0. */ + u32 byte0; + /*! The 8 bit value used to compare with extracted value for byte 1. */ + u32 byte1; + /*! The 8 bit value used to compare with extracted value for byte 2. */ + u32 byte2; + /*! The 8 bit value used to compare with extracted value for byte 3. */ + u32 byte3; + /*! Ethertype in the packet. */ + u32 eth_type; + /*! Ether Type value > 1500 (0x5dc). */ + u32 eth_type_valid; + /*! VLAN ID after parsing. */ + u32 vlan_id; + /*! VLAN priority after parsing. */ + u32 vlan_up; + /*! Valid VLAN coding. */ + u32 vlan_valid; + /*! SA index. */ + u32 sai; + /*! SAI hit, i.e. controlled packet. */ + u32 sai_hit; + /*! Mask for payload ethertype field. */ + u32 eth_type_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte3_location; + /*! Mask for Byte Offset 3. */ + u32 byte3_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte2_location; + /*! Mask for Byte Offset 2. */ + u32 byte2_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte1_location; + /*! Mask for Byte Offset 1. */ + u32 byte1_mask; + /*! 0~63: byte location used extracted by packets comparator, which + * can be anything from the first 64 bytes of the MAC packets. + * This byte location counted from MAC' DA address. i.e. set to 0 + * will point to byte 0 of DA address. + */ + u32 byte0_location; + /*! Mask for Byte Offset 0. */ + u32 byte0_mask; + /*! Mask for Ethertype valid field. Indicates 802.3 vs. Other. */ + u32 eth_type_valid_mask; + /*! Mask for VLAN ID field. */ + u32 vlan_id_mask; + /*! Mask for VLAN UP field. */ + u32 vlan_up_mask; + /*! Mask for VLAN valid field. */ + u32 vlan_valid_mask; + /*! Mask for SAI. */ + u32 sai_mask; + /*! Mask for SAI_HIT. */ + u32 sai_hit_mask; + /*! Action if only first level matches and second level does not. + * 0: pass + * 1: drop (fail). + */ + u32 firstlevel_actions; + /*! Action if both first and second level matched. + * 0: pass + * 1: drop (fail). + */ + u32 secondlevel_actions; + /*! Reserved. */ + u32 reserved; + /*! 0: Not valid entry. This entry is not used + * 1: valid entry. + */ + u32 valid; +}; + +/*! Represents the bitfields of a single row in the Ingress Post- + * MACSec CTL Filter table. + */ +struct aq_mss_ingress_postctlf_record { + /*! This is used to store the 48 bit value used to compare SA, DA + * or halfDA+half SA value. + */ + u32 sa_da[2]; + /*! This is used to store the 16 bit ethertype value used for + * comparison. + */ + u32 eth_type; + /*! The match mask is per-nibble. 0 means don't care, i.e. every + * value will match successfully. The total data is 64 bit, i.e. + * 16 nibbles masks. + */ + u32 match_mask; + /*! 0: No compare, i.e. This entry is not used + * 1: compare DA only + * 2: compare SA only + * 3: compare half DA + half SA + * 4: compare ether type only + * 5: compare DA + ethertype + * 6: compare SA + ethertype + * 7: compare DA+ range. + */ + u32 match_type; + /*! 0: Bypass the remaining modules if matched. + * 1: Forward to next module for more classifications. + */ + u32 action; +}; + +/*! Represents the Egress MIB counters for a single SC. Counters are + * 64 bits, lower 32 bits in field[0]. + */ +struct aq_mss_egress_sc_counters { + /*! The number of integrity protected but not encrypted packets + * for this transmitting SC. + */ + u32 sc_protected_pkts[2]; + /*! The number of integrity protected and encrypted packets for + * this transmitting SC. + */ + u32 sc_encrypted_pkts[2]; + /*! The number of plain text octets that are integrity protected + * but not encrypted on the transmitting SC. + */ + u32 sc_protected_octets[2]; + /*! The number of plain text octets that are integrity protected + * and encrypted on the transmitting SC. + */ + u32 sc_encrypted_octets[2]; +}; + +/*! Represents the Egress MIB counters for a single SA. Counters are + * 64 bits, lower 32 bits in field[0]. + */ +struct aq_mss_egress_sa_counters { + /*! The number of dropped packets for this transmitting SA. */ + u32 sa_hit_drop_redirect[2]; + /*! TODO */ + u32 sa_protected2_pkts[2]; + /*! The number of integrity protected but not encrypted packets + * for this transmitting SA. + */ + u32 sa_protected_pkts[2]; + /*! The number of integrity protected and encrypted packets for + * this transmitting SA. + */ + u32 sa_encrypted_pkts[2]; +}; + +/*! Represents the common Egress MIB counters; the counter not + * associated with a particular SC/SA. Counters are 64 bits, lower 32 + * bits in field[0]. + */ +struct aq_mss_egress_common_counters { + /*! The number of transmitted packets classified as MAC_CTL packets. */ + u32 ctl_pkt[2]; + /*! The number of transmitted packets that did not match any rows + * in the Egress Packet Classifier table. + */ + u32 unknown_sa_pkts[2]; + /*! The number of transmitted packets where the SC table entry has + * protect=0 (so packets are forwarded unchanged). + */ + u32 untagged_pkts[2]; + /*! The number of transmitted packets discarded because the packet + * length is greater than the ifMtu of the Common Port interface. + */ + u32 too_long[2]; + /*! The number of transmitted packets for which table memory was + * affected by an ECC error during processing. + */ + u32 ecc_error_pkts[2]; + /*! The number of transmitted packets for where the matched row in + * the Egress Packet Classifier table has action=drop. + */ + u32 unctrl_hit_drop_redir[2]; +}; + +/*! Represents the Ingress MIB counters for a single SA. Counters are + * 64 bits, lower 32 bits in field[0]. + */ +struct aq_mss_ingress_sa_counters { + /*! For this SA, the number of received packets without a SecTAG. */ + u32 untagged_hit_pkts[2]; + /*! For this SA, the number of received packets that were dropped. */ + u32 ctrl_hit_drop_redir_pkts[2]; + /*! For this SA which is not currently in use, the number of + * received packets that have been discarded, and have either the + * packets encrypted or the matched row in the Ingress SC Lookup + * table has validate_frames=Strict. + */ + u32 not_using_sa[2]; + /*! For this SA which is not currently in use, the number of + * received, unencrypted, packets with the matched row in the + * Ingress SC Lookup table has validate_frames!=Strict. + */ + u32 unused_sa[2]; + /*! For this SA, the number discarded packets with the condition + * that the packets are not valid and one of the following + * conditions are true: either the matched row in the Ingress SC + * Lookup table has validate_frames=Strict or the packets + * encrypted. + */ + u32 not_valid_pkts[2]; + /*! For this SA, the number of packets with the condition that the + * packets are not valid and the matched row in the Ingress SC + * Lookup table has validate_frames=Check. + */ + u32 invalid_pkts[2]; + /*! For this SA, the number of validated packets. */ + u32 ok_pkts[2]; + /*! For this SC, the number of received packets that have been + * discarded with the condition: the matched row in the Ingress + * SC Lookup table has replay_protect=1 and the PN of the packet + * is lower than the lower bound replay check PN. + */ + u32 late_pkts[2]; + /*! For this SA, the number of packets with the condition that the + * PN of the packets is lower than the lower bound replay + * protection PN. + */ + u32 delayed_pkts[2]; + /*! For this SC, the number of packets with the following condition: + * - the matched row in the Ingress SC Lookup table has + * replay_protect=0 or + * - the matched row in the Ingress SC Lookup table has + * replay_protect=1 and the packet is not encrypted and the + * integrity check has failed or + * - the matched row in the Ingress SC Lookup table has + * replay_protect=1 and the packet is encrypted and integrity + * check has failed. + */ + u32 unchecked_pkts[2]; + /*! The number of octets of plaintext recovered from received + * packets that were integrity protected but not encrypted. + */ + u32 validated_octets[2]; + /*! The number of octets of plaintext recovered from received + * packets that were integrity protected and encrypted. + */ + u32 decrypted_octets[2]; +}; + +/*! Represents the common Ingress MIB counters; the counter not + * associated with a particular SA. Counters are 64 bits, lower 32 + * bits in field[0]. + */ +struct aq_mss_ingress_common_counters { + /*! The number of received packets classified as MAC_CTL packets. */ + u32 ctl_pkts[2]; + /*! The number of received packets with the MAC security tag + * (SecTAG), not matching any rows in the Ingress Pre-MACSec + * Packet Classifier table. + */ + u32 tagged_miss_pkts[2]; + /*! The number of received packets without the MAC security tag + * (SecTAG), not matching any rows in the Ingress Pre-MACSec + * Packet Classifier table. + */ + u32 untagged_miss_pkts[2]; + /*! The number of received packets discarded without the MAC + * security tag (SecTAG) and with the matched row in the Ingress + * SC Lookup table having validate_frames=Strict. + */ + u32 notag_pkts[2]; + /*! The number of received packets without the MAC security tag + * (SecTAG) and with the matched row in the Ingress SC Lookup + * table having validate_frames!=Strict. + */ + u32 untagged_pkts[2]; + /*! The number of received packets discarded with an invalid + * SecTAG or a zero value PN or an invalid ICV. + */ + u32 bad_tag_pkts[2]; + /*! The number of received packets discarded with unknown SCI + * information with the condition: + * the matched row in the Ingress SC Lookup table has + * validate_frames=Strict or the C bit in the SecTAG is set. + */ + u32 no_sci_pkts[2]; + /*! The number of received packets with unknown SCI with the condition: + * The matched row in the Ingress SC Lookup table has + * validate_frames!=Strict and the C bit in the SecTAG is not set. + */ + u32 unknown_sci_pkts[2]; + /*! The number of received packets by the controlled port service + * that passed the Ingress Post-MACSec Packet Classifier table + * check. + */ + u32 ctrl_prt_pass_pkts[2]; + /*! The number of received packets by the uncontrolled port + * service that passed the Ingress Post-MACSec Packet Classifier + * table check. + */ + u32 unctrl_prt_pass_pkts[2]; + /*! The number of received packets by the controlled port service + * that failed the Ingress Post-MACSec Packet Classifier table + * check. + */ + u32 ctrl_prt_fail_pkts[2]; + /*! The number of received packets by the uncontrolled port + * service that failed the Ingress Post-MACSec Packet Classifier + * table check. + */ + u32 unctrl_prt_fail_pkts[2]; + /*! The number of received packets discarded because the packet + * length is greater than the ifMtu of the Common Port interface. + */ + u32 too_long_pkts[2]; + /*! The number of received packets classified as MAC_CTL by the + * Ingress Post-MACSec CTL Filter table. + */ + u32 igpoc_ctl_pkts[2]; + /*! The number of received packets for which table memory was + * affected by an ECC error during processing. + */ + u32 ecc_error_pkts[2]; + /*! The number of received packets by the uncontrolled port + * service that were dropped. + */ + u32 unctrl_hit_drop_redir[2]; +}; + +#endif diff --git a/drivers/net/ethernet/aquantia/atlantic-fwd/release_notes.txt b/drivers/net/ethernet/aquantia/atlantic-fwd/release_notes.txt new file mode 100644 index 000000000000..9c2fe9380cb6 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic-fwd/release_notes.txt @@ -0,0 +1,352 @@ +Version 1.1.11 +============== +[ATLDRV-1567] - forwarding driver issue with adj_freq + +Version 1.1.10 +============== +[ATLDRV-1559] - Add support of icmp proto to the rx filters +[ATLDRV-1401] - A2: Filtering +[ATLDRV-1513] - PM runtime failure + +Version 1.1.9 +============= +[ATLDRV-1505] - FWD: New dev ids +[ATLDRV-1529] - FWD: Segmentation fault occurs for user operations +[ATLDRV-1542] - FWD: build fails on 5.8-rcX +[ATLDRV-1552] - Fwd: build fails with old gcc +[ATLDRV-1553] - Fwd: BUG message when LOCKDEP is enabled + +Version 1.1.8 +============= +[ATLDRV-1438] - Fwd: port PTP from Linux driver + +Version 1.1.7 +============= +[ATLDRV-1469] - AQ083: After suspend resume EAPOL frames is not received +[ATLDRV-1497] - Incorrect temperature is displayed in hwmon +[ATLDRV-1519] - AQ086: Configuring default ring size + +Version 1.1.6 +============= +[ATLDRV-1355] - Sporadically unable to set EEE +[ATLDRV-1407] - AQ078: Link comes back up after rmmod +[ATLDRV-1416] - Failed probe during stress remove/rescan +[ATLDRV-1435] - AQ080: MACSEC offload packet loss under concurrent streams +[ATLDRV-1436] - AQ079: non-MACSEC_Data packets received in macsec0 +[ATLDRV-1456] - FWD: BUG: KASAN: use-after-free in atl_remove+0x1c9/0x220 [atlantic_fwd] +[ATLDRV-1458] - AQ082: Accounting for RX bytes and packets +[ATLDRV-1441] - FWD: Run Flexera Code Insight +[ATLDRV-1457] - FWD: Linux-5.7.0 support + +Version 1.1.5 +============= +[ATLDRV-1382] - Failed install Driver 1.1.4 on Antigua +[ATLDRV-1384] - WOL does not work on Antigua + +Version 1.1.4 +============== +[ATLDRV-1371] - Add supported eee with 2.5G and 5G +[ATLDRV-1376] - Send notification when macsec offload turned on/off +[ATLDRV-1378] - check FW capability instead of FW version for MSM Apply Settings + +Version 1.1.3 +============== +[ATLDRV-1185] - Enable stripping of padding doesnt work on Felicity card +[ATLDRV-1351] - Problem with setting mtu +[ATLDRV-1352] - Implement stripping of padding +[ATLDRV-1354] - Incorrect thermal thresholds +[ATLDRV-1359] - Secure all packets when macsec service ON + +Version 1.1.2 +============== +[ATLDRV-1348] - FWD: IPv6 and L4 ntuple filters not work + +Version 1.1.1 +============== +[ATLDRV-1315] - driver 1.1.0 crashes at start on A1 +[ATLDRV-1317] - A2: Incorrect ntuple L4 filter behavior +[ATLDRV-1325] - A2: can not turn off flow control on Antigua +[ATLDRV-1326] - A2: Cannot get EEE settings on Antigua +[ATLDRV-1331] - A2: update reset sequence + +Version 1.1.0 +============== +[ATLDRV-1210] - A2: FWD: Basic functionality and bringup +[ATLDRV-1212] - A2: FWD: RX flow filters infrastructure + + +Version 1.0.31 +============== +[ATLDRV-1243] - Clean up MACsec implementation +[ATLDRV-1135] - ethtool shows incorrect Flow Control information +[ATLDRV-1229] - HW access on mdo_dev_stop(ctx->prepare) +[ATLDRV-1256] - AQ072: Static analysis error in atl_setup_datapath() +[ATLDRV-1303] - Make driver C90 compilable +[ATLDRV-1064] - Fwd: Software flowcontrol workaround adaptations + +Version 1.0.30 +============== +[ATLDRV-1217] - FWD driver refactoring for unit tests +[ATLDRV-1102] - TCP/UDP checksum workaround for non-zero padded packets doesn't work +[ATLDRV-1138] - vlan promiscuous mode is not enabled, when deleting the last vlan interface +[ATLDRV-1139] - after ifconfig -multicast Unicast filters registers have valid values for multicast addresses +[ATLDRV-1174] - FWD: deleting MacSec iface doesn't clear Ingress/Egress SA and SAKey records +[ATLDRV-1193] - No link UP if driver was started in MSI mode before being started in legacy mode +[ATLDRV-1208] - Driver crashes after link down up in D3 state +[ATLDRV-1209] - FWD: Kernel panic after configuring macsec +[ATLDRV-1218] - Kernel panic upon macsec0 deletion +[ATLDRV-1220] - AQ069: atl_fwd_receive_skb() support NAPI +[ATLDRV-1222] - atl_rotate_keys() produces unexpected results for 192-bit keys (24-bytes) +[ATLDRV-1194] - Add trace of descriptors for normal traffic + +Version 1.0.29 +============== +[ATLDRV-1139] - after ifconfig -multicast Unicast filters registers have valid values for multicast addresses +[ATLDRV-1170] - macsec key cleaned on SA updates +[ATLDRV-1171] - AQ066: MSI index leak for head pointer writeback event +[ATLDRV-1172] - AQ068: shift-count-overflow error in DESCR_FIELD() +[ATLDRV-1187] - atlfwdtool not compiled on Centos +[ATLDRV-1186] - FWD: MACsec offload (cleanup) + +Version 1.0.28 +============== +[ATLDRV-1128] - Traffic ingress on FWD ring +[ATLDRV-1157] - Apply upstream patch for macsec offload to 4.14 +[ATLDRV-1158] - Implement Macsec EGRESS offload +[ATLDRV-1159] - Implement Macsec INGRESS offload +[ATLDRV-1160] - Provide macsec statistics to ethtool -s +[ATLDRV-1161] - Add get_tx_queue_index command to atlfwdtool +[ATLDRV-1153] - traffic dropped when lro enabled +[ATLDRV-777] - FWD: BCM: Better support for systems with legacy interrupts + +Version 1.0.27 +============== +[ATLDRV-1132] - Media detect is disabled after sleep +[ATLDRV-1129] - Add support for egress FWD queue pause/resume + +Version 1.0.26 +============== +[ATLDRV-660] - Wrong device ids are present in atl_main.c +[ATLDRV-1113] - VLAN-PROMISC is turned off when PROMISC flag is set +[ATLDRV-1124] - link is not up at 2.5G speed +[ATLDRV-1105] - Allocate/destroy/disable/enable channels +[ATLDRV-1114] - Send traffic via FWD ring +[ATLDRV-1116] - Add support for FWD rings status / stats via ethtool +[ATLDRV-1120] - reproduce descriptor writeback on disabled ring +[ATLDRV-1126] - Print statistics for created FWD rings only + +Version 1.0.25 +============== +[ATLDRV-1118] - Page fault in atl_fwd_init_ring() upon ring creation (ATL_FWR_ALLOC_BUFS is not set) +[ATLDRV-1119] - AQ058: again Bug CR 2491137 - Crash caused due to 32 Tx descs are zero'ed + +Version 1.0.24 +============== +[ATLDRV-1112] - descriptor corruption after suspend/resume +[ATLDRV-1097] - EEE must be disabled by-default +[ATLDRV-1099] - Strange EEE configuration behavior +[ATLDRV-1107] - Strange message in dmesg +[ATLDRV-1108] - FW not loaded after FWD driver unloaded on B0 +[ATLDRV-899] - Fwd driver: add more loopbacks to ethtool private flags +[ATLDRV-1057] - Support for network loopback + +Version 1.0.23 +============== +[ATLDRV-983] - runtime D3 support +[ATLDRV-1023] - AQ056: Wake on link race condition with host power manament + +Version 1.0.22 +============== +fix build of fwd api + +Version 1.0.21 +============== +[ATLDRV-871] - pause frame does not turn off +[ATLDRV-994] - failed set StripEtherPadding +[ATLDRV-1062] - FWD: IPA: AQ058: Crash caused due to 32 Tx descs are zero'ed +[ATLDRV-1066] - AQ062: atl_update_eth_stats() - sleeping function called from invalid context +[ATLDRV-1067] - AQ061: AQC FWD ring descriptor memory reset +[ATLDRV-1068] - AQ060: AQC Rx accessing buffer address from previous atl_fwd_request_ring() +[ATLDRV-1075] - AQ063: v1.0.20 driver - atl_update_thermal_flag() missing atl_unlock_fw() +[ATLDRV-803] - AQ040: Validation for Interrupt Moderation in FWD API +[ATLDRV-1022] - Media detect control + + +Version 1.0.20 +============== +net: aquantia: Re-initialize fwd rings on driver resume +Do entire suspend from dev_pm_ops->suspend() (AQ052) +Prevent atl_update_global_stats() calls when suspended (AQ056) (ATLDRV-1021) +Implement thermal throttling (ATLDRV-1009) +Add log message when high temperature treshold is reached (ATLDRV-998)) +Implement FW heartbeat watchdog (AQ010) (ATLDRV-502) +Workaround checksum offload bugs (ATLDRV-1013) + +Version 1.0.19 +============== +Fix freeing of buf vaddr vector in FWD API (AQ050) +Add a private data pointer to atl_fwd_mem_ops (AQ051) +Implement wake-on-link-up support (AQ049) +Introduce the forwarding engine API versioning (AQ048) +Fix build errors (AQ047) +Delay enabling bus-mastering until after reset sequence (AQ045) +Validate interrupt moderation delays in FWD API (AQ040) + +Version 1.0.18 +============== +Fix double free in the new fwd ring memory allocation code. +Fix 32-bit warnings (AQ036) +Add a pointer for forwarding engine's private data. (AQ043) + +Version 1.0.17 +============== +Add compatibility to kernel 4.1 +create empty atl_hwmon_init for old kernels +Fix legacy interrupt handling +Fix MSM register writes +Free link interrupt before reconfiguring the rings +Implement FW settings table access +Use new FW interface to control ethernet padding stripping +Disable LRO by default until all issues are resolved. +Clamp max number of rings as reported by ethtool to num_present_cpus() (AQ037) +Use __GFP_NOWARN for Rx buffers only in NAPI poll (AQ037) +Put PHY into low-power state on ifdown or disabling link +Improve error reporting for forwarding-engine ring creation (AQ041) +Allow custom memory allocators for atl_fwd_request_ring() (AQ041) + +Version 1.0.16 +============== +Introduce the config options for forwarding API (AQ033) +Implement catchall ethtool filter (AQ034) +Fix MSI-X vectors being disabled on cpu unplug (AQ032) +Fix Kconfig changes for AQ033 + +Version 1.0.15 +============== +Reset statistics via private flag +Implement interrupt moderation for offload engine's rings +Introduce a FW mutex, handle link interrupts in a workqueue +Implement PHY temperature sensor +Report ring's and buffer's physical addresses on fwd ring request +Implement ethtool control for stripping Rx ethernet frame's padding + +Version 1.0.14 +============== +Remove old IPA implementation (replaced by atl_fwd.[ch]) +Fix forwarding engine interface +Change the driver name to reflect the new module name +Implement Linux PM support +Implement WoL support + +Version 1.0.13 +============== +Integrate Linux VLAN filter hw acceleration with ethtool filters +Rename the module to atlantic-fwd + +Version 1.0.12 +============== +Expose PHY LPI state via ethtool private flags + +Version 1.0.11 +============== +Minor IPA interface fixes +Turn on/off Rx XOFF flow control when needed +Fix advertizing EEE link modes and show LPI timer +Add some HW counters from RMON MIB +Add Host and Lost counters to statistics +Add more Felicity PCI IDs +Add ethtool private flags tracking LPI state + +Version 1.0.10 +============== +Add module parameter to force align IP headers + +Version 1.0.9 +============== +Add AQC100 / -111 / -112 IDs +Fix adding vlan with vlan id > 64 +Fix ethtool IPv4 filter reporting (AQ022) +Implement ethtool interface for ethertype filters. +Restore backward compat for Linux 4.4 + +Version 1.0.8 +============== +Fix checkpatch.pl and smatch warnings and errors + +Version 1.0.7 +============== +Make Tx ring free space watermarks configurable +Misc filter fixes +Implement ethtool interface for VLAN filters +Implement pause statistics + +Version 1.0.6 +============== +Implement compat fixes for kernels without IPv6 ethtool NFC API + +Version 1.0.5 +============== +Fix ifdown / ifup link state race +Fix autoneg re-enable on pre-ethtool_ksettings kernels +Implement n-tuple Rx filters +Misc clean-ups + +Version 1.0.4 +============== +Switch to using spin lock for updating global stats +Fix out-of-bounds dma_sync_single_range() for packet header +Fix an Rx error corner case +Refactor delayed dma-unmapping of headers for RSC +Implement HW loopbacks +Implement interrupt coalescing ethtool method + +Version 1.0.3 +============== +Fix crash in mutex_lock on init +Add compat cast for ->ndo_get_stats64 for old kernels +Get rid of struct atl_common_stats +Refactor ring structure to remove the need for -fplan9-extensions +Drop -fplan9-extensions + +Version 1.0.2 +============== +Implement setting the MAC address +Rework VLAN filters +Implement per-ring counters +Fix Tx queue start / stop handling +Implement basic netdev stats +Implement ethtool stats + +Version 1.0.1 +============== +Update the README +Refactor the ring structure +Fix LRO timers timebase +Initial Qualcomm IPA support +Refactor Link management +Implement Flow Control +Implement EEE +Set default link on init +Implement dynamic datapath reconfiguration +Implement ethtool -> [gs]et_ringparam, set_channels + +Version 1.0.0 +============== +Add dma-mapping tracepoints +Implement VLAN offloads +Implement promiscuous mode and UC / MC filters +Implement HW header splitting, fix Rx buffer mgmt for RSC +Fix multicast filter support +Implement new reset sequence +Implement multi-fw switch +Implement new link management +Introduce link interrupt +Implement ethtool set_settings and _ksettings variants +Implement multiring support +Implement RSS support +Enable jumbo frames +Fix ethtool RSS methods +Add RDM / TDM descriptor thresholds +Switch to INTR block based interrupt throttling +Enable LRO +Implement linear skb mode From 1a1d611aff95e282740834388e9b504cdec74d0c Mon Sep 17 00:00:00 2001 From: Lakshit Tyagi Date: Mon, 28 Sep 2020 15:01:25 +0530 Subject: [PATCH 2/2] net: aquantia: support for 32-bit ARM target Add support to build the Aquantia forwarding driver for 32-bit ARM target. Change-Id: I153e85e8808895c6979e5597bcea09f0e2e22914 Signed-off-by: Lakshit Tyagi --- drivers/net/ethernet/aquantia/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/aquantia/Kconfig b/drivers/net/ethernet/aquantia/Kconfig index 32f1e8dc82c4..58a1cf46fff2 100644 --- a/drivers/net/ethernet/aquantia/Kconfig +++ b/drivers/net/ethernet/aquantia/Kconfig @@ -25,7 +25,7 @@ config AQTION config AQFWD tristate "aQuantia Forwarding driver" - depends on PCI && (X86_64 || ARM64) + depends on PCI && (X86_64 || ARM64 || ARM ) ---help--- This enables the support for forwarding driver for the aQuantia AQtion(tm) Ethernet card.