[Openvpn-devel,net-next,v2,7/9] ovpn: prefetch encrypted records before GRO batch decryption
Commit Message
Frag-list GRO exposes later encrypted records before the receive path decrypts
the current one. Use this lookahead to request write ownership of linear
ciphertext cache lines two records in advance (and maintain the same distance
throughout the batch), overlapping their memory access latency with the
current AEAD operation. An ordinary single-record UDP receive has no later
record and therefore skips the prefetch path.
Only prefetch the linear part of each skb. Walking non-linear fragments here
would duplicate the scatterlist walk performed by crypto and could cost more
than the cache hint saves.
This optimization primarily targets synchronous software AEAD, where the CPU
which issues the hint subsequently overwrites the prefetched lines. An
asynchronous accelerator may instead access them through DMA and is not
expected to benefit in the same way; the cache hint remains
correctness-neutral.
A same-binary comparison using three 30-second samples per direction found
distances one and two effectively tied forward, while distance two was 3.3%
faster reverse and less variable in both directions. Profiling also measured
slightly fewer decrypt cycles at distance two than at one, while wider
distances provided no repeatable benefit.
On two directly connected 100-Gbit/s mlx5 ports, five interleaved
iperf3 -t 60 -O 10 single-flow AES-128-GCM runs in each direction produced
the following mean throughput:
Forward Reverse
Without RX prefetch 22.773 Gbit/s 22.796 Gbit/s
With RX prefetch 24.028 Gbit/s 23.405 Gbit/s
RX prefetch improved throughput by 5.5% forward and 2.7% reverse.
Signed-off-by: Ralf Lici <ralf@mandelbit.com>
---
Changes since v1 https://lore.kernel.org/openvpn-devel/0651258e11d4a0c290152a1f286b9d8a45c5acd4.1790843153.git.ralf@mandelbit.com/
- Document that RX prefetch primarily targets synchronous software AEAD;
asynchronous transforms are not expected to benefit in the same way.
drivers/net/ovpn/io.h | 15 +++++++++++++++
drivers/net/ovpn/udp.c | 21 ++++++++++++++++++++-
2 files changed, 35 insertions(+), 1 deletion(-)
@@ -10,6 +10,9 @@
#ifndef _NET_OVPN_OVPN_H_
#define _NET_OVPN_OVPN_H_
+#include <linux/cache.h>
+#include <linux/prefetch.h>
+
/* DATA_V2 header size with AEAD encryption */
#define OVPN_HEAD_ROOM (OVPN_DATA_V2_OVERHEAD + \
max(sizeof(struct udphdr), sizeof(struct tcphdr)) +\
@@ -21,6 +24,18 @@
#define OVPN_KEEPALIVE_SIZE 16
extern const unsigned char ovpn_keepalive_message[OVPN_KEEPALIVE_SIZE];
+static inline void ovpn_skb_prefetchw_linear(const struct sk_buff *skb)
+{
+ unsigned long addr = (unsigned long)skb->data;
+ unsigned long end = addr + skb_headlen(skb);
+
+ /* crypto overwrites data in place, so request write ownership of each
+ * linear cache line before the AEAD implementation reaches it
+ */
+ for (; addr < end; addr = ALIGN(addr + 1, L1_CACHE_BYTES))
+ prefetchw((void *)addr);
+}
+
netdev_tx_t ovpn_net_xmit(struct sk_buff *skb, struct net_device *dev);
void ovpn_recv(struct ovpn_peer *peer, struct sk_buff *skb);
@@ -32,6 +32,9 @@
/* like UDP and TCP frag-list GRO */
#define OVPN_UDP_GRO_CNT_MAX 64
+/* leave enough work between a cache hint and the record which consumes it */
+#define OVPN_UDP_GRO_PREFETCH_DISTANCE 2
+
static bool ovpn_udp_gro_header(struct sk_buff *skb, u32 *header)
{
const unsigned int offset = skb_gro_offset(skb);
@@ -189,7 +192,8 @@ static struct sk_buff *ovpn_udp_gro_detach(struct sk_buff *skb)
static void ovpn_udp_recv(struct ovpn_peer *peer, struct sk_buff *skb)
{
- struct sk_buff *list, *next;
+ struct sk_buff *list, *next, *prefetch;
+ unsigned int i;
list = ovpn_udp_gro_detach(skb);
if (IS_ERR(list)) {
@@ -200,8 +204,23 @@ static void ovpn_udp_recv(struct ovpn_peer *peer, struct sk_buff *skb)
}
skb->next = list;
+ /* a fraglist GRO aggregate makes later ciphertext visible before the
+ * current record is decrypted, so we prime the first two records, then
+ * keep the cache hints the same distance ahead while draining the list
+ */
+ prefetch = skb->next ? skb : NULL;
+ for (i = 0; i < OVPN_UDP_GRO_PREFETCH_DISTANCE && prefetch; i++) {
+ ovpn_skb_prefetchw_linear(prefetch);
+ prefetch = prefetch->next;
+ }
+
skb_list_walk_safe(skb, skb, next)
{
+ if (prefetch) {
+ ovpn_skb_prefetchw_linear(prefetch);
+ prefetch = prefetch->next;
+ }
+
/* skb_unclone can leave fraglist children shared with a packet
* tap, so make each skb header private before changing its
* list pointer or control block. skb_cow_data makes the packet