From patchwork Thu Oct 8 14:13:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Baffo X-Patchwork-Id: 5446 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7000:32d1:b0:8d1:cccb:4552 with SMTP id y17csp1747595mad; Thu, 8 Oct 2026 07:14:06 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvBz2tfNTNUPMceLwXeQE/CJalMrfl/UXAjhMIojOBO6eDvp2kuZmh8FHkzRQVc8+lfYFVbiE8QVeeFI=@openvpn.net X-Received: by 2002:a05:6808:16a1:b0:4b2:8e1c:5920 with SMTP id 5614622812f47-4fc436002d2mr4997706b6e.9.1791468845880; Thu, 08 Oct 2026 07:14:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1791468845; cv=none; d=google.com; s=arc-20260327; b=D+Mq1hd1rvKibjgcBakWNz08vdrwzWB6DoxBPeJW8s6P06Dtjr5OUwABqEixpy8yuL /hS8OhLwHW4/SC9CrOjS82x7duiwq8KxkRLNejJZRjV5Ero6W9DZG0AcMbX8krBmtNtQ w05Y1RsF6OnWLWRXbz2K6FtgQxcxI922Q3W8noi9/XZ4yL6Ty+mEyp8O5tuyIZ3jOFbZ VBEQgLyOJIf/0QC3nvoxPifP0QLIVhVk3ZR4iY7nphi/yZHQn5uoHGAUNQaLz+OnBKeF BRy+X2d8zfvfSnkQ2Qs1ANUM7gUppkeG3kPap2t5t0LyehkN8gSNvcf5xoMBtWPHDkmE aPhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=hDZoMNbCi7PabcfTWLS7KpGPPB4ihnyHcKIRwZxyFGE=; fh=4NbAC/LsuMLI0S0hprUlLSLCiHwg6SCAifhH718Jh0Q=; b=k20ip46rU+/ZdqMT24s1eV3uWC7drE0YhRy99BHep6laink5D2G2n/HdsoSX8djQQ0 4gLn8RABaPCmoR/Uury2biNJYj2PDLe6yqBPdNfvEa71BI/1fSaNAMY9pJNyNAVvGXfI wg4czkxAEZnoK7jzii1Rdb4GBGWFHu6UcyK/8XS7KYVbOC1Fa/MBTiHPKk9Kx2WcTnUG jY2H2k8iEhqqFnzKh7Xr4OhEwtAbsOv5HcmQOsXtmdTNraXVQmOiWEuhwlWxdoai92qA Vq8RESNUrLsJEmHRZH1H8vFPv6r0cFzDCnxE2lkxbYuxWRWnzW69iodqfIDQbt5L+oyh 5Ysg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=lwXv5LmC; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=En3oqMEm; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=gEQ8dtQo; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=Zav93gDH; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id 5614622812f47-4fc4a6cf605si8004667b6e.118.2026.10.08.07.14.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Oct 2026 07:14:05 -0700 (PDT) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=lwXv5LmC; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=En3oqMEm; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=gEQ8dtQo; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=Zav93gDH; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Content-Transfer-Encoding:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Subject:MIME-Version:References:In-Reply-To:Message-ID:Date:To:From:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=hDZoMNbCi7PabcfTWLS7KpGPPB4ihnyHcKIRwZxyFGE=; b=lwXv5LmC3SFq4060e+tz4vIp1Z O7MhQyn3GVKpNjWOBgPmMKF94YkuQLdifYpCWXO6VGiHsVg+GGOps8OcnBEM60wFswJOXMGYVGkh8 NPe712twBZIp81q3o28TIOOK7WxutL8Vk0fjf4/0JDSgvAeBHr6Bn7zJjfgnnwXZybgc=; Received: from [127.0.0.1] (helo=sfs-ml-3.v29.lw.sourceforge.com) by sfs-ml-3.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1xEosi-00051o-UF; Thu, 08 Oct 2026 14:14:01 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-3.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1xEosh-00051W-M1 for openvpn-devel@lists.sourceforge.net; Thu, 08 Oct 2026 14:14:00 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=w2Y9pEmzjEqLQFOTPpt3DmxGNsFcoFTGNpEVo1k9cbY=; b=En3oqMEmHe5YQb6ZEG0YsngiZc oovvHGRAxej8BfSwxCGnLBNiRhEbKBIGBRcAZSlrvMn4dgtnVD8N8lmWw7r/1e0WCqXO7NfZbcfpL e4cAkM/myp0gToms/CKcI6dmst0TQNUg/3e/ZUFSmvd7eY39uHUWtSl2mMy6OAZQ7Ut0=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=w2Y9pEmzjEqLQFOTPpt3DmxGNsFcoFTGNpEVo1k9cbY=; b=gEQ8dtQoiMfGhT7CVhK85CdVlF SkX6ybQ4zWp5dA8lsdhzXeE7Pug5sB/eunAYw35wdSCoQii+AtkzOk0BOKG3FjLqWOp6otsWZPC9W 4QuZESUupSzGh8OjTFuXM72GgENtt65TKGbLADVCOwgU36hh6cfy9rLutWdewdmRt5OI=; Received: from mout-b-202.mailbox.org ([195.10.208.62]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1xEosg-0003eK-Ok for openvpn-devel@lists.sourceforge.net; Thu, 08 Oct 2026 14:14:00 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:b231:465::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519MLKEM768 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-b-202.mailbox.org (Postfix) with ESMTPS id 4j0sRf5Y02zKmHL; Thu, 08 Oct 2026 16:13:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandelbit.com; s=MBO0001; t=1791468830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w2Y9pEmzjEqLQFOTPpt3DmxGNsFcoFTGNpEVo1k9cbY=; b=Zav93gDHHnIHjYlOsBQjcoV+zftDd3nzmcei4aOuEbKYFbQE/310dfXzKCl0Be2T//QvJm 28yaGDuVyyRzNjF7J0SDuAws8EI1bISTyzWuiEpNMzLkuZcgaK88Arvfw0Zt292oslpKDd e4LNM9qDhrLYxxANSjOOquf0V8PE2T8aYIijqAL5fwIt8UuQEPmnclpo7G8JjNUjP7Y0+P qfp2GbzooN2f7jichhnilLQEcijryPKBO1ovvA+6MyJtCRv+5ziL8UWPKI2GyHLm+MXo4k aB5xba7NHzaiPB3vNfbV2OgARrC+qztS3+u/utQMzOgHWEl/SCDQ6D8D1xGD1Q== Authentication-Results: outgoing_mbo_mout; dkim=none; spf=pass (outgoing_mbo_mout: domain of marco@mandelbit.com designates 2001:67c:2050:b231:465::1 as permitted sender) smtp.mailfrom=marco@mandelbit.com From: Marco Baffo To: openvpn-devel@lists.sourceforge.net Date: Thu, 8 Oct 2026 16:13:43 +0200 Message-ID: <20261008141343.553249-4-marco@mandelbit.com> In-Reply-To: <20261008141343.553249-1-marco@mandelbit.com> References: <20261008141343.553249-1-marco@mandelbit.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4j0sRf5Y02zKmHL X-Spam-Score: -0.2 (/) X-Spam-Report: Spam detection software, running on the system "sfi-spamd-1.hosts.colo.sdot.me", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Per-peer VPN and transport counters use atomic64 operations on every packet, causing cacheline contention when CPUs share a peer. Replace these counters with pcpu_sw_netstats and aggregate them when reporting peers over netlink. Keep the statistics pointers in a separate cacheline to avoid false sharing with frequently updated p [...] Content analysis details: (-0.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-Headers-End: 1xEosg-0003eK-Ok Subject: [Openvpn-devel] [PATCH ovpn net-next 3/3] ovpn: use per-CPU peer statistics X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: 1878491236402238817 X-GMAIL-MSGID: 1878491236402238817 Per-peer VPN and transport counters use atomic64 operations on every packet, causing cacheline contention when CPUs share a peer. Replace these counters with pcpu_sw_netstats and aggregate them when reporting peers over netlink. Keep the statistics pointers in a separate cacheline to avoid false sharing with frequently updated peer fields. Disable BHs around the link RX update to cover TCP strparser callbacks running in process context. Measurements on a laptop with a Ryzen 7 PRO 7840U, with four concurrent iperf3 streams on separate cores using the same ovpn peer, showed statistics HITM samples dropping from 8-12 to zero for UDP and 16-18 to zero for TCP. UDP cycles per packet fell from 26,389 to 26,339 (-0.19%) and TCP cycles per segment fell from 18,920 to 18,861 (-0.31%). The peer and key-slot cacheline layout changes reduce total HITM samples by up to 75% for UDP and up to 63% for TCP. Signed-off-by: Marco Baffo --- drivers/net/ovpn/Makefile | 1 - drivers/net/ovpn/crypto.h | 11 ++++++- drivers/net/ovpn/io.c | 8 ++--- drivers/net/ovpn/netlink.c | 20 +++++++----- drivers/net/ovpn/peer.c | 26 ++++++++++++---- drivers/net/ovpn/peer.h | 22 +++++++++---- drivers/net/ovpn/stats.c | 21 ------------- drivers/net/ovpn/stats.h | 63 ++++++++++++++++++++++---------------- drivers/net/ovpn/tcp.c | 3 +- 9 files changed, 101 insertions(+), 74 deletions(-) delete mode 100644 drivers/net/ovpn/stats.c diff --git a/drivers/net/ovpn/Makefile b/drivers/net/ovpn/Makefile index 229be66167e1f..ebea3a24336f7 100644 --- a/drivers/net/ovpn/Makefile +++ b/drivers/net/ovpn/Makefile @@ -17,6 +17,5 @@ ovpn-y += netlink-gen.o ovpn-y += peer.o ovpn-y += pktid.o ovpn-y += socket.o -ovpn-y += stats.o ovpn-y += tcp.o ovpn-y += udp.o diff --git a/drivers/net/ovpn/crypto.h b/drivers/net/ovpn/crypto.h index a0209b1280de4..a7351cae3f068 100644 --- a/drivers/net/ovpn/crypto.h +++ b/drivers/net/ovpn/crypto.h @@ -46,14 +46,23 @@ struct ovpn_crypto_key_slot { struct crypto_aead *decrypt; u8 nonce_tail_xmit[OVPN_NONCE_TAIL_SIZE]; u8 nonce_tail_recv[OVPN_NONCE_TAIL_SIZE]; + struct percpu_ref refcount; + + /* The fields up to refcount are read on every packet but written only when + * the key is installed or torn down, so they share the first cacheline. + * pid_recv and pid_xmit are written on every packet and get cachelines of + * their own. Keep per-packet writes out of the first block. + */ struct ovpn_pktid_recv pid_recv ____cacheline_aligned_in_smp; struct ovpn_pktid_xmit pid_xmit ____cacheline_aligned_in_smp; struct work_struct free_work; unsigned long rcu_state; - struct percpu_ref refcount; }; +static_assert(offsetofend(struct ovpn_crypto_key_slot, refcount) <= + SMP_CACHE_BYTES); + struct ovpn_crypto_state { struct ovpn_crypto_key_slot __rcu *slots[2]; u8 primary_idx; diff --git a/drivers/net/ovpn/io.c b/drivers/net/ovpn/io.c index 15072ce960fa4..75c869ffc5d75 100644 --- a/drivers/net/ovpn/io.c +++ b/drivers/net/ovpn/io.c @@ -99,7 +99,7 @@ static void ovpn_netdev_write(struct ovpn_peer *peer, struct sk_buff *skb) ret = gro_cells_receive(&peer->ovpn->gro_cells, skb); if (likely(ret == NET_RX_SUCCESS)) { /* update RX stats with the size of decrypted packet */ - ovpn_peer_stats_increment_rx(&peer->vpn_stats, pkt_len); + ovpn_peer_stats_increment_rx(peer->vpn_stats, pkt_len); dev_dstats_rx_add(peer->ovpn->dev, pkt_len); } local_bh_enable(); @@ -216,7 +216,7 @@ void ovpn_recv(struct ovpn_peer *peer, struct sk_buff *skb) struct ovpn_crypto_key_slot *ks; u8 key_id; - ovpn_peer_stats_increment_rx(&peer->link_stats, skb->len); + ovpn_peer_stats_increment_rx(peer->link_stats, skb->len); /* get the key slot matching the key ID in the received packet */ key_id = ovpn_key_id_from_skb(skb); @@ -292,7 +292,7 @@ void ovpn_encrypt_post(void *data, int ret) goto err_unlock; } - ovpn_peer_stats_increment_tx(&peer->link_stats, orig_len); + ovpn_peer_stats_increment_tx(peer->link_stats, orig_len); /* keep track of last sent packet for keepalive */ WRITE_ONCE(peer->last_sent, ktime_get_boottime_seconds()); /* skb passed down the stack - don't free it */ @@ -428,7 +428,7 @@ netdev_tx_t ovpn_net_xmit(struct sk_buff *skb, struct net_device *dev) } skb_list.prev->next = NULL; - ovpn_peer_stats_increment_tx(&peer->vpn_stats, tx_bytes); + ovpn_peer_stats_increment_tx(peer->vpn_stats, tx_bytes); ovpn_send(ovpn, skb_list.next, peer); return NETDEV_TX_OK; diff --git a/drivers/net/ovpn/netlink.c b/drivers/net/ovpn/netlink.c index 8cdf46b611425..10a277a0f96af 100644 --- a/drivers/net/ovpn/netlink.c +++ b/drivers/net/ovpn/netlink.c @@ -550,6 +550,7 @@ static int ovpn_nl_send_peer(struct sk_buff *skb, const struct genl_info *info, const struct ovpn_peer *peer, u32 portid, u32 seq, int flags) { + struct rtnl_link_stats64 vpn_stats = {}, link_stats = {}; const struct ovpn_bind *bind; struct ovpn_socket *sock; int ret = -EMSGSIZE; @@ -631,27 +632,30 @@ static int ovpn_nl_send_peer(struct sk_buff *skb, const struct genl_info *info, } rcu_read_unlock(); + dev_fetch_sw_netstats(&vpn_stats, peer->vpn_stats); + dev_fetch_sw_netstats(&link_stats, peer->link_stats); + if (nla_put_net16(skb, OVPN_A_PEER_LOCAL_PORT, local_port) || /* VPN RX stats */ nla_put_uint(skb, OVPN_A_PEER_VPN_RX_BYTES, - atomic64_read(&peer->vpn_stats.rx.bytes)) || + vpn_stats.rx_bytes) || nla_put_uint(skb, OVPN_A_PEER_VPN_RX_PACKETS, - atomic64_read(&peer->vpn_stats.rx.packets)) || + vpn_stats.rx_packets) || /* VPN TX stats */ nla_put_uint(skb, OVPN_A_PEER_VPN_TX_BYTES, - atomic64_read(&peer->vpn_stats.tx.bytes)) || + vpn_stats.tx_bytes) || nla_put_uint(skb, OVPN_A_PEER_VPN_TX_PACKETS, - atomic64_read(&peer->vpn_stats.tx.packets)) || + vpn_stats.tx_packets) || /* link RX stats */ nla_put_uint(skb, OVPN_A_PEER_LINK_RX_BYTES, - atomic64_read(&peer->link_stats.rx.bytes)) || + link_stats.rx_bytes) || nla_put_uint(skb, OVPN_A_PEER_LINK_RX_PACKETS, - atomic64_read(&peer->link_stats.rx.packets)) || + link_stats.rx_packets) || /* link TX stats */ nla_put_uint(skb, OVPN_A_PEER_LINK_TX_BYTES, - atomic64_read(&peer->link_stats.tx.bytes)) || + link_stats.tx_bytes) || nla_put_uint(skb, OVPN_A_PEER_LINK_TX_PACKETS, - atomic64_read(&peer->link_stats.tx.packets))) + link_stats.tx_packets)) goto err; nla_nest_end(skb, attr); diff --git a/drivers/net/ovpn/peer.c b/drivers/net/ovpn/peer.c index 970ca7232ae45..82998051db916 100644 --- a/drivers/net/ovpn/peer.c +++ b/drivers/net/ovpn/peer.c @@ -115,30 +115,42 @@ struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id) RCU_INIT_POINTER(peer->bind, NULL); ovpn_crypto_state_init(&peer->crypto); spin_lock_init(&peer->lock); - ovpn_peer_stats_init(&peer->vpn_stats); - ovpn_peer_stats_init(&peer->link_stats); INIT_WORK(&peer->keepalive_work, ovpn_peer_keepalive_send); + ret = -ENOMEM; + peer->vpn_stats = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats); + if (!peer->vpn_stats) + goto err_free_peer; + + peer->link_stats = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats); + if (!peer->link_stats) + goto err_free_stats; + ret = dst_cache_init(&peer->dst_cache, GFP_KERNEL); if (ret < 0) { netdev_err(ovpn->dev, "cannot initialize dst cache for peer %u\n", peer->id); - kfree(peer); - return ERR_PTR(ret); + goto err_free_stats; } ret = percpu_ref_init(&peer->refcount, ovpn_peer_release_ref, 0, GFP_KERNEL); if (ret < 0) { dst_cache_destroy(&peer->dst_cache); - kfree(peer); - return ERR_PTR(ret); + goto err_free_stats; } netdev_hold(ovpn->dev, &peer->dev_tracker, GFP_KERNEL); return peer; + +err_free_stats: + free_percpu(peer->link_stats); + free_percpu(peer->vpn_stats); +err_free_peer: + kfree(peer); + return ERR_PTR(ret); } /** @@ -357,6 +369,8 @@ static void ovpn_peer_release_rcu(struct rcu_head *head) * perform it in the RCU callback, when all contexts are done */ dst_cache_destroy(&peer->dst_cache); + free_percpu(peer->link_stats); + free_percpu(peer->vpn_stats); percpu_ref_exit(&peer->refcount); kfree(peer); } diff --git a/drivers/net/ovpn/peer.h b/drivers/net/ovpn/peer.h index 489bf26466847..550f600ac120a 100644 --- a/drivers/net/ovpn/peer.h +++ b/drivers/net/ovpn/peer.h @@ -10,6 +10,7 @@ #ifndef _NET_OVPN_OVPNPEER_H_ #define _NET_OVPN_OVPNPEER_H_ +#include #include #include #include @@ -53,14 +54,14 @@ * @keepalive_timeout: seconds after which an inactive peer is considered dead * @keepalive_recv_exp: future timestamp when the peer should expire * @last_recv: timestamp of the last authenticated received packet - * @vpn_stats: per-peer in-VPN TX/RX stats - * @link_stats: per-peer link/transport TX/RX stats * @delete_reason: why peer was deleted (i.e. timeout, transport error, ..) * @lock: protects binding to peer (bind) and keepalive* fields - * @refcount: reference counter * @rcu: used to free peer in an RCU safe way * @release_entry: entry for the socket release list * @keepalive_work: used to schedule keepalive sending + * @refcount: reference counter + * @vpn_stats: per-CPU in-VPN TX/RX stats + * @link_stats: per-CPU link/transport TX/RX stats */ struct ovpn_peer { struct ovpn_priv *ovpn; @@ -107,14 +108,23 @@ struct ovpn_peer { unsigned long keepalive_timeout; unsigned long keepalive_recv_exp; time64_t last_recv; - struct ovpn_peer_stats vpn_stats; - struct ovpn_peer_stats link_stats; enum ovpn_del_peer_reason delete_reason; spinlock_t lock; /* protects bind and keepalive* */ - struct percpu_ref refcount; struct rcu_head rcu; struct llist_node release_entry; struct work_struct keepalive_work; + + /* Read on the packet path but written only at setup + * and teardown: keep away from per-packet writes. + */ + /* private: */ + __cacheline_group_begin_aligned(read_mostly); + /* public: */ + struct pcpu_sw_netstats __percpu *vpn_stats; + struct pcpu_sw_netstats __percpu *link_stats; + struct percpu_ref refcount; + /* private: */ + __cacheline_group_end_aligned(read_mostly); }; /** diff --git a/drivers/net/ovpn/stats.c b/drivers/net/ovpn/stats.c deleted file mode 100644 index d637143473bb9..0000000000000 --- a/drivers/net/ovpn/stats.c +++ /dev/null @@ -1,21 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* OpenVPN data channel offload - * - * Copyright (C) 2020-2025 OpenVPN, Inc. - * - * Author: James Yonan - * Antonio Quartulli - */ - -#include - -#include "stats.h" - -void ovpn_peer_stats_init(struct ovpn_peer_stats *ps) -{ - atomic64_set(&ps->rx.bytes, 0); - atomic64_set(&ps->rx.packets, 0); - - atomic64_set(&ps->tx.bytes, 0); - atomic64_set(&ps->tx.packets, 0); -} diff --git a/drivers/net/ovpn/stats.h b/drivers/net/ovpn/stats.h index 3a45b97c00568..b342c952fce65 100644 --- a/drivers/net/ovpn/stats.h +++ b/drivers/net/ovpn/stats.h @@ -13,37 +13,48 @@ #include -/* one stat */ -struct ovpn_peer_stat { - atomic64_t bytes; - atomic64_t packets; -}; - -/* rx and tx stats combined */ -struct ovpn_peer_stats { - struct ovpn_peer_stat rx; - struct ovpn_peer_stat tx; -}; - -void ovpn_peer_stats_init(struct ovpn_peer_stats *ps); - -static inline void ovpn_peer_stats_increment(struct ovpn_peer_stat *stat, - const unsigned int n) +/** + * ovpn_peer_stats_increment_rx - account one received packet + * @stats: per-CPU counters + * @n: received bytes + * + * Context: Softirq or task context with BHs disabled. + * Not callable from hardirq or NMI context. + */ +static inline void +ovpn_peer_stats_increment_rx(struct pcpu_sw_netstats __percpu *stats, + const unsigned int n) { - atomic64_add(n, &stat->bytes); - atomic64_inc(&stat->packets); -} + struct pcpu_sw_netstats *s; -static inline void ovpn_peer_stats_increment_rx(struct ovpn_peer_stats *stats, - const unsigned int n) -{ - ovpn_peer_stats_increment(&stats->rx, n); + lockdep_assert_in_softirq(); + s = this_cpu_ptr(stats); + u64_stats_update_begin(&s->syncp); + u64_stats_add(&s->rx_bytes, n); + u64_stats_inc(&s->rx_packets); + u64_stats_update_end(&s->syncp); } -static inline void ovpn_peer_stats_increment_tx(struct ovpn_peer_stats *stats, - const unsigned int n) +/** + * ovpn_peer_stats_increment_tx - account one transmitted packet + * @stats: per-CPU counters + * @n: transmitted bytes + * + * Context: Softirq or task context with BHs disabled. + * Not callable from hardirq or NMI context. + */ +static inline void +ovpn_peer_stats_increment_tx(struct pcpu_sw_netstats __percpu *stats, + const unsigned int n) { - ovpn_peer_stats_increment(&stats->tx, n); + struct pcpu_sw_netstats *s; + + lockdep_assert_in_softirq(); + s = this_cpu_ptr(stats); + u64_stats_update_begin(&s->syncp); + u64_stats_add(&s->tx_bytes, n); + u64_stats_inc(&s->tx_packets); + u64_stats_update_end(&s->syncp); } static inline void ovpn_dev_dstats_tx_dropped(struct net_device *dev) diff --git a/drivers/net/ovpn/tcp.c b/drivers/net/ovpn/tcp.c index 29981c0279e27..440adeada9102 100644 --- a/drivers/net/ovpn/tcp.c +++ b/drivers/net/ovpn/tcp.c @@ -143,8 +143,9 @@ static void ovpn_tcp_rcv(struct strparser *strp, struct sk_buff *skb) * this peer */ ovpn_peer_hold(peer); - + local_bh_disable(); ovpn_recv(peer, skb); + local_bh_enable(); return; err: /* take reference for deferred peer deletion */