From patchwork Mon Oct 12 16:33:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Baffo X-Patchwork-Id: 5462 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7000:20d9:b0:8d7:ebbe:b250 with SMTP id f25csp412740mad; Mon, 12 Oct 2026 09:34:02 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvByaFOpyjiwZx2Gz1OI+NZagwVsAErDGIy+O9bmFX4zwKTDvEeedzxM4yo9NsDWALaD2J0ERmxEReVI=@openvpn.net X-Received: by 2002:a05:6870:c69b:b0:494:b0db:4e0d with SMTP id 586e51a60fabf-4a2a86a728bmr7977928fac.7.1791822842680; Mon, 12 Oct 2026 09:34:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1791822842; cv=none; d=google.com; s=arc-20260327; b=ND94eb+jFQoNXYHP7cI5+imbX7U0WRjIhfISo9cPM6x4RKcleJKPnIGm2y0EZyh99u ER9GQuhX0Dd6Zy0rMWdEDM97prE7j1LuOYtprxOo29wrzzPcazrUaPAcIi85cueV4cJV RB+g5yDKP/nnS6UuKuxbKx9ftPR1p8ITcl3QginxDlK9BLNZcKZR5pWhMq3WofK67uWd sRKOtQ3MBU2srCqSrGA4JRUKd1IpnsDXJx7d1PC4l5YqAJFvieDwFH7n5vb8ZG5suBiu Z4O6HwLeCTPTGceWKBhH27RaBR9M0Bp6u4fOH8POr4Ab36Q/vYadGrMayCVCBQD0qPdx 2aiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=ZCWHPptl4Yh3Q4TLXXodiRvB2T9gDYlGv+t8rOxremE=; fh=4NbAC/LsuMLI0S0hprUlLSLCiHwg6SCAifhH718Jh0Q=; b=iwnZ5rPqF8fUT5BJG4SbBGBOiGpGLbMN+0IrYBhlQUswl1OpwevR6XqZ9gpU79rXHQ Z7CAc4hQIQUgAULcf92jOhRtPsxjuXS6zQWM2YB2A0BViWGzi0taiHWmWCnIKfsDMgEe 7nTvQch+AqJdXZMGVS0rtT2L8KMdnGvm5Yl+bAn74T0JxelUrw6CfMidmWm7dgdfkec0 IWHsMGMG9KGNQzLgUcLrg/sSKNqnXcDgrAKe14q/T2um2N6+KUmSYbAELWQp82dwT9ik cu1pYb7NTgDMM+Udb7Vz0tKKmN5daIvWqeSK88rDTgSfLOagJ6WHpdFGNv3645qQLttH lv0g==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b="TOZg/q1i"; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=Dj4bLkvl; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=JUWGWKv0; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=heQiwtiT; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id 586e51a60fabf-4a2a84f1f53si15217623fac.99.2026.10.12.09.34.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Oct 2026 09:34:02 -0700 (PDT) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b="TOZg/q1i"; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=Dj4bLkvl; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=JUWGWKv0; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=heQiwtiT; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Content-Transfer-Encoding:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Subject:MIME-Version:References:In-Reply-To:Message-ID:Date:To:From:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ZCWHPptl4Yh3Q4TLXXodiRvB2T9gDYlGv+t8rOxremE=; b=TOZg/q1iECViIHEo8X8lqROeLg AUeZbFVy2QHyuzHquaQ/P/GaPJeE37k1z5xN4chB8o9Gds6C4RY9YiRpVk2bfCI7nT5saND9JtsGe qdKa5RCRZar8JM/gZgyljXa+fdKLUwvegox1sQ7y3x+FXoaqa0R6I3w6KB51OMvBQE4A=; Received: from [127.0.0.1] (helo=sfs-ml-2.v29.lw.sourceforge.com) by sfs-ml-2.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1xGIyL-0004UY-JT; Mon, 12 Oct 2026 16:33:58 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-2.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1xGIyB-0004Tp-L5 for openvpn-devel@lists.sourceforge.net; Mon, 12 Oct 2026 16:33:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=lC78oS6FkcAzgRhNt6765NBkVrLRShSXP8c8UsjLbaA=; b=Dj4bLkvlEJMW2OeIHmb+oBzBC6 /ZPABScDfX3LT6xSoXUjuo4+Gwn8XVRq0OZlu+roFFqxOSGF/s8G4QQFELKG5C1hed8cxvzCDnzx7 4ISpE2/YUPgZ3DkIDpR++3WAeC/uqgzKvnKHWfAR6CMaPFW+TSfJP43RsskVNMWsB7Vk=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=lC78oS6FkcAzgRhNt6765NBkVrLRShSXP8c8UsjLbaA=; b=JUWGWKv0CfWaV7Hjxy7Y7WcvYU cr7vJK7+MJVVWXjNR8/de3Ao9A8OJM2CgBigWMRwbKMcuJvVohuW1/2iIKeThyTLrVqcM1pRrwlNv iDh49JvJ3CHrZMjzLBc5Lm4KnGZKu30SujsOM5joq6x82u/TerOtTDidAymcYx1vIH6s=; Received: from mout-b-203.mailbox.org ([195.10.208.52]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1xGIyA-0006pT-V2 for openvpn-devel@lists.sourceforge.net; Mon, 12 Oct 2026 16:33:48 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:b231:465::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519MLKEM768 server-signature RSA-PSS (4096 bits) server-digest SHA512) (No client certificate requested) by mout-b-203.mailbox.org (Postfix) with ESMTPS id 4j3NM71Wv8zLm3w; Mon, 12 Oct 2026 18:33:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandelbit.com; s=MBO0001; t=1791822819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lC78oS6FkcAzgRhNt6765NBkVrLRShSXP8c8UsjLbaA=; b=heQiwtiTy10Q36+XGKGD7I7r2zPNXsrXlfdpTLfpe6QiEn6WQ6iRAyg+HAsGoL9VJd8cVl BDgbwmXt9qW/CJSAKIih7sGxDpU8aLWl2xzFXpTW4QQaWrJMyc6b0//yAr8Ucd0GKfzF8/ ocsrN8n2Fu02hBeTC1m94XBKdVPq4+1TwsU/ReXdFCm+7Vz9qyKU+izXtNI+VAr+vXk94M tjriR+ZzZxw83DYQeixAGHeHB0kaE8corLLTpOJ7zW7QPTIEc338x9/MHHQnpkb+tthlIb xzF/bBIsnkcPvArmm7R/CR08wJ/Kdm3q4+R6i5o5zzHBMWRicWPS0LNA/Qr8Lg== Authentication-Results: outgoing_mbo_mout; dkim=none; spf=pass (outgoing_mbo_mout: domain of marco@mandelbit.com designates 2001:67c:2050:b231:465::1 as permitted sender) smtp.mailfrom=marco@mandelbit.com From: Marco Baffo To: openvpn-devel@lists.sourceforge.net Date: Mon, 12 Oct 2026 18:33:31 +0200 Message-ID: <20261012163333.60389-2-marco@mandelbit.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261012163333.60389-1-marco@mandelbit.com> References: <20261012163333.60389-1-marco@mandelbit.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4j3NM71Wv8zLm3w X-Spam-Score: -0.2 (/) X-Spam-Report: Spam detection software, running on the system "sfi-spamd-2.hosts.colo.sdot.me", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: From: Ralf Lici Each packet holds its peer across packet processing and asynchronous crypto completion. When traffic for one peer is spread across several CPUs, the shared kref cache line therefore moves between thos [...] Content analysis details: (-0.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain X-Headers-End: 1xGIyA-0006pT-V2 Subject: [Openvpn-devel] [PATCH ovpn net-next v3 1/3] ovpn: use percpu references for peers X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: 1878862429115872852 X-GMAIL-MSGID: 1878862429115872852 From: Ralf Lici Each packet holds its peer across packet processing and asynchronous crypto completion. When traffic for one peer is spread across several CPUs, the shared kref cache line therefore moves between those CPUs for every packet. Replace the peer kref with percpu_ref and kill the initial reference exactly once after the peer has been removed from its lookup structures and detached from its socket. Keep the existing RCU-delayed destruction and release the percpu_ref storage in the final RCU callback. Use an unconditional percpu_ref_get where the caller already owns a peer reference or otherwise excludes teardown. This includes TCP callbacks that observe an ovpn_socket under RCU: socket teardown clears sk_user_data and waits for RCU readers before killing the peer. RCU-protected peer lookup paths instead use percpu_ref_tryget_live_rcu, so they stop admitting new users as soon as teardown starts. percpu_ref_init can fail, so propagate allocation failure from peer creation. This trades a per-CPU counter allocation for a cheaper live data path on these long-lived, heavily shared objects. Specifically, in terms of memory, this costs: 8 bytes per possible CPU for the counters, a 56-byte control object, and 8 additional bytes in struct ovpn_peer. In a 32-stream test using one peer and one key, perf c2c placed the peer kref cacheline second among system-wide shared cachelines, with 37 sampled HITM loads attributed to the locked reference update. After this change that shared reference line was no longer sampled, while the untouched key-slot kref addressed by the next commit remained visible. RX used a single receive queue, so only TX exercised CPU fan-out in this test. Signed-off-by: Ralf Lici --- No changes in v3. Changes in v2: - Use unconditional peer holds in TCP socket callbacks, as observing an ovpn_socket under RCU guarantees that peer teardown has not reached percpu_ref_kill. - Document the socket-detachment and peer-kill ordering in the commit message and ovpn_tcp_close. drivers/net/ovpn/io.c | 6 +----- drivers/net/ovpn/netlink.c | 6 +++--- drivers/net/ovpn/peer.c | 37 ++++++++++++++++++++++-------------- drivers/net/ovpn/peer.h | 37 ++++++++++++++++++++++++++++++------ drivers/net/ovpn/tcp.c | 39 +++++++++++++++++++------------------- 5 files changed, 77 insertions(+), 48 deletions(-) diff --git a/drivers/net/ovpn/io.c b/drivers/net/ovpn/io.c index 9526f8096da60..15072ce960fa4 100644 --- a/drivers/net/ovpn/io.c +++ b/drivers/net/ovpn/io.c @@ -321,11 +321,7 @@ static bool ovpn_encrypt_one(struct ovpn_peer *peer, struct sk_buff *skb) /* take a reference to the peer because the crypto code may run async. * ovpn_encrypt_post() will release it upon completion */ - if (unlikely(!ovpn_peer_hold(peer))) { - DEBUG_NET_WARN_ON_ONCE(1); - ovpn_crypto_key_slot_put(ks); - return false; - } + ovpn_peer_hold(peer); memset(ovpn_skb_cb(skb), 0, sizeof(struct ovpn_cb)); ovpn_encrypt_post(skb, ovpn_aead_encrypt(peer, ks, skb)); diff --git a/drivers/net/ovpn/netlink.c b/drivers/net/ovpn/netlink.c index 4dad852941982..8cdf46b611425 100644 --- a/drivers/net/ovpn/netlink.c +++ b/drivers/net/ovpn/netlink.c @@ -463,11 +463,11 @@ int ovpn_nl_peer_new_doit(struct sk_buff *skb, struct genl_info *info) ovpn_socket_release(peer); peer_release: /* For UDP, the peer is unreachable until added to the hashtables, so - * dropping the initial reference is enough. For TCP, the peer may be + * killing the initial reference is enough. For TCP, the peer may be * concurrently reachable via sk_user_data->peer until - * ovpn_socket_release() detaches; rely on the refcount. + * ovpn_socket_release() detaches; rely on the percpu reference. */ - ovpn_peer_put(peer); + ovpn_peer_kill(peer); return ret; } diff --git a/drivers/net/ovpn/peer.c b/drivers/net/ovpn/peer.c index c95656ca7c357..970ca7232ae45 100644 --- a/drivers/net/ovpn/peer.c +++ b/drivers/net/ovpn/peer.c @@ -22,6 +22,8 @@ #include "peer.h" #include "socket.h" +static void ovpn_peer_release_ref(struct percpu_ref *ref); + static void unlock_ovpn(struct ovpn_priv *ovpn, struct llist_head *release_list) __releases(&ovpn->lock) @@ -33,7 +35,7 @@ static void unlock_ovpn(struct ovpn_priv *ovpn, llist_for_each_entry_safe(peer, next, release_list->first, release_entry) { ovpn_socket_release(peer); - ovpn_peer_put(peer); + ovpn_peer_kill(peer); } } @@ -113,7 +115,6 @@ struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id) RCU_INIT_POINTER(peer->bind, NULL); ovpn_crypto_state_init(&peer->crypto); spin_lock_init(&peer->lock); - kref_init(&peer->refcount); ovpn_peer_stats_init(&peer->vpn_stats); ovpn_peer_stats_init(&peer->link_stats); INIT_WORK(&peer->keepalive_work, ovpn_peer_keepalive_send); @@ -127,6 +128,14 @@ struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id) return ERR_PTR(ret); } + ret = percpu_ref_init(&peer->refcount, ovpn_peer_release_ref, 0, + GFP_KERNEL); + if (ret < 0) { + dst_cache_destroy(&peer->dst_cache); + kfree(peer); + return ERR_PTR(ret); + } + netdev_hold(ovpn->dev, &peer->dev_tracker, GFP_KERNEL); return peer; @@ -348,6 +357,7 @@ static void ovpn_peer_release_rcu(struct rcu_head *head) * perform it in the RCU callback, when all contexts are done */ dst_cache_destroy(&peer->dst_cache); + percpu_ref_exit(&peer->refcount); kfree(peer); } @@ -366,12 +376,12 @@ static void ovpn_peer_release(struct ovpn_peer *peer) } /** - * ovpn_peer_release_kref - callback for kref_put - * @kref: the kref object belonging to the peer + * ovpn_peer_release_ref - callback for the peer percpu reference + * @ref: the percpu_ref object belonging to the peer */ -void ovpn_peer_release_kref(struct kref *kref) +static void ovpn_peer_release_ref(struct percpu_ref *ref) { - struct ovpn_peer *peer = container_of(kref, struct ovpn_peer, refcount); + struct ovpn_peer *peer = container_of(ref, struct ovpn_peer, refcount); ovpn_peer_release(peer); } @@ -569,7 +579,7 @@ ovpn_peer_get_by_transp_addr_p2p(struct ovpn_priv *ovpn, rcu_read_lock(); tmp = rcu_dereference(ovpn->peer); if (likely(tmp && ovpn_peer_transp_match(tmp, ss) && - ovpn_peer_hold(tmp))) + ovpn_peer_hold_rcu(tmp))) peer = tmp; rcu_read_unlock(); @@ -610,7 +620,7 @@ struct ovpn_peer *ovpn_peer_get_by_transp_addr(struct ovpn_priv *ovpn, if (!ovpn_peer_transp_match(tmp, &ss)) continue; - if (!ovpn_peer_hold(tmp)) + if (!ovpn_peer_hold_rcu(tmp)) continue; peer = tmp; @@ -641,7 +651,7 @@ static struct ovpn_peer *ovpn_peer_get_by_id_p2p(struct ovpn_priv *ovpn, rcu_read_lock(); tmp = rcu_dereference(ovpn->peer); - if (likely(tmp && tmp->id == peer_id && ovpn_peer_hold(tmp))) + if (likely(tmp && tmp->id == peer_id && ovpn_peer_hold_rcu(tmp))) peer = tmp; rcu_read_unlock(); @@ -671,7 +681,7 @@ struct ovpn_peer *ovpn_peer_get_by_id(struct ovpn_priv *ovpn, u32 peer_id) if (tmp->id != peer_id) continue; - if (!ovpn_peer_hold(tmp)) + if (!ovpn_peer_hold_rcu(tmp)) continue; peer = tmp; @@ -745,7 +755,7 @@ struct ovpn_peer *ovpn_peer_get_by_dst(struct ovpn_priv *ovpn, if (ovpn->mode == OVPN_MODE_P2P) { rcu_read_lock(); peer = rcu_dereference(ovpn->peer); - if (unlikely(peer && !ovpn_peer_hold(peer))) + if (unlikely(peer && !ovpn_peer_hold_rcu(peer))) peer = NULL; rcu_read_unlock(); return peer; @@ -763,7 +773,7 @@ struct ovpn_peer *ovpn_peer_get_by_dst(struct ovpn_priv *ovpn, break; } - if (unlikely(peer && !ovpn_peer_hold(peer))) + if (unlikely(peer && !ovpn_peer_hold_rcu(peer))) peer = NULL; rcu_read_unlock(); @@ -1369,8 +1379,7 @@ static time64_t ovpn_peer_keepalive_work_single(struct ovpn_peer *peer, netdev_dbg(peer->ovpn->dev, "sending keepalive to peer %u\n", peer->id); - if (WARN_ON(!ovpn_peer_hold(peer))) - return 0; + ovpn_peer_hold(peer); if (!queue_work(ovpn_wq, &peer->keepalive_work)) ovpn_peer_put(peer); } diff --git a/drivers/net/ovpn/peer.h b/drivers/net/ovpn/peer.h index dfa5c0037e02b..489bf26466847 100644 --- a/drivers/net/ovpn/peer.h +++ b/drivers/net/ovpn/peer.h @@ -10,6 +10,7 @@ #ifndef _NET_OVPN_OVPNPEER_H_ #define _NET_OVPN_OVPNPEER_H_ +#include #include #include @@ -110,7 +111,7 @@ struct ovpn_peer { struct ovpn_peer_stats link_stats; enum ovpn_del_peer_reason delete_reason; spinlock_t lock; /* protects bind and keepalive* */ - struct kref refcount; + struct percpu_ref refcount; struct rcu_head rcu; struct llist_node release_entry; struct work_struct keepalive_work; @@ -120,14 +121,26 @@ struct ovpn_peer { * ovpn_peer_hold - increase reference counter * @peer: the peer whose counter should be increased * - * Return: true if the counter was increased or false if it was zero already + * The caller must already own a peer reference or otherwise guarantee that + * teardown cannot complete while the new reference is acquired. */ -static inline bool ovpn_peer_hold(struct ovpn_peer *peer) +static inline void ovpn_peer_hold(struct ovpn_peer *peer) { - return kref_get_unless_zero(&peer->refcount); + percpu_ref_get(&peer->refcount); } -void ovpn_peer_release_kref(struct kref *kref); +/** + * ovpn_peer_hold_rcu - acquire a live peer reference under RCU + * @peer: peer to acquire + * + * The caller must hold rcu_read_lock. + * + * Return: true if the reference was acquired, false if teardown has started + */ +static inline bool ovpn_peer_hold_rcu(struct ovpn_peer *peer) +{ + return percpu_ref_tryget_live_rcu(&peer->refcount); +} /** * ovpn_peer_put - decrease reference counter @@ -135,7 +148,19 @@ void ovpn_peer_release_kref(struct kref *kref); */ static inline void ovpn_peer_put(struct ovpn_peer *peer) { - kref_put(&peer->refcount, ovpn_peer_release_kref); + percpu_ref_put(&peer->refcount); +} + +/** + * ovpn_peer_kill - drop the peer's initial reference + * @peer: peer which has been unpublished and is being destroyed + * + * This must be called exactly once, after the peer is no longer reachable + * through its owning ovpn instance. + */ +static inline void ovpn_peer_kill(struct ovpn_peer *peer) +{ + percpu_ref_kill(&peer->refcount); } struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id); diff --git a/drivers/net/ovpn/tcp.c b/drivers/net/ovpn/tcp.c index 8fe8a8e750a40..47af3a62f3551 100644 --- a/drivers/net/ovpn/tcp.c +++ b/drivers/net/ovpn/tcp.c @@ -140,21 +140,18 @@ static void ovpn_tcp_rcv(struct strparser *strp, struct sk_buff *skb) /* hold reference to peer as required by ovpn_recv(). * * NOTE: in this context we should already be holding a reference to - * this peer, therefore ovpn_peer_hold() is not expected to fail + * this peer */ - if (WARN_ON(!ovpn_peer_hold(peer))) - goto err_nopeer; + ovpn_peer_hold(peer); ovpn_recv(peer, skb); return; err: - /* take reference for deferred peer deletion. should never fail */ - if (WARN_ON(!ovpn_peer_hold(peer))) - goto err_nopeer; + /* take reference for deferred peer deletion */ + ovpn_peer_hold(peer); if (!queue_work(ovpn_wq, &peer->tcp.defer_del_work)) ovpn_peer_put(peer); ovpn_dev_dstats_rx_dropped(peer->ovpn->dev); -err_nopeer: kfree_skb(skb); } @@ -168,11 +165,12 @@ static int ovpn_tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, rcu_read_lock(); sock = rcu_dereference_sk_user_data(sk); - if (unlikely(!sock || !sock->peer || !ovpn_peer_hold(sock->peer))) { + if (unlikely(!sock || !sock->peer)) { rcu_read_unlock(); return -EBADF; } peer = sock->peer; + ovpn_peer_hold(peer); rcu_read_unlock(); skb = __skb_recv_datagram(sk, &peer->tcp.user_queue, flags, &off, &err); @@ -385,10 +383,11 @@ static void ovpn_tcp_release(struct sock *sk) /* during initialization this function is called before * assigning sock->peer */ - if (unlikely(!peer || !ovpn_peer_hold(peer))) { + if (unlikely(!peer)) { rcu_read_unlock(); return; } + ovpn_peer_hold(peer); rcu_read_unlock(); __skb_queue_head_init(&queue); @@ -411,13 +410,14 @@ static int ovpn_tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) lock_sock(sk); rcu_read_lock(); sock = rcu_dereference_sk_user_data(sk); - if (unlikely(!sock || !sock->peer || !ovpn_peer_hold(sock->peer))) { + if (unlikely(!sock || !sock->peer)) { rcu_read_unlock(); release_sock(sk); return -EIO; } - rcu_read_unlock(); peer = sock->peer; + ovpn_peer_hold(peer); + rcu_read_unlock(); if (msg->msg_flags & ~(MSG_DONTWAIT | MSG_NOSIGNAL)) { ret = -EOPNOTSUPP; @@ -588,10 +588,12 @@ static void ovpn_tcp_close(struct sock *sk, long timeout) } peer = sock->peer; - if (!peer || !ovpn_peer_hold(peer)) { + if (!peer) { rcu_read_unlock(); return; } + /* socket teardown waits for RCU readers before killing the peer */ + ovpn_peer_hold(peer); rcu_read_unlock(); ovpn_peer_del(peer, OVPN_DEL_PEER_REASON_TRANSPORT_DISCONNECT); @@ -604,7 +606,7 @@ static __poll_t ovpn_tcp_poll(struct file *file, struct socket *sock, { struct sk_buff_head *queue = &sock->sk->sk_receive_queue; struct ovpn_socket *ovpn_sock; - struct ovpn_peer *peer = NULL; + struct ovpn_peer *peer; __poll_t mask; rcu_read_lock(); @@ -619,16 +621,13 @@ static __poll_t ovpn_tcp_poll(struct file *file, struct socket *sock, return 0; } - if (ovpn_peer_hold(ovpn_sock->peer)) { - peer = ovpn_sock->peer; - queue = &peer->tcp.user_queue; - } + peer = ovpn_sock->peer; + ovpn_peer_hold(peer); + queue = &peer->tcp.user_queue; rcu_read_unlock(); mask = datagram_poll_queue(file, sock, wait, queue); - - if (peer) - ovpn_peer_put(peer); + ovpn_peer_put(peer); return mask; } From patchwork Mon Oct 12 16:33:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Baffo X-Patchwork-Id: 5461 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7000:20d9:b0:8d7:ebbe:b250 with SMTP id f25csp412733mad; Mon, 12 Oct 2026 09:34:02 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvByaBn90J77G+poYOjLvQF4edmTxXdwhJWSy7jQ6Z33mMGa08e0mVJfddTnrMKfqVf2RpNqjMItY1bU=@openvpn.net X-Received: by 2002:a05:6820:1a04:b0:6d8:bba5:658d with SMTP id 006d021491bc7-6ef0ff9d047mr6743905eaf.41.1791822842466; Mon, 12 Oct 2026 09:34:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1791822842; cv=none; d=google.com; s=arc-20260327; b=pbHr7+V7ra7zNg4rI9eeCIGnFf9ejlmWTpVnylLldKhLPpUcM8sstQ4crdcp4oZEyb hJV/GPtNJKRkoWlpommBJdE+4Utr0VB8rKRfEC7lINb7Ag2ByZfOrA+wyYEqb/F4AxEP gPHxdEDIHEGqTBJ3o4sYx6GSnHc2rgYBR7e/xWCjPpT0KfUYwfPmSbRxgoksMLDudGi6 aVEzNhMDmh1S3qYKkR0nCGUG1kNVtrlYHqImO0ckXj+NI0snU48mW4xFTijNRNOGzAVA LkHre+2cVgyr3Fs5LZIaYr+yiG8FLsK0PTAi7/9ayojjK6xyNr6JX52miopOabaKhFER 7vEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=5KazvK6ldjyyBYn8LDOR1JAVXXNHg14zr65jRi/jtwg=; fh=4NbAC/LsuMLI0S0hprUlLSLCiHwg6SCAifhH718Jh0Q=; b=ssNT/zNaoLvtCFuBU9yiGLKBGka/KT/0aDKjnxYJjl0FRdOYqDw3eNlGMEyknfst6u uvZSB2naPHaJ1onfPUSAHDcKZULSVy+Gxq8sjzxmhTY0GCGZHPKZv3bUo9hdStEZDpVY zE21wzIbYf3PUaq7o1BRe5BL5pVZFyhT9eki92D3I6dhZEXPHIXlc+xxklgI4lK5k0qI OqjF3oFFg2JoXz2CMGvJglHijlOU/KxpQmDmr3mgWqtWHTSpwMv4e1EPyNQp7pycYRYv dKnDW37tn/9GTJAuVNMSzEK7xVIWGtReNhmc8Dev96FX+Ms+N2WAOvuK3WiT4tDe+6aI 1ttg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=Et7DGjwe; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=dwPwUp2R; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=VdYswOaR; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=1mm3D2ZM; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id 006d021491bc7-6f4c9fff595si5215423eaf.50.2026.10.12.09.34.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Oct 2026 09:34:02 -0700 (PDT) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=Et7DGjwe; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=dwPwUp2R; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=VdYswOaR; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=1mm3D2ZM; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Content-Transfer-Encoding:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Subject:MIME-Version:References:In-Reply-To:Message-ID:Date:To:From:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5KazvK6ldjyyBYn8LDOR1JAVXXNHg14zr65jRi/jtwg=; b=Et7DGjwe/xgyVmTM9azyploV0s el9h2MHlAlxz3WFXNbO2re9HtWpfg3e4hvK1neIaqzf9d1sd//mJ7TmkKRfveS9sP7AtYptneC+t1 v+o8U9Vf9fzhEuU9hldVEDasE4voTTB4ZTpl8yC+pJaY96nhMDv6U1VXCB9DioZd7vac=; Received: from [127.0.0.1] (helo=sfs-ml-4.v29.lw.sourceforge.com) by sfs-ml-4.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1xGIyM-0003Wq-Sc; Mon, 12 Oct 2026 16:33:58 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-4.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1xGIyK-0003Wj-Tv for openvpn-devel@lists.sourceforge.net; Mon, 12 Oct 2026 16:33:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=7SB78RaSfqnnlRQ4XIQB2AyFrFkK4ePOJX7ugufSCP8=; b=dwPwUp2RkfSc8/qgq/ItXAvgRI NLE9GAhHO/ez9ee50VePElv+8XY/1PlCv08a0brbzz9GgFfrVZKUQPWrEj9gUdh3v59ChgvDJJKlb AFNJFQxIVwyy0HKdwI0Hk1wpwPVT7qVXpb70xEbxpQ/hhXMosbrcwzcvfKA9j1MS6ikA=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=7SB78RaSfqnnlRQ4XIQB2AyFrFkK4ePOJX7ugufSCP8=; b=VdYswOaRw5kIH4i/e0LK6A+g3Z Xn+5I5XckPAORGL6e3ayFxO3ZpIof1u+hFTlVromExJHnGmVBa8aJc7I3mI2s6Ac13pAOweu2dMEA zgT9RI/5rcNaWCp7S00GsQuWJAdBcrbGsgCWlpZ7oTy9i/K2VdDUw9MJyZT3SOR8VUdg=; Received: from mout-b-201.mailbox.org ([195.10.208.61]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1xGIyI-0006pa-1g for openvpn-devel@lists.sourceforge.net; Mon, 12 Oct 2026 16:33:56 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:b231:465::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519MLKEM768 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-b-201.mailbox.org (Postfix) with ESMTPS id 4j3NMG2d2yzLm0f; Mon, 12 Oct 2026 18:33:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandelbit.com; s=MBO0001; t=1791822826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7SB78RaSfqnnlRQ4XIQB2AyFrFkK4ePOJX7ugufSCP8=; b=1mm3D2ZMy8P/QIEx5xOQuPOfNT2dPMDEErTVphH8MPkr38sdmhL/A7fttabP9SpbXNOapo 859SiLt2XZl4gUSG7XUCIKlSZxGDODcjTUvjiCxhwEpf4OsbX1XZMD6G6QyJ5NQyPDaT0E CPfgep/ETKXz+KmSLej6wPgHYVVeNc/9y41TFwjnRy/3ceKV7S6Lm8AUmEuwMy+9T2pTq4 MtVLtqLr2stXQpo3MLIWJW51RsSGMqEXa76IQ/Arb1yKRn/dPNCV7hzLfAOnhgiuYKl0ba 8X5QJAXTPRzkKqKK1evt6pbHO4qLQOQpwa7yikf3xTZ2gBr93nr+OYJC98JFYQ== Authentication-Results: outgoing_mbo_mout; dkim=none; spf=pass (outgoing_mbo_mout: domain of marco@mandelbit.com designates 2001:67c:2050:b231:465::1 as permitted sender) smtp.mailfrom=marco@mandelbit.com From: Marco Baffo To: openvpn-devel@lists.sourceforge.net Date: Mon, 12 Oct 2026 18:33:32 +0200 Message-ID: <20261012163333.60389-3-marco@mandelbit.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261012163333.60389-1-marco@mandelbit.com> References: <20261012163333.60389-1-marco@mandelbit.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4j3NMG2d2yzLm0f X-Spam-Score: -0.2 (/) X-Spam-Report: Spam detection software, running on the system "sfi-spamd-1.hosts.colo.sdot.me", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: From: Ralf Lici Packet processing holds a key slot until asynchronous crypto completion. Parallel traffic using one key consequently updates the same kref cache line for every packet, even though key slots are normal [...] Content analysis details: (-0.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1xGIyI-0006pa-1g Subject: [Openvpn-devel] [PATCH ovpn net-next v3 2/3] ovpn: use percpu references for key slots X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: 1878862429286780140 X-GMAIL-MSGID: 1878862429286780140 From: Ralf Lici Packet processing holds a key slot until asynchronous crypto completion. Parallel traffic using one key consequently updates the same kref cache line for every packet, even though key slots are normally long-lived. Replace the key-slot kref with percpu_ref. Kill the initial reference after atomically unpublishing a slot, while ordinary packet completions only put their live references. Key destruction needs process context to release the crypto transforms and an RCU grace period for lockless configuration readers which do not hold a key reference. With percpu_ref, the release callback may run from the internal RCU callback which switches the reference to atomic mode. Queueing rcu_work from there would create a nested callback which the module cleanup rcu_barrier is not guaranteed to wait for. Record the RCU state after unpublishing the slot and queue ordinary work from the release callback instead. The worker conditionally waits for the recorded grace period before freeing the key. This provides explicit RCU synchronization without depending on percpu_ref implementation details or unconditionally starting another grace period. Peer teardown kills its keys before dropping the final netdevice reference, so rtnl_link_unregister cannot return before the key percpu-ref callbacks have been registered. The existing rcu_barrier drains those callbacks and destroy_workqueue subsequently drains the ordinary key cleanup work. Initialize the reference only after the rest of the key slot is ready, allowing the existing destruction path to handle allocation failure. The conversion adds an 8-byte counter per possible CPU and a 56-byte control allocation for each installed key slot; with at most two long-lived slots per peer, this is a bounded memory tradeoff for removing the shared reference cacheline from the packet path. After the peer conversion, perf c2c still identified the key-slot kref cacheline, with 15 sampled HITM loads attributed to its locked reference update. This change removed that line from the shared-cacheline profile as well. In a set of interleaved 32-flow iperf3 runs, the combined peer and key conversion moved throughput to 8.812 Gbit/s, against 7.734 Gbit/s for the baseline, a +13.95% improvement. Single-stream control found no reproducible regression. Signed-off-by: Ralf Lici --- No changes in v3. No changes in v2. drivers/net/ovpn/crypto.c | 16 ++++++++-------- drivers/net/ovpn/crypto.h | 27 ++++++++++++++++++++++----- drivers/net/ovpn/crypto_aead.c | 17 +++++++++++++---- 3 files changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/net/ovpn/crypto.c b/drivers/net/ovpn/crypto.c index 7e545428900a9..25c3a2b6da481 100644 --- a/drivers/net/ovpn/crypto.c +++ b/drivers/net/ovpn/crypto.c @@ -18,12 +18,12 @@ #include "crypto_aead.h" #include "crypto.h" -void ovpn_crypto_key_slot_release(struct kref *kref) +void ovpn_crypto_key_slot_release(struct percpu_ref *ref) { struct ovpn_crypto_key_slot *ks; - ks = container_of(kref, struct ovpn_crypto_key_slot, refcount); - queue_rcu_work(ovpn_wq, &ks->free_work); + ks = container_of(ref, struct ovpn_crypto_key_slot, refcount); + queue_work(ovpn_wq, &ks->free_work); } /* can only be invoked when all peer references have been dropped (i.e. RCU @@ -36,13 +36,13 @@ void ovpn_crypto_state_release(struct ovpn_crypto_state *cs) ks = rcu_access_pointer(cs->slots[0]); if (ks) { RCU_INIT_POINTER(cs->slots[0], NULL); - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); } ks = rcu_access_pointer(cs->slots[1]); if (ks) { RCU_INIT_POINTER(cs->slots[1], NULL); - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); } } @@ -66,7 +66,7 @@ bool ovpn_crypto_kill_key(struct ovpn_crypto_state *cs, u8 key_id) spin_unlock_bh(&cs->lock); if (ks) - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); /* let the caller know if a key was actually killed */ return ks; @@ -104,7 +104,7 @@ int ovpn_crypto_state_reset(struct ovpn_crypto_state *cs, spin_unlock_bh(&cs->lock); if (old) - ovpn_crypto_key_slot_put(old); + ovpn_crypto_key_slot_kill(old); return 0; } @@ -141,7 +141,7 @@ void ovpn_crypto_key_slot_delete(struct ovpn_crypto_state *cs, } pr_debug("deleting key slot %u, key_id=%u\n", slot, ks->key_id); - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); } void ovpn_crypto_key_slots_swap(struct ovpn_crypto_state *cs) diff --git a/drivers/net/ovpn/crypto.h b/drivers/net/ovpn/crypto.h index e3feb16d54985..a0209b1280de4 100644 --- a/drivers/net/ovpn/crypto.h +++ b/drivers/net/ovpn/crypto.h @@ -10,6 +10,8 @@ #ifndef _NET_OVPN_OVPNCRYPTO_H_ #define _NET_OVPN_OVPNCRYPTO_H_ +#include +#include #include #include "pktid.h" @@ -47,8 +49,9 @@ struct ovpn_crypto_key_slot { struct ovpn_pktid_recv pid_recv ____cacheline_aligned_in_smp; struct ovpn_pktid_xmit pid_xmit ____cacheline_aligned_in_smp; - struct rcu_work free_work; - struct kref refcount; + struct work_struct free_work; + unsigned long rcu_state; + struct percpu_ref refcount; }; struct ovpn_crypto_state { @@ -61,7 +64,7 @@ struct ovpn_crypto_state { static inline bool ovpn_crypto_key_slot_hold(struct ovpn_crypto_key_slot *ks) { - return kref_get_unless_zero(&ks->refcount); + return percpu_ref_tryget_live_rcu(&ks->refcount); } static inline void ovpn_crypto_state_init(struct ovpn_crypto_state *cs) @@ -121,11 +124,25 @@ ovpn_crypto_key_slot_primary(const struct ovpn_crypto_state *cs) return ks; } -void ovpn_crypto_key_slot_release(struct kref *kref); +void ovpn_crypto_key_slot_release(struct percpu_ref *ref); static inline void ovpn_crypto_key_slot_put(struct ovpn_crypto_key_slot *ks) { - kref_put(&ks->refcount, ovpn_crypto_key_slot_release); + percpu_ref_put(&ks->refcount); +} + +/** + * ovpn_crypto_key_slot_kill - stop new users and drop the initial reference + * @ks: key slot which has already been unpublished + * + * percpu_ref does not guarantee an RCU grace period before release. Record + * the RCU state after unpublishing the key so the final worker can synchronize + * with lockless readers without unconditionally starting another grace period. + */ +static inline void ovpn_crypto_key_slot_kill(struct ovpn_crypto_key_slot *ks) +{ + ks->rcu_state = get_state_synchronize_rcu(); + percpu_ref_kill(&ks->refcount); } int ovpn_crypto_state_reset(struct ovpn_crypto_state *cs, diff --git a/drivers/net/ovpn/crypto_aead.c b/drivers/net/ovpn/crypto_aead.c index 74eaf6fac2f52..77ae958a971a5 100644 --- a/drivers/net/ovpn/crypto_aead.c +++ b/drivers/net/ovpn/crypto_aead.c @@ -391,8 +391,13 @@ static void ovpn_aead_crypto_key_slot_free_work(struct work_struct *work) { struct ovpn_crypto_key_slot *ks; - ks = container_of(to_rcu_work(work), struct ovpn_crypto_key_slot, - free_work); + ks = container_of(work, struct ovpn_crypto_key_slot, free_work); + /* Reaching this worker means every reference held by packet processing + * and asynchronous crypto has been returned. Separately wait for any + * RCU-only reader which observed the slot before it was unpublished. + */ + cond_synchronize_rcu(ks->rcu_state); + percpu_ref_exit(&ks->refcount); ovpn_aead_crypto_key_slot_free(ks); kfree(ks); } @@ -427,8 +432,7 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc) ks->encrypt = NULL; ks->decrypt = NULL; - INIT_RCU_WORK(&ks->free_work, ovpn_aead_crypto_key_slot_free_work); - kref_init(&ks->refcount); + INIT_WORK(&ks->free_work, ovpn_aead_crypto_key_slot_free_work); ks->key_id = kc->key_id; ks->encrypt = ovpn_aead_init("encrypt", alg_name, @@ -458,6 +462,11 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc) ovpn_pktid_xmit_init(&ks->pid_xmit); ovpn_pktid_recv_init(&ks->pid_recv); + ret = percpu_ref_init(&ks->refcount, ovpn_crypto_key_slot_release, 0, + GFP_KERNEL); + if (ret < 0) + goto destroy_ks; + return ks; destroy_ks: From patchwork Mon Oct 12 16:33:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Baffo X-Patchwork-Id: 5463 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7000:20d9:b0:8d7:ebbe:b250 with SMTP id f25csp412860mad; Mon, 12 Oct 2026 09:34:08 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvBxh+0QOF5GnxboagP8eX6njjGMe3yAhFgaMXUqqFgp+LnbQN2Gg/I6QykMmgQWX/bWYyFZr0ErkX6M=@openvpn.net X-Received: by 2002:a05:6808:238b:b0:4f5:e13:95e2 with SMTP id 5614622812f47-50c438a3eacmr10003508b6e.9.1791822848086; Mon, 12 Oct 2026 09:34:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1791822848; cv=none; d=google.com; s=arc-20260327; b=Y/AbJDP1qOq9vkWhHk7NDwifoABgGTyktifK75QggLoarQUXAzx8to9Z7j57HpuiHM nVmfRcX9atqGdC5djA5pThWC4TZ+g0RDYZ+QsGl45lg614eAWJP8/oHeuqA1O60pqJdw BV5jvJHC3WPAKIXHOQGXHVQUfJCXrMvDeNb1gmN30p3eeYvXFZlSZEGTCBVYNpYJnjtA RULCFP+0SIpBA8xOog3a/LsgQdg+OlCMhWIPdh1lbCWrhmyDpEZNGkDva1TZYecXeHe1 4paFqIadDXH8DcV7cvd4GzPD/Ory4Mf9SlGRRiRBX7fQxWsOhIiOE51Kd+jFnz3DrYzP xCVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=Sf/GHWOnQpSh+RRhV1ErLYEN7gj6AF4AUsT/a+rBH2M=; fh=4NbAC/LsuMLI0S0hprUlLSLCiHwg6SCAifhH718Jh0Q=; b=mhD4Ef5SUri4yIO2O8zdDViFvEULoV8vZnhxawNLCszMwZOZ5aO9YGyPpqc8ktRKQ2 pRxCTwcQ+glML3u3EiqUlIS3BgW3T+2z+8e+RPFn0REJSOWJapQM0HhvQ0Me26Hnb/KC JtU94cnnVnqGZQlAou6xL1vypagV54K4rNlO1D2tjcJiT33e+VfR5P+5xZbbHk3LiLru FZb0+/rbf/FiwMp8SBLWPgnDFLT0cOwqFYhLr83Q8UNSciGKZIIJtd3SbMskFoE86Me0 zBDt/WqPCRwqGHJxBy1BAOwUyKm1vL5pLnvwHKF8hfqPd08r2p5kCUj9sCvwpIdTeDMh q8OA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b="C/GNE0rP"; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=fbUz5cxw; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=ZFNgrNu6; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=gbUFabeh; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id 586e51a60fabf-4a2a86e7d7csi15002872fac.321.2026.10.12.09.34.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Oct 2026 09:34:08 -0700 (PDT) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b="C/GNE0rP"; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=fbUz5cxw; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=ZFNgrNu6; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=gbUFabeh; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Content-Transfer-Encoding:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Subject:MIME-Version:References:In-Reply-To:Message-ID:Date:To:From:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Sf/GHWOnQpSh+RRhV1ErLYEN7gj6AF4AUsT/a+rBH2M=; b=C/GNE0rPxA0ZRgr6cpMdegTjrP sXMwKZLyNRTcX/WPE2n2eNDxA51rbJXJl7Gc4uqEoEaXR2cPoD5ogqPFg8jK48qLTbFMaT2Ico1xQ 3duBhBXRD1125yXt93Pi116mHOddtGnY3eAX8jXqBLNqW4JkNnMv5TnOQnWFBEes6rtA=; Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1xGIyS-0008Uo-J0; Mon, 12 Oct 2026 16:34:02 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1xGIyO-0008US-Jh for openvpn-devel@lists.sourceforge.net; Mon, 12 Oct 2026 16:33:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=3NpB9my/ufZs150iOuC3+a/gJJfOh/X8bLErTNrG/9Y=; b=fbUz5cxwPZ2WLcIRhSlG0UseqP 13UAFjkl8NEnkhYvSYcgESl9t8V117AByZlUGmxW/5n7x5jtEl0EhchcWd1vx72z7G4sFq7iXmKuv dE+8YXXqygfbvWVvaNxuo4FRZgfF7BYcu67S5xYhhCwmk/d7M8QYhT1SMmYtkVGdQFi4=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3NpB9my/ufZs150iOuC3+a/gJJfOh/X8bLErTNrG/9Y=; b=ZFNgrNu6OqXIANGSdNSPwOD6dy KmOwXB/Plu25RhHFfmLM/mrI9cPtB9p6RD7nGk+jw8eOulYrHyNjR4nQ2Dp7CBzssfDTo7gVweWRz eWE0EV6GyvpN2Ku6HitZqzz3gBSApJgvRIytqh8DvRdrpzIYOIHYY5iP3p4torBBjYl8=; Received: from mout-b-203.mailbox.org ([195.10.208.52]) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1xGIyK-00017H-5s for openvpn-devel@lists.sourceforge.net; Mon, 12 Oct 2026 16:33:57 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:b231:465::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519MLKEM768 server-signature RSA-PSS (4096 bits) server-digest SHA512) (No client certificate requested) by mout-b-203.mailbox.org (Postfix) with ESMTPS id 4j3NMJ1SWwzLmFC; Mon, 12 Oct 2026 18:33:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandelbit.com; s=MBO0001; t=1791822828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3NpB9my/ufZs150iOuC3+a/gJJfOh/X8bLErTNrG/9Y=; b=gbUFabeh1U6fUXwjGTTiNAd62uv1O8T11IkTx1uTFHBx7ETNtUA49wSPJeu3XXOQyZl2Aq PNQmbI21ABY2lS/erJORdkWvnGaSvfx3MNbsTnZir/dcUYAiiYqxc6VzHdEBb5DdYTzEsI YWEdgqiJBLjPBYfn5UBANXEIBBYnI3yKx7b5k+n/1Hep2u/3dxKZ1kyt1Xz8Erre/rp6XQ EH4fhpxcGR4TxYzEgwRPQs23YLub5vJtJEnU4X7Kf5nRbwcfH2nXem9SEXAIoKPfrx4w/c tp/aNefCd7RU5WYQHuQ8NrUYcM5/a0lDF9Ow0Lt6CZtQSebGwQ68ZeC/UVunrA== Authentication-Results: outgoing_mbo_mout; dkim=none; spf=pass (outgoing_mbo_mout: domain of marco@mandelbit.com designates 2001:67c:2050:b231:465::1 as permitted sender) smtp.mailfrom=marco@mandelbit.com From: Marco Baffo To: openvpn-devel@lists.sourceforge.net Date: Mon, 12 Oct 2026 18:33:33 +0200 Message-ID: <20261012163333.60389-4-marco@mandelbit.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261012163333.60389-1-marco@mandelbit.com> References: <20261012163333.60389-1-marco@mandelbit.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4j3NMJ1SWwzLmFC X-Spam-Score: -0.2 (/) X-Spam-Report: Spam detection software, running on the system "sfi-spamd-1.hosts.colo.sdot.me", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Per-peer VPN and transport counters use atomic64 operations on every packet, causing cacheline contention when CPUs share a peer. Replace these counters with pcpu_sw_netstats and aggregate them when reporting peers over netlink. Keep the statistics pointers in a separate cacheline to avoid false sharing with frequently updated p [...] Content analysis details: (-0.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1xGIyK-00017H-5s Subject: [Openvpn-devel] [PATCH ovpn net-next v3 3/3] ovpn: use per-CPU peer statistics X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: 1878862435118825732 X-GMAIL-MSGID: 1878862435118825732 Per-peer VPN and transport counters use atomic64 operations on every packet, causing cacheline contention when CPUs share a peer. Replace these counters with pcpu_sw_netstats and aggregate them when reporting peers over netlink. Keep the statistics pointers in a separate cacheline to avoid false sharing with frequently updated peer fields. Disable BHs around the link RX update to cover TCP strparser callbacks running in process context. Disable BHs around the link TX statistics update in ovpn_encrypt_post(), since some hardware crypto drivers, such as HiSilicon SEC2, invoke completion callbacks with BHs enabled. Measurements on a laptop with a Ryzen 7 PRO 7840U, with four concurrent iperf3 streams on separate cores using the same ovpn peer, showed statistics HITM samples dropping from 8-12 to zero for UDP and 16-18 to zero for TCP. UDP cycles per packet fell from 26,389 to 26,339 (-0.19%) and TCP cycles per segment fell from 18,920 to 18,861 (-0.31%). The peer and key-slot cacheline layout changes reduce total HITM samples by up to 75% for UDP and up to 63% for TCP. Signed-off-by: Marco Baffo --- Changes in v3: - Disable BHs around the link TX statistics update in ovpn_encrypt_post() to handle crypto completions running with BHs enabled. Changes in v2: - Replace the cacheline static_assert() with DEBUG_NET_WARN_ON_ONCE() during key-slot creation, allowing builds with 16-byte or 32-byte cache lines. The warning is enabled only with CONFIG_DEBUG_NET. - Update the layout comment to describe fitting within one cacheline as a goal rather than a guarantee. drivers/net/ovpn/Makefile | 1 - drivers/net/ovpn/crypto.h | 8 ++++- drivers/net/ovpn/crypto_aead.c | 3 ++ drivers/net/ovpn/io.c | 11 +++--- drivers/net/ovpn/netlink.c | 20 ++++++----- drivers/net/ovpn/peer.c | 26 ++++++++++---- drivers/net/ovpn/peer.h | 22 ++++++++---- drivers/net/ovpn/stats.c | 21 ------------ drivers/net/ovpn/stats.h | 63 ++++++++++++++++++++-------------- drivers/net/ovpn/tcp.c | 3 +- 10 files changed, 104 insertions(+), 74 deletions(-) delete mode 100644 drivers/net/ovpn/stats.c diff --git a/drivers/net/ovpn/Makefile b/drivers/net/ovpn/Makefile index 229be66167e1f..ebea3a24336f7 100644 --- a/drivers/net/ovpn/Makefile +++ b/drivers/net/ovpn/Makefile @@ -17,6 +17,5 @@ ovpn-y += netlink-gen.o ovpn-y += peer.o ovpn-y += pktid.o ovpn-y += socket.o -ovpn-y += stats.o ovpn-y += tcp.o ovpn-y += udp.o diff --git a/drivers/net/ovpn/crypto.h b/drivers/net/ovpn/crypto.h index a0209b1280de4..3e64ac3a05c32 100644 --- a/drivers/net/ovpn/crypto.h +++ b/drivers/net/ovpn/crypto.h @@ -46,12 +46,18 @@ struct ovpn_crypto_key_slot { struct crypto_aead *decrypt; u8 nonce_tail_xmit[OVPN_NONCE_TAIL_SIZE]; u8 nonce_tail_recv[OVPN_NONCE_TAIL_SIZE]; + struct percpu_ref refcount; + + /* The fields up to refcount are read on every packet but written only when + * the key is installed or torn down, so ideally they fit in one cacheline. + * pid_recv and pid_xmit are written on every packet and get cachelines of + * their own. Keep per-packet writes out of the first block. + */ struct ovpn_pktid_recv pid_recv ____cacheline_aligned_in_smp; struct ovpn_pktid_xmit pid_xmit ____cacheline_aligned_in_smp; struct work_struct free_work; unsigned long rcu_state; - struct percpu_ref refcount; }; struct ovpn_crypto_state { diff --git a/drivers/net/ovpn/crypto_aead.c b/drivers/net/ovpn/crypto_aead.c index 77ae958a971a5..34cea87c0ce7d 100644 --- a/drivers/net/ovpn/crypto_aead.c +++ b/drivers/net/ovpn/crypto_aead.c @@ -409,6 +409,9 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc) const char *alg_name; int ret; + DEBUG_NET_WARN_ON_ONCE(offsetofend(struct ovpn_crypto_key_slot, refcount) > + SMP_CACHE_BYTES); + /* validate crypto alg */ switch (kc->cipher_alg) { case OVPN_CIPHER_ALG_AES_GCM: diff --git a/drivers/net/ovpn/io.c b/drivers/net/ovpn/io.c index 15072ce960fa4..7f313dc3449a0 100644 --- a/drivers/net/ovpn/io.c +++ b/drivers/net/ovpn/io.c @@ -99,7 +99,7 @@ static void ovpn_netdev_write(struct ovpn_peer *peer, struct sk_buff *skb) ret = gro_cells_receive(&peer->ovpn->gro_cells, skb); if (likely(ret == NET_RX_SUCCESS)) { /* update RX stats with the size of decrypted packet */ - ovpn_peer_stats_increment_rx(&peer->vpn_stats, pkt_len); + ovpn_peer_stats_increment_rx(peer->vpn_stats, pkt_len); dev_dstats_rx_add(peer->ovpn->dev, pkt_len); } local_bh_enable(); @@ -216,7 +216,7 @@ void ovpn_recv(struct ovpn_peer *peer, struct sk_buff *skb) struct ovpn_crypto_key_slot *ks; u8 key_id; - ovpn_peer_stats_increment_rx(&peer->link_stats, skb->len); + ovpn_peer_stats_increment_rx(peer->link_stats, skb->len); /* get the key slot matching the key ID in the received packet */ key_id = ovpn_key_id_from_skb(skb); @@ -292,7 +292,10 @@ void ovpn_encrypt_post(void *data, int ret) goto err_unlock; } - ovpn_peer_stats_increment_tx(&peer->link_stats, orig_len); + /* some crypto backends invoke completion with BHs enabled */ + local_bh_disable(); + ovpn_peer_stats_increment_tx(peer->link_stats, orig_len); + local_bh_enable(); /* keep track of last sent packet for keepalive */ WRITE_ONCE(peer->last_sent, ktime_get_boottime_seconds()); /* skb passed down the stack - don't free it */ @@ -428,7 +431,7 @@ netdev_tx_t ovpn_net_xmit(struct sk_buff *skb, struct net_device *dev) } skb_list.prev->next = NULL; - ovpn_peer_stats_increment_tx(&peer->vpn_stats, tx_bytes); + ovpn_peer_stats_increment_tx(peer->vpn_stats, tx_bytes); ovpn_send(ovpn, skb_list.next, peer); return NETDEV_TX_OK; diff --git a/drivers/net/ovpn/netlink.c b/drivers/net/ovpn/netlink.c index 8cdf46b611425..10a277a0f96af 100644 --- a/drivers/net/ovpn/netlink.c +++ b/drivers/net/ovpn/netlink.c @@ -550,6 +550,7 @@ static int ovpn_nl_send_peer(struct sk_buff *skb, const struct genl_info *info, const struct ovpn_peer *peer, u32 portid, u32 seq, int flags) { + struct rtnl_link_stats64 vpn_stats = {}, link_stats = {}; const struct ovpn_bind *bind; struct ovpn_socket *sock; int ret = -EMSGSIZE; @@ -631,27 +632,30 @@ static int ovpn_nl_send_peer(struct sk_buff *skb, const struct genl_info *info, } rcu_read_unlock(); + dev_fetch_sw_netstats(&vpn_stats, peer->vpn_stats); + dev_fetch_sw_netstats(&link_stats, peer->link_stats); + if (nla_put_net16(skb, OVPN_A_PEER_LOCAL_PORT, local_port) || /* VPN RX stats */ nla_put_uint(skb, OVPN_A_PEER_VPN_RX_BYTES, - atomic64_read(&peer->vpn_stats.rx.bytes)) || + vpn_stats.rx_bytes) || nla_put_uint(skb, OVPN_A_PEER_VPN_RX_PACKETS, - atomic64_read(&peer->vpn_stats.rx.packets)) || + vpn_stats.rx_packets) || /* VPN TX stats */ nla_put_uint(skb, OVPN_A_PEER_VPN_TX_BYTES, - atomic64_read(&peer->vpn_stats.tx.bytes)) || + vpn_stats.tx_bytes) || nla_put_uint(skb, OVPN_A_PEER_VPN_TX_PACKETS, - atomic64_read(&peer->vpn_stats.tx.packets)) || + vpn_stats.tx_packets) || /* link RX stats */ nla_put_uint(skb, OVPN_A_PEER_LINK_RX_BYTES, - atomic64_read(&peer->link_stats.rx.bytes)) || + link_stats.rx_bytes) || nla_put_uint(skb, OVPN_A_PEER_LINK_RX_PACKETS, - atomic64_read(&peer->link_stats.rx.packets)) || + link_stats.rx_packets) || /* link TX stats */ nla_put_uint(skb, OVPN_A_PEER_LINK_TX_BYTES, - atomic64_read(&peer->link_stats.tx.bytes)) || + link_stats.tx_bytes) || nla_put_uint(skb, OVPN_A_PEER_LINK_TX_PACKETS, - atomic64_read(&peer->link_stats.tx.packets))) + link_stats.tx_packets)) goto err; nla_nest_end(skb, attr); diff --git a/drivers/net/ovpn/peer.c b/drivers/net/ovpn/peer.c index 970ca7232ae45..82998051db916 100644 --- a/drivers/net/ovpn/peer.c +++ b/drivers/net/ovpn/peer.c @@ -115,30 +115,42 @@ struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id) RCU_INIT_POINTER(peer->bind, NULL); ovpn_crypto_state_init(&peer->crypto); spin_lock_init(&peer->lock); - ovpn_peer_stats_init(&peer->vpn_stats); - ovpn_peer_stats_init(&peer->link_stats); INIT_WORK(&peer->keepalive_work, ovpn_peer_keepalive_send); + ret = -ENOMEM; + peer->vpn_stats = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats); + if (!peer->vpn_stats) + goto err_free_peer; + + peer->link_stats = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats); + if (!peer->link_stats) + goto err_free_stats; + ret = dst_cache_init(&peer->dst_cache, GFP_KERNEL); if (ret < 0) { netdev_err(ovpn->dev, "cannot initialize dst cache for peer %u\n", peer->id); - kfree(peer); - return ERR_PTR(ret); + goto err_free_stats; } ret = percpu_ref_init(&peer->refcount, ovpn_peer_release_ref, 0, GFP_KERNEL); if (ret < 0) { dst_cache_destroy(&peer->dst_cache); - kfree(peer); - return ERR_PTR(ret); + goto err_free_stats; } netdev_hold(ovpn->dev, &peer->dev_tracker, GFP_KERNEL); return peer; + +err_free_stats: + free_percpu(peer->link_stats); + free_percpu(peer->vpn_stats); +err_free_peer: + kfree(peer); + return ERR_PTR(ret); } /** @@ -357,6 +369,8 @@ static void ovpn_peer_release_rcu(struct rcu_head *head) * perform it in the RCU callback, when all contexts are done */ dst_cache_destroy(&peer->dst_cache); + free_percpu(peer->link_stats); + free_percpu(peer->vpn_stats); percpu_ref_exit(&peer->refcount); kfree(peer); } diff --git a/drivers/net/ovpn/peer.h b/drivers/net/ovpn/peer.h index 489bf26466847..550f600ac120a 100644 --- a/drivers/net/ovpn/peer.h +++ b/drivers/net/ovpn/peer.h @@ -10,6 +10,7 @@ #ifndef _NET_OVPN_OVPNPEER_H_ #define _NET_OVPN_OVPNPEER_H_ +#include #include #include #include @@ -53,14 +54,14 @@ * @keepalive_timeout: seconds after which an inactive peer is considered dead * @keepalive_recv_exp: future timestamp when the peer should expire * @last_recv: timestamp of the last authenticated received packet - * @vpn_stats: per-peer in-VPN TX/RX stats - * @link_stats: per-peer link/transport TX/RX stats * @delete_reason: why peer was deleted (i.e. timeout, transport error, ..) * @lock: protects binding to peer (bind) and keepalive* fields - * @refcount: reference counter * @rcu: used to free peer in an RCU safe way * @release_entry: entry for the socket release list * @keepalive_work: used to schedule keepalive sending + * @refcount: reference counter + * @vpn_stats: per-CPU in-VPN TX/RX stats + * @link_stats: per-CPU link/transport TX/RX stats */ struct ovpn_peer { struct ovpn_priv *ovpn; @@ -107,14 +108,23 @@ struct ovpn_peer { unsigned long keepalive_timeout; unsigned long keepalive_recv_exp; time64_t last_recv; - struct ovpn_peer_stats vpn_stats; - struct ovpn_peer_stats link_stats; enum ovpn_del_peer_reason delete_reason; spinlock_t lock; /* protects bind and keepalive* */ - struct percpu_ref refcount; struct rcu_head rcu; struct llist_node release_entry; struct work_struct keepalive_work; + + /* Read on the packet path but written only at setup + * and teardown: keep away from per-packet writes. + */ + /* private: */ + __cacheline_group_begin_aligned(read_mostly); + /* public: */ + struct pcpu_sw_netstats __percpu *vpn_stats; + struct pcpu_sw_netstats __percpu *link_stats; + struct percpu_ref refcount; + /* private: */ + __cacheline_group_end_aligned(read_mostly); }; /** diff --git a/drivers/net/ovpn/stats.c b/drivers/net/ovpn/stats.c deleted file mode 100644 index d637143473bb9..0000000000000 --- a/drivers/net/ovpn/stats.c +++ /dev/null @@ -1,21 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* OpenVPN data channel offload - * - * Copyright (C) 2020-2025 OpenVPN, Inc. - * - * Author: James Yonan - * Antonio Quartulli - */ - -#include - -#include "stats.h" - -void ovpn_peer_stats_init(struct ovpn_peer_stats *ps) -{ - atomic64_set(&ps->rx.bytes, 0); - atomic64_set(&ps->rx.packets, 0); - - atomic64_set(&ps->tx.bytes, 0); - atomic64_set(&ps->tx.packets, 0); -} diff --git a/drivers/net/ovpn/stats.h b/drivers/net/ovpn/stats.h index 3a45b97c00568..b342c952fce65 100644 --- a/drivers/net/ovpn/stats.h +++ b/drivers/net/ovpn/stats.h @@ -13,37 +13,48 @@ #include -/* one stat */ -struct ovpn_peer_stat { - atomic64_t bytes; - atomic64_t packets; -}; - -/* rx and tx stats combined */ -struct ovpn_peer_stats { - struct ovpn_peer_stat rx; - struct ovpn_peer_stat tx; -}; - -void ovpn_peer_stats_init(struct ovpn_peer_stats *ps); - -static inline void ovpn_peer_stats_increment(struct ovpn_peer_stat *stat, - const unsigned int n) +/** + * ovpn_peer_stats_increment_rx - account one received packet + * @stats: per-CPU counters + * @n: received bytes + * + * Context: Softirq or task context with BHs disabled. + * Not callable from hardirq or NMI context. + */ +static inline void +ovpn_peer_stats_increment_rx(struct pcpu_sw_netstats __percpu *stats, + const unsigned int n) { - atomic64_add(n, &stat->bytes); - atomic64_inc(&stat->packets); -} + struct pcpu_sw_netstats *s; -static inline void ovpn_peer_stats_increment_rx(struct ovpn_peer_stats *stats, - const unsigned int n) -{ - ovpn_peer_stats_increment(&stats->rx, n); + lockdep_assert_in_softirq(); + s = this_cpu_ptr(stats); + u64_stats_update_begin(&s->syncp); + u64_stats_add(&s->rx_bytes, n); + u64_stats_inc(&s->rx_packets); + u64_stats_update_end(&s->syncp); } -static inline void ovpn_peer_stats_increment_tx(struct ovpn_peer_stats *stats, - const unsigned int n) +/** + * ovpn_peer_stats_increment_tx - account one transmitted packet + * @stats: per-CPU counters + * @n: transmitted bytes + * + * Context: Softirq or task context with BHs disabled. + * Not callable from hardirq or NMI context. + */ +static inline void +ovpn_peer_stats_increment_tx(struct pcpu_sw_netstats __percpu *stats, + const unsigned int n) { - ovpn_peer_stats_increment(&stats->tx, n); + struct pcpu_sw_netstats *s; + + lockdep_assert_in_softirq(); + s = this_cpu_ptr(stats); + u64_stats_update_begin(&s->syncp); + u64_stats_add(&s->tx_bytes, n); + u64_stats_inc(&s->tx_packets); + u64_stats_update_end(&s->syncp); } static inline void ovpn_dev_dstats_tx_dropped(struct net_device *dev) diff --git a/drivers/net/ovpn/tcp.c b/drivers/net/ovpn/tcp.c index 47af3a62f3551..ed2984e80ef77 100644 --- a/drivers/net/ovpn/tcp.c +++ b/drivers/net/ovpn/tcp.c @@ -143,8 +143,9 @@ static void ovpn_tcp_rcv(struct strparser *strp, struct sk_buff *skb) * this peer */ ovpn_peer_hold(peer); - + local_bh_disable(); ovpn_recv(peer, skb); + local_bh_enable(); return; err: /* take reference for deferred peer deletion */