From patchwork Fri Sep 18 16:03:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ralf Lici X-Patchwork-Id: 5388 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7000:6446:b0:8a0:ea1f:253a with SMTP id n6csp8968931mag; Fri, 18 Sep 2026 09:04:30 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvBxTrBC8lBYvn/uj7b9P8W+3YfbtJXFiJSTfegkK48mdZf5ApMQqkz4bQVcr9DbsSAEpsHVL4ZnRolI=@openvpn.net X-Received: by 2002:a05:6820:80a:b0:6b7:83d6:2920 with SMTP id 006d021491bc7-6ca9c650d72mr2911380eaf.35.1789747470249; Fri, 18 Sep 2026 09:04:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1789747470; cv=none; d=google.com; s=arc-20260327; b=Ytzn5nENw2KCLT9pupfPLHAdL0x/RBaE7EVMz98m2Ph7Yub29ufpC4WXb5AyACnDC3 6tm8UwdSgtWYzx1aeo2pxEjPqTtfgAvVDJqjQFKsQpSg2ucApUfg2Oi5xJ1+sEx4UXYK 3FJnWrUQ5R+G3yxtNAERCtvBnajJGBo9J+iz2pLh4AwthGHazA0/3cPRaV4B8GA43raK G/y7xF6X9uaHMSQ1QngngMIJ2pTVUpIUjbvzphlUnsqJhM2qtGTnOK4ODlRoYY63rCU3 hhC5/vSsp5b/ANSEICAKDxGaQg06TamT3v8/SHVjYOaW2mkuAZkoRnwCxmJXAyn0diKH zFdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=98iLrjGLhYDO7xvYyzKPwp9RkZ+yEbM4mpK8mftLLOQ=; fh=4NbAC/LsuMLI0S0hprUlLSLCiHwg6SCAifhH718Jh0Q=; b=EFdExdr4izSSRM0v8gusaXZEjYHujD1YDkrvsfY4+/5Oudj8XyHfFppxgGAflLifqi w53WR/cPqY7gVpSDfnlD1ffoVCB9ZmN0njiH3BqpvTdQ+2E/7JyLV+gk2Yu0D8Q4CupB yiIGkbHRfrfszmN0ltxC5GLacyCJc9GIFGCM3mz7fHQhhZ3EqhnNs1dsIiheIMyfwMtg 2euvW1yq/OnwjA8tF8nt6/5NeWuVZhkMAlpq6Mgxlu9t3tZtsJQfxHG13yO7IyyqylsI CEWRcrybgXdV/zVCe/l4YZx0mNy2T+zwl60XXgGovk7ynJ/saOhen416whihYwvFMkR5 5BZw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=idN0iFau; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=DUwR3elH; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=f2BNS6ic; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=ipkYqMUn; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id 586e51a60fabf-48739da67e8si2557824fac.182.2026.09.18.09.04.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Sep 2026 09:04:30 -0700 (PDT) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=idN0iFau; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=DUwR3elH; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=f2BNS6ic; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=ipkYqMUn; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Content-Transfer-Encoding:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Subject:MIME-Version:References:In-Reply-To:Message-ID:Date:To:From:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=98iLrjGLhYDO7xvYyzKPwp9RkZ+yEbM4mpK8mftLLOQ=; b=idN0iFauSOml9KqdoSJhOXPpkf 24wiaI5ajf01K/CJoSHU7wGn3H5bv80SAmcgtJXF/3vdH0EjzvbD9wUCm3CZte6388JA8lKI/3Gx7 fP3dUkZxTCSiBnOLDWE0A5tZK9zmD6Zoe1mtJOLNfg86jQInnz2pxD2C0CWwCxSBFa58=; Received: from [127.0.0.1] (helo=sfs-ml-3.v29.lw.sourceforge.com) by sfs-ml-3.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1x7b4b-0008Mr-A5; Fri, 18 Sep 2026 16:04:26 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-3.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1x7b4Z-0008Mh-J5 for openvpn-devel@lists.sourceforge.net; Fri, 18 Sep 2026 16:04:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:To:From:Sender:Reply-To:Cc:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=n4EfA2fQrBgpsYY1FMRLxmafR6qIX9X/ghS1fPplJ+A=; b=DUwR3elHyPKHlcl7rXwvdPGo3R KGlW9IfkcOoJjf+D9oDU9EneQU9+HviLDNWr7sJTFy3JDLF7nILOb6t7iGnHtjI/iRvSepTZfPIdK 0CKLMoNhj5ImjM8skaJnmm1Nz0uEjmJsWA3Zv9pazUgPpQiiWqs/XHOKu96s2GgChsWA=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID: Date:Subject:To:From:Sender:Reply-To:Cc:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=n4EfA2fQrBgpsYY1FMRLxmafR6qIX9X/ghS1fPplJ+A=; b=f2BNS6icZwXKTfaeQNM66BfNkP 1KJCSHYkQm6ZfGjI2pdB/MQW/USVWNGT9lnnG5/mnMSIN8G05u+7JRuY5euT9X5pLDARlgPNqAz0w 510xHJaF76a582reqgBRsNp6KX1HGMTxW4ESjC53x5+Oyn2xw9o8+LFUmiJpkm3EnpAA=; Received: from mout-b-203.mailbox.org ([195.10.208.52]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1x7b4X-0006St-9n for openvpn-devel@lists.sourceforge.net; Fri, 18 Sep 2026 16:04:24 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:b231:465::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-b-203.mailbox.org (Postfix) with ESMTPS id 4hmcrD43FNzLmFv for ; Fri, 18 Sep 2026 18:04:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandelbit.com; s=MBO0001; t=1789747452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n4EfA2fQrBgpsYY1FMRLxmafR6qIX9X/ghS1fPplJ+A=; b=ipkYqMUnQcJZidx1r6EEhOMRzGbswDT+tDaVRGUwvSilwLUuBVzfZpTnioK/Kj2c1Soswe cpuCG5APZWdvoUCC2OcS1d97XVNhiPIcdWmXnKwP92Kq6qTiqtplDevAHlakXIP09aqG7t tZKyWfzvewZvGbMkTNIFfKGjGlWNjMyUWvM9K8nVqQnKlYEqe/FD9pV1VKg21xD38c2gZT NLUZ0Egg7FGvZARt80J2ERhcgRvc+kpKHr6CdsIAFRiz5T4/4bj42dwYA9djOY6P3GKqyj LxDdb+J1Mik+x7ADFftuz31nfZnKyeS3pyxRYNahj9RCaHS1FTCKbHQezTXZEA== Authentication-Results: outgoing_mbo_mout; dkim=none; spf=pass (outgoing_mbo_mout: domain of ralf@mandelbit.com designates 2001:67c:2050:b231:465::1 as permitted sender) smtp.mailfrom=ralf@mandelbit.com From: Ralf Lici To: openvpn-devel@lists.sourceforge.net Date: Fri, 18 Sep 2026 18:03:35 +0200 Message-ID: <3769e52eb5b9af578f1f67520882ceaf40537c39.1789746543.git.ralf@mandelbit.com> In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 4hmcrD43FNzLmFv X-Spam-Score: -0.2 (/) X-Spam-Report: Spam detection software, running on the system "sfi-spamd-2.hosts.colo.sdot.me", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Each packet holds its peer across packet processing and asynchronous crypto completion. When traffic for one peer is spread across several CPUs, the shared kref cache line therefore moves between thos [...] Content analysis details: (-0.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid 0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [195.10.208.52 listed in wl.mailspike.net] X-Headers-End: 1x7b4X-0006St-9n Subject: [Openvpn-devel] [PATCH ovpn net-next v2 1/2] ovpn: use percpu references for peers X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: 1876686243295978375 X-GMAIL-MSGID: 1876686243295978375 Each packet holds its peer across packet processing and asynchronous crypto completion. When traffic for one peer is spread across several CPUs, the shared kref cache line therefore moves between those CPUs for every packet. Replace the peer kref with percpu_ref and kill the initial reference exactly once after the peer has been removed from its lookup structures and detached from its socket. Keep the existing RCU-delayed destruction and release the percpu_ref storage in the final RCU callback. Use an unconditional percpu_ref_get where the caller already owns a peer reference or otherwise excludes teardown. Such callers must remain able to extend their existing ownership while the killed reference drains. RCU-protected lookup paths instead use percpu_ref_tryget_live_rcu, so they stop admitting new users as soon as teardown starts. This deliberately makes stale TCP socket lookups fail earlier than kref_get_unless_zero, which continued succeeding until the count reached zero. percpu_ref_init can fail, so propagate allocation failure from peer creation. This trades a per-CPU counter allocation for a cheaper live data path on these long-lived, heavily shared objects. Specifically, in terms of memory, this costs: 8 bytes per possible CPU for the counters, a 56-byte control object, and 8 additional bytes in struct ovpn_peer. In a 32-stream test using one peer and one key, perf c2c placed the peer kref cacheline second among system-wide shared cachelines, with 37 sampled HITM loads attributed to the locked reference update. After this change that shared reference line was no longer sampled, while the untouched key-slot kref addressed by the next commit remained visible. RX used a single receive queue, so only TX exercised CPU fan-out in this test. Signed-off-by: Ralf Lici --- Changes since v1 https://lore.kernel.org/openvpn-devel/da742f32b00e7a6872062ce36a3dc4add665f189.1789658051.git.ralf@mandelbit.com/ - Use unconditional percpu_ref_get where the caller already owns the peer or otherwise excludes teardown and percpu_ref_tryget_live_rcu for TCP socket lookups under RCU. (Sashiko) - Make the percpu-ref release callback private to peer.c. drivers/net/ovpn/io.c | 6 +----- drivers/net/ovpn/netlink.c | 6 +++--- drivers/net/ovpn/peer.c | 37 +++++++++++++++++++++++-------------- drivers/net/ovpn/peer.h | 37 +++++++++++++++++++++++++++++++------ drivers/net/ovpn/tcp.c | 23 +++++++++++------------ 5 files changed, 69 insertions(+), 40 deletions(-) diff --git a/drivers/net/ovpn/io.c b/drivers/net/ovpn/io.c index 9526f8096da6..15072ce960fa 100644 --- a/drivers/net/ovpn/io.c +++ b/drivers/net/ovpn/io.c @@ -321,11 +321,7 @@ static bool ovpn_encrypt_one(struct ovpn_peer *peer, struct sk_buff *skb) /* take a reference to the peer because the crypto code may run async. * ovpn_encrypt_post() will release it upon completion */ - if (unlikely(!ovpn_peer_hold(peer))) { - DEBUG_NET_WARN_ON_ONCE(1); - ovpn_crypto_key_slot_put(ks); - return false; - } + ovpn_peer_hold(peer); memset(ovpn_skb_cb(skb), 0, sizeof(struct ovpn_cb)); ovpn_encrypt_post(skb, ovpn_aead_encrypt(peer, ks, skb)); diff --git a/drivers/net/ovpn/netlink.c b/drivers/net/ovpn/netlink.c index 4dad85294198..8cdf46b61142 100644 --- a/drivers/net/ovpn/netlink.c +++ b/drivers/net/ovpn/netlink.c @@ -463,11 +463,11 @@ int ovpn_nl_peer_new_doit(struct sk_buff *skb, struct genl_info *info) ovpn_socket_release(peer); peer_release: /* For UDP, the peer is unreachable until added to the hashtables, so - * dropping the initial reference is enough. For TCP, the peer may be + * killing the initial reference is enough. For TCP, the peer may be * concurrently reachable via sk_user_data->peer until - * ovpn_socket_release() detaches; rely on the refcount. + * ovpn_socket_release() detaches; rely on the percpu reference. */ - ovpn_peer_put(peer); + ovpn_peer_kill(peer); return ret; } diff --git a/drivers/net/ovpn/peer.c b/drivers/net/ovpn/peer.c index c95656ca7c35..970ca7232ae4 100644 --- a/drivers/net/ovpn/peer.c +++ b/drivers/net/ovpn/peer.c @@ -22,6 +22,8 @@ #include "peer.h" #include "socket.h" +static void ovpn_peer_release_ref(struct percpu_ref *ref); + static void unlock_ovpn(struct ovpn_priv *ovpn, struct llist_head *release_list) __releases(&ovpn->lock) @@ -33,7 +35,7 @@ static void unlock_ovpn(struct ovpn_priv *ovpn, llist_for_each_entry_safe(peer, next, release_list->first, release_entry) { ovpn_socket_release(peer); - ovpn_peer_put(peer); + ovpn_peer_kill(peer); } } @@ -113,7 +115,6 @@ struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id) RCU_INIT_POINTER(peer->bind, NULL); ovpn_crypto_state_init(&peer->crypto); spin_lock_init(&peer->lock); - kref_init(&peer->refcount); ovpn_peer_stats_init(&peer->vpn_stats); ovpn_peer_stats_init(&peer->link_stats); INIT_WORK(&peer->keepalive_work, ovpn_peer_keepalive_send); @@ -127,6 +128,14 @@ struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id) return ERR_PTR(ret); } + ret = percpu_ref_init(&peer->refcount, ovpn_peer_release_ref, 0, + GFP_KERNEL); + if (ret < 0) { + dst_cache_destroy(&peer->dst_cache); + kfree(peer); + return ERR_PTR(ret); + } + netdev_hold(ovpn->dev, &peer->dev_tracker, GFP_KERNEL); return peer; @@ -348,6 +357,7 @@ static void ovpn_peer_release_rcu(struct rcu_head *head) * perform it in the RCU callback, when all contexts are done */ dst_cache_destroy(&peer->dst_cache); + percpu_ref_exit(&peer->refcount); kfree(peer); } @@ -366,12 +376,12 @@ static void ovpn_peer_release(struct ovpn_peer *peer) } /** - * ovpn_peer_release_kref - callback for kref_put - * @kref: the kref object belonging to the peer + * ovpn_peer_release_ref - callback for the peer percpu reference + * @ref: the percpu_ref object belonging to the peer */ -void ovpn_peer_release_kref(struct kref *kref) +static void ovpn_peer_release_ref(struct percpu_ref *ref) { - struct ovpn_peer *peer = container_of(kref, struct ovpn_peer, refcount); + struct ovpn_peer *peer = container_of(ref, struct ovpn_peer, refcount); ovpn_peer_release(peer); } @@ -569,7 +579,7 @@ ovpn_peer_get_by_transp_addr_p2p(struct ovpn_priv *ovpn, rcu_read_lock(); tmp = rcu_dereference(ovpn->peer); if (likely(tmp && ovpn_peer_transp_match(tmp, ss) && - ovpn_peer_hold(tmp))) + ovpn_peer_hold_rcu(tmp))) peer = tmp; rcu_read_unlock(); @@ -610,7 +620,7 @@ struct ovpn_peer *ovpn_peer_get_by_transp_addr(struct ovpn_priv *ovpn, if (!ovpn_peer_transp_match(tmp, &ss)) continue; - if (!ovpn_peer_hold(tmp)) + if (!ovpn_peer_hold_rcu(tmp)) continue; peer = tmp; @@ -641,7 +651,7 @@ static struct ovpn_peer *ovpn_peer_get_by_id_p2p(struct ovpn_priv *ovpn, rcu_read_lock(); tmp = rcu_dereference(ovpn->peer); - if (likely(tmp && tmp->id == peer_id && ovpn_peer_hold(tmp))) + if (likely(tmp && tmp->id == peer_id && ovpn_peer_hold_rcu(tmp))) peer = tmp; rcu_read_unlock(); @@ -671,7 +681,7 @@ struct ovpn_peer *ovpn_peer_get_by_id(struct ovpn_priv *ovpn, u32 peer_id) if (tmp->id != peer_id) continue; - if (!ovpn_peer_hold(tmp)) + if (!ovpn_peer_hold_rcu(tmp)) continue; peer = tmp; @@ -745,7 +755,7 @@ struct ovpn_peer *ovpn_peer_get_by_dst(struct ovpn_priv *ovpn, if (ovpn->mode == OVPN_MODE_P2P) { rcu_read_lock(); peer = rcu_dereference(ovpn->peer); - if (unlikely(peer && !ovpn_peer_hold(peer))) + if (unlikely(peer && !ovpn_peer_hold_rcu(peer))) peer = NULL; rcu_read_unlock(); return peer; @@ -763,7 +773,7 @@ struct ovpn_peer *ovpn_peer_get_by_dst(struct ovpn_priv *ovpn, break; } - if (unlikely(peer && !ovpn_peer_hold(peer))) + if (unlikely(peer && !ovpn_peer_hold_rcu(peer))) peer = NULL; rcu_read_unlock(); @@ -1369,8 +1379,7 @@ static time64_t ovpn_peer_keepalive_work_single(struct ovpn_peer *peer, netdev_dbg(peer->ovpn->dev, "sending keepalive to peer %u\n", peer->id); - if (WARN_ON(!ovpn_peer_hold(peer))) - return 0; + ovpn_peer_hold(peer); if (!queue_work(ovpn_wq, &peer->keepalive_work)) ovpn_peer_put(peer); } diff --git a/drivers/net/ovpn/peer.h b/drivers/net/ovpn/peer.h index dfa5c0037e02..489bf2646684 100644 --- a/drivers/net/ovpn/peer.h +++ b/drivers/net/ovpn/peer.h @@ -10,6 +10,7 @@ #ifndef _NET_OVPN_OVPNPEER_H_ #define _NET_OVPN_OVPNPEER_H_ +#include #include #include @@ -110,7 +111,7 @@ struct ovpn_peer { struct ovpn_peer_stats link_stats; enum ovpn_del_peer_reason delete_reason; spinlock_t lock; /* protects bind and keepalive* */ - struct kref refcount; + struct percpu_ref refcount; struct rcu_head rcu; struct llist_node release_entry; struct work_struct keepalive_work; @@ -120,14 +121,26 @@ struct ovpn_peer { * ovpn_peer_hold - increase reference counter * @peer: the peer whose counter should be increased * - * Return: true if the counter was increased or false if it was zero already + * The caller must already own a peer reference or otherwise guarantee that + * teardown cannot complete while the new reference is acquired. */ -static inline bool ovpn_peer_hold(struct ovpn_peer *peer) +static inline void ovpn_peer_hold(struct ovpn_peer *peer) { - return kref_get_unless_zero(&peer->refcount); + percpu_ref_get(&peer->refcount); } -void ovpn_peer_release_kref(struct kref *kref); +/** + * ovpn_peer_hold_rcu - acquire a live peer reference under RCU + * @peer: peer to acquire + * + * The caller must hold rcu_read_lock. + * + * Return: true if the reference was acquired, false if teardown has started + */ +static inline bool ovpn_peer_hold_rcu(struct ovpn_peer *peer) +{ + return percpu_ref_tryget_live_rcu(&peer->refcount); +} /** * ovpn_peer_put - decrease reference counter @@ -135,7 +148,19 @@ void ovpn_peer_release_kref(struct kref *kref); */ static inline void ovpn_peer_put(struct ovpn_peer *peer) { - kref_put(&peer->refcount, ovpn_peer_release_kref); + percpu_ref_put(&peer->refcount); +} + +/** + * ovpn_peer_kill - drop the peer's initial reference + * @peer: peer which has been unpublished and is being destroyed + * + * This must be called exactly once, after the peer is no longer reachable + * through its owning ovpn instance. + */ +static inline void ovpn_peer_kill(struct ovpn_peer *peer) +{ + percpu_ref_kill(&peer->refcount); } struct ovpn_peer *ovpn_peer_new(struct ovpn_priv *ovpn, u32 id); diff --git a/drivers/net/ovpn/tcp.c b/drivers/net/ovpn/tcp.c index 8fe8a8e750a4..29981c0279e2 100644 --- a/drivers/net/ovpn/tcp.c +++ b/drivers/net/ovpn/tcp.c @@ -140,21 +140,18 @@ static void ovpn_tcp_rcv(struct strparser *strp, struct sk_buff *skb) /* hold reference to peer as required by ovpn_recv(). * * NOTE: in this context we should already be holding a reference to - * this peer, therefore ovpn_peer_hold() is not expected to fail + * this peer */ - if (WARN_ON(!ovpn_peer_hold(peer))) - goto err_nopeer; + ovpn_peer_hold(peer); ovpn_recv(peer, skb); return; err: - /* take reference for deferred peer deletion. should never fail */ - if (WARN_ON(!ovpn_peer_hold(peer))) - goto err_nopeer; + /* take reference for deferred peer deletion */ + ovpn_peer_hold(peer); if (!queue_work(ovpn_wq, &peer->tcp.defer_del_work)) ovpn_peer_put(peer); ovpn_dev_dstats_rx_dropped(peer->ovpn->dev); -err_nopeer: kfree_skb(skb); } @@ -168,7 +165,8 @@ static int ovpn_tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, rcu_read_lock(); sock = rcu_dereference_sk_user_data(sk); - if (unlikely(!sock || !sock->peer || !ovpn_peer_hold(sock->peer))) { + if (unlikely(!sock || !sock->peer || + !ovpn_peer_hold_rcu(sock->peer))) { rcu_read_unlock(); return -EBADF; } @@ -385,7 +383,7 @@ static void ovpn_tcp_release(struct sock *sk) /* during initialization this function is called before * assigning sock->peer */ - if (unlikely(!peer || !ovpn_peer_hold(peer))) { + if (unlikely(!peer || !ovpn_peer_hold_rcu(peer))) { rcu_read_unlock(); return; } @@ -411,7 +409,8 @@ static int ovpn_tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) lock_sock(sk); rcu_read_lock(); sock = rcu_dereference_sk_user_data(sk); - if (unlikely(!sock || !sock->peer || !ovpn_peer_hold(sock->peer))) { + if (unlikely(!sock || !sock->peer || + !ovpn_peer_hold_rcu(sock->peer))) { rcu_read_unlock(); release_sock(sk); return -EIO; @@ -588,7 +587,7 @@ static void ovpn_tcp_close(struct sock *sk, long timeout) } peer = sock->peer; - if (!peer || !ovpn_peer_hold(peer)) { + if (!peer || !ovpn_peer_hold_rcu(peer)) { rcu_read_unlock(); return; } @@ -619,7 +618,7 @@ static __poll_t ovpn_tcp_poll(struct file *file, struct socket *sock, return 0; } - if (ovpn_peer_hold(ovpn_sock->peer)) { + if (ovpn_peer_hold_rcu(ovpn_sock->peer)) { peer = ovpn_sock->peer; queue = &peer->tcp.user_queue; } From patchwork Fri Sep 18 16:03:36 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ralf Lici X-Patchwork-Id: 5387 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7000:6446:b0:8a0:ea1f:253a with SMTP id n6csp8968841mag; Fri, 18 Sep 2026 09:04:28 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvBz69nTFl8R68zHQEazQIF9dBqAckPdGWa2QZmJ7/UhQ/UeWinpqataN8mkn5+i09/5+QkR6/b8EQn4=@openvpn.net X-Received: by 2002:a05:6870:7d16:b0:485:d31b:775d with SMTP id 586e51a60fabf-486e74133b2mr2912132fac.27.1789747467954; Fri, 18 Sep 2026 09:04:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1789747467; cv=none; d=google.com; s=arc-20260327; b=cIayppy9tmP6DIBTzssEznKP4p9gESQ6gMA6ylFso8l0fRYqXSGWwenDRpejHszs09 Emgst9bECFDTE7Pu9J53NBPhUwLSW63FlocCnaLdgiYH6MlbOgK9KraYdCRt/Bx48G78 giV22d3HDMEMdK8DwZ/OoX9Te36Ik1F7pz5MZu8SOnjCqGS7Fd1OIsVQfb1Ivw+kIjIY PQ9oZ5Ej0whGNZLj73sCN8+sX3inFSOcL07qopC+HVmRh7xDRwt0mv5nMQpYrfLJCLCB DiVV9Na+FEhV0kTKsX7b7lKmxVaZlh/fbuQ6I4sXb3jh1qOLK+pBRgdZNlduuU1hXarN NOlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=NuepX+4H+xSDVsB5hRRP8xRJYW9ELdN81oe93SUnOvc=; fh=4NbAC/LsuMLI0S0hprUlLSLCiHwg6SCAifhH718Jh0Q=; b=m/f8esbtYAeJyp04rhmypTE4a5U0vm0U+F8TPo2o2Qsrt7vyG61wCeh5WRSCVe+cLZ CROSO9GjAAA+66fZjzDXAWwNOq23I9Br6TzFM7oLsFS3Tt5HcvEFV6ki8lUwPwkt76Cc EhHD2tmldHy1jOt673WnhmKqez7vv7tWp5TOhwQFdsIzoVH1alh8t8kzz4MJeJ1dCnW0 //uoi/945JRCNlzelj2Ue9HL3bavFp3RVylHV/vtPsWsMXxe8GApq5BnLyHRZ+MIvhWe EyBlTrxcnmsG/LfL5Kn9paMOt/neD/StqI+AdvqARBU6WoMtGhA2M4hCCpGsfl5RRRIN kmBQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=eIESosN5; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=HISOtwEY; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b="OL9K/vtm"; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=CLiFlLJX; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id 586e51a60fabf-487397560c6si2456969fac.91.2026.09.18.09.04.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Sep 2026 09:04:27 -0700 (PDT) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=pass header.i=@lists.sourceforge.net header.s=beta header.b=eIESosN5; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=HISOtwEY; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b="OL9K/vtm"; dkim=neutral (body hash did not verify) header.i=@mandelbit.com header.s=MBO0001 header.b=CLiFlLJX; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Content-Transfer-Encoding:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Subject:MIME-Version:References:In-Reply-To:Message-ID:Date:To:From:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=NuepX+4H+xSDVsB5hRRP8xRJYW9ELdN81oe93SUnOvc=; b=eIESosN5scip90TwyhYaLo3y48 WnMuIMxQttNa/ZEadoLwDSO45dNMTGz7T0YpS4797o18/ZaNmgM7IIjQBwInJWc7Ci/sgDVjboZSQ rMst9/o88Z08jHRog64RcebPwz/Uh73MGKkP0+Bl/ITEJHXQoDkkfFhni/aEDGIPIctU=; Received: from [127.0.0.1] (helo=sfs-ml-2.v29.lw.sourceforge.com) by sfs-ml-2.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1x7b4Z-0000TG-B2; Fri, 18 Sep 2026 16:04:24 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-2.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1x7b4Y-0000T5-7g for openvpn-devel@lists.sourceforge.net; Fri, 18 Sep 2026 16:04:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:To:From:Sender:Reply-To:Cc:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=Fh7eou2oqxg+8c/XKPIVykOTEtVLOayPCyw61ORg9DM=; b=HISOtwEYuW6Gr4EC6zGiG1ouhZ WyNGyVFUkV9w+/eRceYa79nEEjWc/yX2PGSoClLwlHw36/6TH4Kkasj3Aif/SlOrl9fTVvbjWMolC 1+6guHr7dktQ2yMnElbeMoVbljDOTkuACbWEGB/K/IbgIol/DoD5BFVHFn5LBC/TZTLs=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID: Date:Subject:To:From:Sender:Reply-To:Cc:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Fh7eou2oqxg+8c/XKPIVykOTEtVLOayPCyw61ORg9DM=; b=OL9K/vtm1ZlZFaylF6llG+bTiS T9Gy87IV6udFxrx04Np5yR13v9SiGm7prd+CoXg4WpRevu+ckwP253K7OEBYxRa9vdOrtDRJDighW cBtN4xw9SmN7lS/D6zwFOKVLVbR4T2kKTdI0Ae/6Gbrp2Lwi9eRJueYwV9dwDLVnEXaA=; Received: from mout-b-203.mailbox.org ([195.10.208.52]) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1x7b4X-00067o-QM for openvpn-devel@lists.sourceforge.net; Fri, 18 Sep 2026 16:04:22 +0000 Received: from smtp1.mailbox.org (smtp1.mailbox.org [10.196.197.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-b-203.mailbox.org (Postfix) with ESMTPS id 4hmcrF2BKRzLmFJ for ; Fri, 18 Sep 2026 18:04:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandelbit.com; s=MBO0001; t=1789747453; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fh7eou2oqxg+8c/XKPIVykOTEtVLOayPCyw61ORg9DM=; b=CLiFlLJXu7UIMMrrKdLynrDrK7x4ExQusb81k3zbIDTBfYQBwYWOxu/2GlzGOO79Yo+xKz B7a+o/SXKXDdZD3l5mreNsegzjrVaJ+WP9gO9F73W5OiScKVKZfG0wEpTenrtUOsBXrFZw nb9knAgdPQDiBn4MbQRU4Gd5QfOHopkvx8lQTk/MDxZbR20Zuy4l8KMo3eIUBhb9l9dgNf Ycz8GBqIKc1DYsXFFA3eQq0ToZvbLKKzMQLZu+ppbxTPbCQeH7D6f1p0Bb/FgI3ZUieQa9 F+vW9yn4P5mjeB/q8MB6m98EIZgssuxLnUPtZQn+gLI8YSVQn2lNUb60E41o2A== From: Ralf Lici To: openvpn-devel@lists.sourceforge.net Date: Fri, 18 Sep 2026 18:03:36 +0200 Message-ID: <6743db29e84f7e7018e7d2ce71cbdf379d228c1b.1789746543.git.ralf@mandelbit.com> In-Reply-To: References: MIME-Version: 1.0 X-Spam-Score: -0.2 (/) X-Spam-Report: Spam detection software, running on the system "sfi-spamd-1.hosts.colo.sdot.me", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Packet processing holds a key slot until asynchronous crypto completion. Parallel traffic using one key consequently updates the same kref cache line for every packet, even though key slots are normal [...] Content analysis details: (-0.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [195.10.208.52 listed in wl.mailspike.net] -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain X-Headers-End: 1x7b4X-00067o-QM Subject: [Openvpn-devel] [PATCH ovpn net-next v2 2/2] ovpn: use percpu references for key slots X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: 1876686241235512578 X-GMAIL-MSGID: 1876686241235512578 Packet processing holds a key slot until asynchronous crypto completion. Parallel traffic using one key consequently updates the same kref cache line for every packet, even though key slots are normally long-lived. Replace the key-slot kref with percpu_ref. Kill the initial reference after atomically unpublishing a slot, while ordinary packet completions only put their live references. Key destruction needs process context to release the crypto transforms and an RCU grace period for lockless configuration readers which do not hold a key reference. With percpu_ref, the release callback may run from the internal RCU callback which switches the reference to atomic mode. Queueing rcu_work from there would create a nested callback which the module cleanup rcu_barrier is not guaranteed to wait for. Record the RCU state after unpublishing the slot and queue ordinary work from the release callback instead. The worker conditionally waits for the recorded grace period before freeing the key. This provides explicit RCU synchronization without depending on percpu_ref implementation details or unconditionally starting another grace period. Peer teardown kills its keys before dropping the final netdevice reference, so rtnl_link_unregister cannot return before the key percpu-ref callbacks have been registered. The existing rcu_barrier drains those callbacks and destroy_workqueue subsequently drains the ordinary key cleanup work. Initialize the reference only after the rest of the key slot is ready, allowing the existing destruction path to handle allocation failure. The conversion adds an 8-byte counter per possible CPU and a 56-byte control allocation for each installed key slot; with at most two long-lived slots per peer, this is a bounded memory tradeoff for removing the shared reference cacheline from the packet path. After the peer conversion, perf c2c still identified the key-slot kref cacheline, with 15 sampled HITM loads attributed to its locked reference update. This change removed that line from the shared-cacheline profile as well. In a set of interleaved 32-flow iperf3 runs, the combined peer and key conversion moved throughput to 8.812 Gbit/s, against 7.734 Gbit/s for the baseline, a +13.95% improvement. Single-stream control found no reproducible regression. Signed-off-by: Ralf Lici --- Changes since v1 https://lore.kernel.org/openvpn-devel/9551c9c842e4ac926089badc81f445becca513e5.1789658051.git.ralf@mandelbit.com/ - Replace the nested rcu_work teardown with ordinary work so module cleanup cannot destroy the workqueue before the work is queued. (Sashiko) - Record the RCU state after unpublishing a key and conditionally wait for RCU-only readers in the final worker. (Sashiko) drivers/net/ovpn/crypto.c | 16 ++++++++-------- drivers/net/ovpn/crypto.h | 27 ++++++++++++++++++++++----- drivers/net/ovpn/crypto_aead.c | 17 +++++++++++++---- 3 files changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/net/ovpn/crypto.c b/drivers/net/ovpn/crypto.c index 7e545428900a..25c3a2b6da48 100644 --- a/drivers/net/ovpn/crypto.c +++ b/drivers/net/ovpn/crypto.c @@ -18,12 +18,12 @@ #include "crypto_aead.h" #include "crypto.h" -void ovpn_crypto_key_slot_release(struct kref *kref) +void ovpn_crypto_key_slot_release(struct percpu_ref *ref) { struct ovpn_crypto_key_slot *ks; - ks = container_of(kref, struct ovpn_crypto_key_slot, refcount); - queue_rcu_work(ovpn_wq, &ks->free_work); + ks = container_of(ref, struct ovpn_crypto_key_slot, refcount); + queue_work(ovpn_wq, &ks->free_work); } /* can only be invoked when all peer references have been dropped (i.e. RCU @@ -36,13 +36,13 @@ void ovpn_crypto_state_release(struct ovpn_crypto_state *cs) ks = rcu_access_pointer(cs->slots[0]); if (ks) { RCU_INIT_POINTER(cs->slots[0], NULL); - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); } ks = rcu_access_pointer(cs->slots[1]); if (ks) { RCU_INIT_POINTER(cs->slots[1], NULL); - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); } } @@ -66,7 +66,7 @@ bool ovpn_crypto_kill_key(struct ovpn_crypto_state *cs, u8 key_id) spin_unlock_bh(&cs->lock); if (ks) - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); /* let the caller know if a key was actually killed */ return ks; @@ -104,7 +104,7 @@ int ovpn_crypto_state_reset(struct ovpn_crypto_state *cs, spin_unlock_bh(&cs->lock); if (old) - ovpn_crypto_key_slot_put(old); + ovpn_crypto_key_slot_kill(old); return 0; } @@ -141,7 +141,7 @@ void ovpn_crypto_key_slot_delete(struct ovpn_crypto_state *cs, } pr_debug("deleting key slot %u, key_id=%u\n", slot, ks->key_id); - ovpn_crypto_key_slot_put(ks); + ovpn_crypto_key_slot_kill(ks); } void ovpn_crypto_key_slots_swap(struct ovpn_crypto_state *cs) diff --git a/drivers/net/ovpn/crypto.h b/drivers/net/ovpn/crypto.h index e3feb16d5498..a0209b1280de 100644 --- a/drivers/net/ovpn/crypto.h +++ b/drivers/net/ovpn/crypto.h @@ -10,6 +10,8 @@ #ifndef _NET_OVPN_OVPNCRYPTO_H_ #define _NET_OVPN_OVPNCRYPTO_H_ +#include +#include #include #include "pktid.h" @@ -47,8 +49,9 @@ struct ovpn_crypto_key_slot { struct ovpn_pktid_recv pid_recv ____cacheline_aligned_in_smp; struct ovpn_pktid_xmit pid_xmit ____cacheline_aligned_in_smp; - struct rcu_work free_work; - struct kref refcount; + struct work_struct free_work; + unsigned long rcu_state; + struct percpu_ref refcount; }; struct ovpn_crypto_state { @@ -61,7 +64,7 @@ struct ovpn_crypto_state { static inline bool ovpn_crypto_key_slot_hold(struct ovpn_crypto_key_slot *ks) { - return kref_get_unless_zero(&ks->refcount); + return percpu_ref_tryget_live_rcu(&ks->refcount); } static inline void ovpn_crypto_state_init(struct ovpn_crypto_state *cs) @@ -121,11 +124,25 @@ ovpn_crypto_key_slot_primary(const struct ovpn_crypto_state *cs) return ks; } -void ovpn_crypto_key_slot_release(struct kref *kref); +void ovpn_crypto_key_slot_release(struct percpu_ref *ref); static inline void ovpn_crypto_key_slot_put(struct ovpn_crypto_key_slot *ks) { - kref_put(&ks->refcount, ovpn_crypto_key_slot_release); + percpu_ref_put(&ks->refcount); +} + +/** + * ovpn_crypto_key_slot_kill - stop new users and drop the initial reference + * @ks: key slot which has already been unpublished + * + * percpu_ref does not guarantee an RCU grace period before release. Record + * the RCU state after unpublishing the key so the final worker can synchronize + * with lockless readers without unconditionally starting another grace period. + */ +static inline void ovpn_crypto_key_slot_kill(struct ovpn_crypto_key_slot *ks) +{ + ks->rcu_state = get_state_synchronize_rcu(); + percpu_ref_kill(&ks->refcount); } int ovpn_crypto_state_reset(struct ovpn_crypto_state *cs, diff --git a/drivers/net/ovpn/crypto_aead.c b/drivers/net/ovpn/crypto_aead.c index 74eaf6fac2f5..77ae958a971a 100644 --- a/drivers/net/ovpn/crypto_aead.c +++ b/drivers/net/ovpn/crypto_aead.c @@ -391,8 +391,13 @@ static void ovpn_aead_crypto_key_slot_free_work(struct work_struct *work) { struct ovpn_crypto_key_slot *ks; - ks = container_of(to_rcu_work(work), struct ovpn_crypto_key_slot, - free_work); + ks = container_of(work, struct ovpn_crypto_key_slot, free_work); + /* Reaching this worker means every reference held by packet processing + * and asynchronous crypto has been returned. Separately wait for any + * RCU-only reader which observed the slot before it was unpublished. + */ + cond_synchronize_rcu(ks->rcu_state); + percpu_ref_exit(&ks->refcount); ovpn_aead_crypto_key_slot_free(ks); kfree(ks); } @@ -427,8 +432,7 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc) ks->encrypt = NULL; ks->decrypt = NULL; - INIT_RCU_WORK(&ks->free_work, ovpn_aead_crypto_key_slot_free_work); - kref_init(&ks->refcount); + INIT_WORK(&ks->free_work, ovpn_aead_crypto_key_slot_free_work); ks->key_id = kc->key_id; ks->encrypt = ovpn_aead_init("encrypt", alg_name, @@ -458,6 +462,11 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc) ovpn_pktid_xmit_init(&ks->pid_xmit); ovpn_pktid_recv_init(&ks->pid_recv); + ret = percpu_ref_init(&ks->refcount, ovpn_crypto_key_slot_release, 0, + GFP_KERNEL); + if (ret < 0) + goto destroy_ks; + return ks; destroy_ks: