From patchwork Tue Dec 27 20:26:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gert Doering X-Patchwork-Id: 2954 Return-Path: Delivered-To: patchwork@openvpn.net Received: by 2002:a05:7300:c95:b0:82:e4b3:40a0 with SMTP id p21csp2885516dyk; Tue, 27 Dec 2022 12:27:24 -0800 (PST) X-Google-Smtp-Source: AMrXdXvOVNdDz1zq3Aw/BNgbgUWSdGLNF1GZS1vdYVDvNZzZT/IsT32ITIHeWHrex27w3mrfOIjS X-Received: by 2002:a05:6a20:3d1d:b0:a3:9d91:b6a6 with SMTP id y29-20020a056a203d1d00b000a39d91b6a6mr35428361pzi.35.1672172844366; Tue, 27 Dec 2022 12:27:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672172844; cv=none; d=google.com; s=arc-20160816; b=Ijjsoc+/d7zsAzAUf+k6hNpA5MyFlE12CMYlrLtZC2DBXWS15UOLoq8duBnH8ixeqC 4HDR064Dc6xX27TdpJ0j4BnmvTxgMMlvXfceA7vDgB8Uiw2PydzeiZolc2mWyME8xB0X 0YLVgQsO1LQmMRCOosoIdeqYieMZb/QoXmkJFVkNalCVcaRuH/oTswCcjkuUa8jby70R RWKmZV0LDKYKWkOEnoFW1mNx3R5iF13cT1vGxbTIYiGXQfKvaxwmopicMAh5MwvHmDbD vA7PInVuG77eJEyj62cmcwF2nimTPbr1lotYo68yElfu4Tld459m2WxGTVrU5SCiB1WO /ipA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:dkim-signature; bh=c5gas4v5FGF0tTSNOesFOJz2mpPdokYY08QchkCzau8=; b=ZJCoCDFYisgYKjbSQptbDeeZdNqSzpTWHavN6Wj+pR6+3WFhY2DhAzxSxCXbdER1Wh mqJ+T1Jbrb9XM/xIYixTV7McoMoaG+/nYbVbiGzauuN/99DzEiGkqZIjecOIZPVl++XI iWVu6jogyiuiAXmXUqIx/3JFiXHbRGK9skjt1JClU7qcyqWgAwIy0ZsJdEsyjjli4ngh Xpz9+mu1zshCT8IbsqEYU6SEML7C7PhHRf9BzfYUEz6X8J7tQrezBZFQ+CzON06MP7NU ubaI20Roz9dD5aBriZ/v2NOi7vj9yErOekFL7h8yYOgp5cdy2iLLR0InLjhyy/wwbMWC a1MA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=cr+0NYhl; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=lQaIUzTK; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from lists.sourceforge.net (lists.sourceforge.net. [216.105.38.7]) by mx.google.com with ESMTPS id f12-20020a63554c000000b0046fc9e82c99si15532452pgm.206.2022.12.27.12.27.23 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Dec 2022 12:27:24 -0800 (PST) Received-SPF: pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) client-ip=216.105.38.7; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@sourceforge.net header.s=x header.b=cr+0NYhl; dkim=neutral (body hash did not verify) header.i=@sf.net header.s=x header.b=lQaIUzTK; spf=pass (google.com: domain of openvpn-devel-bounces@lists.sourceforge.net designates 216.105.38.7 as permitted sender) smtp.mailfrom=openvpn-devel-bounces@lists.sourceforge.net Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1pAGWp-0006sX-FF; Tue, 27 Dec 2022 20:26:27 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pAGWn-0006sQ-Sj for openvpn-devel@lists.sourceforge.net; Tue, 27 Dec 2022 20:26:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:To:From:Sender:Reply-To:Cc:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=Eba6dFbzjoRINfZb7AZadEtl2kGsDN5i6+bwDjxLtRs=; b=cr+0NYhlYXrHiMWMmd0vK/otC8 LTS9hk8TbjloEcBYcSyd9t7maUA+5DXl9TOwYmZhGE6YQ6NcOcuH3Oyi2+ku8/us9KkMR3AV509Dw Dt6ObaH5+NsxmTO9/zMWQWzjuyqyHLwHjtaGEEPy+s8I1OQ1/WepjdW3sp3nZGKsms9A=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-Id: Date:Subject:To:From:Sender:Reply-To:Cc:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Eba6dFbzjoRINfZb7AZadEtl2kGsDN5i6+bwDjxLtRs=; b=lQaIUzTK8kIzDPKNR7ur/3pij0 aYKryG5BY/RP7u7zymZplATCMpY5pq0gST+yNk7TnPauuDTZ+2eeml7TOX+gYFmWrq5Etr0fG6NXW bYE2g2TREz9sUkLs30itKXKI10gCFu6W4oNcn8F3QrJKjIHre/DYiR+bVnc6nHKrcVcw=; Received: from vmail1.greenie.net ([195.30.8.66]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1pAGWl-00EK41-I5 for openvpn-devel@lists.sourceforge.net; Tue, 27 Dec 2022 20:26:25 +0000 Received: from ubuntu2004.ov.greenie.net (ubuntu2004.ov.greenie.net [IPv6:2001:608:1:995a:250:56ff:febb:2084]) by vmail1.greenie.net (8.17.1/8.16.1) with SMTP id 2BRKQEQg050470 for ; Tue, 27 Dec 2022 21:26:14 +0100 (CET) Received: (nullmailer pid 2115019 invoked by uid 1000); Tue, 27 Dec 2022 20:26:14 -0000 From: Gert Doering To: openvpn-devel@lists.sourceforge.net Date: Tue, 27 Dec 2022 21:26:14 +0100 Message-Id: <20221227202614.2114971-1-gert@greenie.muc.de> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221224230807.1827964-1-gert@greenie.muc.de#> References: <20221224230807.1827964-1-gert@greenie.muc.de#> MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (vmail1.greenie.net [IPv6:2001:608:1:995a:20c:29ff:feb8:10eb]); Tue, 27 Dec 2022 21:26:14 +0100 (CET) X-Spam-Score: -2.0 (--) X-Spam-Report: Spam detection software, running on the system "util-spamd-1.v13.lw.sourceforge.com", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: TCP multipoint servers with Linux-DCO can crash under yet-unknown circumstances where a TCP socket gets handed to the kernel (= userland shall not acceess it again) but the socket still lands in the e [...] Content analysis details: (-2.0 points, 6.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [195.30.8.66 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail domains are different 0.0 SPF_NONE SPF: sender does not publish an SPF Record X-Headers-End: 1pAGWl-00EK41-I5 Subject: [Openvpn-devel] [PATCH v3] bandaid fix for TCP multipoint server crash with Linux-DCO X-BeenThere: openvpn-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: openvpn-devel-bounces@lists.sourceforge.net X-getmail-retrieved-from-mailbox: Inbox X-GMAIL-THRID: =?utf-8?q?1753400312333649931?= X-GMAIL-MSGID: =?utf-8?q?1753400312333649931?= TCP multipoint servers with Linux-DCO can crash under yet-unknown circumstances where a TCP socket gets handed to the kernel (= userland shall not acceess it again) but the socket still lands in the event polling mechanism, and is passed to link_socket_read() with sock->fd being "-1" (SOCKET_UNDEFINED). This is a bug, but it happens very unfrequently so not fixed yet. When this happens, the server gets stuck in an endless loop of "trying recvfrom(-1, ..), getting an error, looging that error, continue" until the server's disk is full. The situation is being made a bit more complex by the dco-win approach of treating "all kernel sockets as UDP", so the Linux implementation tries to access the -1 socket as UDP, confusing the picture more. As a bandaid to avoid the crash, this patch changes - socket.h: only do the "if dco_installed, treat as UDP" for WIN32 (link_socket_read()) - socket.c: add ASSERT(sock->fd >= 0); checks to all UDP socket paths (we should never even hit those as this is a TCP specific problem, but in the "sock->fd = -1" case, doing a clean server abort is preferred to "the disk is full with non-helpful logfiles, and then the server crashes anyway") - socket.c: in the TCP read function, link_socket_read_tcp(), check for sock->fd < 0 and trigger "sock->stream_reset = true" (+ write to the log what happened). This change will kill this particular TCP client instance (SIGTERM), but leave the rest of the server running fine - and given that in our tests this issue seems to be triggered by inbound TCP RST in just the wrong moment, it seems to be "a properly-sized bandaid". v2: rebase on top of "move dco_installed back to link_socket" v3: move sock->fd check inside !residual_fully_formed clause (so we can still handle already-read packets) Github: OpenVPN/openvpn#190 Signed-off-by: Gert Doering Acked-By: Arne Schwabe --- src/openvpn/socket.c | 17 +++++++++++++++++ src/openvpn/socket.h | 4 ++++ 2 files changed, 21 insertions(+) diff --git a/src/openvpn/socket.c b/src/openvpn/socket.c index c7ec0e06..2230052f 100644 --- a/src/openvpn/socket.c +++ b/src/openvpn/socket.c @@ -3228,6 +3228,18 @@ link_socket_read_tcp(struct link_socket *sock, if (!sock->stream_buf.residual_fully_formed) { + /* with Linux-DCO, we sometimes try to access a socket that is + * already installed in the kernel and has no valid file descriptor + * anymore. This is a bug. + * Handle by resetting client instance instead of crashing. + */ + if (sock->sd == SOCKET_UNDEFINED) + { + msg(M_INFO, "BUG: link_socket_read_tcp(): sock->sd==-1, reset client instance" ); + sock->stream_reset = true; /* reset client instance */ + return buf->len = 0; /* nothing to read */ + } + #ifdef _WIN32 sockethandle_t sh = { .s = sock->sd }; len = sockethandle_finalize(sh, &sock->reads, buf, NULL); @@ -3285,6 +3297,8 @@ link_socket_read_udp_posix_recvmsg(struct link_socket *sock, struct msghdr mesg; socklen_t fromlen = sizeof(from->dest.addr); + ASSERT(sock->sd >= 0); /* can't happen */ + iov.iov_base = BPTR(buf); iov.iov_len = buf_forward_capacity_total(buf); mesg.msg_iov = &iov; @@ -3351,6 +3365,9 @@ link_socket_read_udp_posix(struct link_socket *sock, socklen_t fromlen = sizeof(from->dest.addr); socklen_t expectedlen = af_addr_size(sock->info.af); addr_zero_host(&from->dest); + + ASSERT(sock->sd >= 0); /* can't happen */ + #if ENABLE_IP_PKTINFO /* Both PROTO_UDPv4 and PROTO_UDPv6 */ if (sock->info.proto == PROTO_UDP && sock->sockflags & SF_USE_IP_PKTINFO) diff --git a/src/openvpn/socket.h b/src/openvpn/socket.h index 05c31b10..3b9c1ba3 100644 --- a/src/openvpn/socket.h +++ b/src/openvpn/socket.h @@ -1058,7 +1058,11 @@ link_socket_read(struct link_socket *sock, struct buffer *buf, struct link_socket_actual *from) { +#ifdef _WIN32 if (proto_is_udp(sock->info.proto) || sock->dco_installed) +#else + if (proto_is_udp(sock->info.proto)) +#endif /* unified UDPv4 and UDPv6, for DCO the kernel * will strip the length header */ {