Message ID | CAOh3LPMK7T2gduJXrKaQRDvPZdjCiBJtCqXWiFT0xj4TAaj2LA@mail.gmail.com |
---|---|
State | New |
Headers | show |
Series | [Openvpn-devel] Bulk mode - Feature request - Patch diff [updated] | expand |
Am 07.08.25 um 20:29 schrieb Jon Chiappetta via Openvpn-devel: > Thanks to Gert's help on this, I was able to finally configure and > compile and run and test the bulk mode changes against the latest git > source code to ensure everything still works correctly. > > I also fixed up some other issues like properly freeing the extra buffer > allocations and removing the unneeded batched data prefixes and > converting a remaining while loop to a max limited for loop and properly > resetting the outgoing tun buffer pointer at the end of the write method > when finished. It would still good to explain what you are trying to achieve here and what the idea behind the patch is to be able to review and understand your patch. The patch itself basically has no comments at all, so it is very hard to decipher for me from the patch what it is trying to to do. Eg there is a variable flag_ciph that fiddles with encryption of packets. You are talking and describing this bulk mode as if it was obvious but it is not. The description on your blog says: > [...] read 8192 bytes off of the client’s TCP sockets directly and > proxy them in one write call over TCP directly to the VPN server > without needing a tunnel interface with a small sized MTU which > bottlenecks reads+writes to <1500 bytes per function call. It also not helping as you talking about TCP write/reads, where I can see some improvement by cutting down the number of reads/writes. But the second part then talks about not using a tunnel with a small sized MTU. But if you use a larger sized TUN interface with a larger MTU, then you already have larger reads/writes to the TCP socket. Also your speedtest showing 562 is meaningless without having any comparison without your patch. Arne
Hi Arne, You are correct, I didn't do a very good job of explaining the code in my blog post, I usually keep those short with more screen captures because I figure that not many people would actually take the time to read through it there. Also, I didn't really add many comments either but I did try to copy the present style of the code to try and make it match and be more consistent throughout, even if not perfect yet. I'm still testing out the change myself in my own home setup here to see if I run into any bad edge cases along the way. I can always try to explain the different code parts as I am indeed modifying the core parts of the read and write operations for tun and tcp so it's a big change to make to the code base. Basically this change is important to me in particular because of my setup and requirements in specific. I have WiFi LAN clients which all assume a 1500 byte MTU on their side and I have a router WAN client which enforces a 1500 byte MTU on the internet's side. In the middle of my core network is a VPN box and almost every VPN software will operate in UDP mode with a sub-1500 MTU in the middle of this network pipeline. This is not a good design to have in general as I don't want to waste cycles fragmenting and/or compressing the data into smaller sized UDP packets. With the code change I am presenting, I am able to specify a true 1500 byte VPN MTU interface with the exact matching 1500 byte read calls to the TUN interface itself (the code base had to be modified to allow for this because it was adding onto the payload_size used in the read call which I didn't want as I am operating on exact multiples of 1500 bytes in specific). With this change, my network pipeline is a true 1500 byte MTU which matches all the way from the client side to the vpn link to the internet side and to the server side (end to end to end to end). In addition, I also added the ability to batch together multiple 1500 byte read calls (specifically 6 x 1500 bytes into 9000 bytes) into one single encrypt sign call and one single TCP write call. This allows the encryption method to operate only once on a much larger payload size as well as allow the linux kernel to efficiently transfer the data with order and delivery guaranteed as fast as possible. The code base had to be modified to allow for all of this as well as it was preventing me from performing this much larger sized ssl sign and encrypt + tcp read and write (the code base assumes you are operating on only 1 tun read call worth of data at a time everywhere). This is exactly why I prefer using TCP to tunnel encrypted network data as my solution provided can properly set a full sized 1500 byte MTU as well as perform an exact matching read call of 1500 bytes to get the full amount of data from the interface and then bulk it together to efficiently encrypt it and then use the magic of TCP to transfer that data all at once as quickly as possible without any need for fragmentation or compression. I don't think any other VPN product on the market offers that kind of functionality as far as I am aware as most other VPN products use a smaller sized MTU as well as the packet size limitations of UDP. I believe that this could be a distinguishing feature for OpenVPN as well as automatically solve some of the issues that folks run into when inserting a VPN appliance into the middle of their network setups. I've been running this change on my own setup to at least make sure it works and it seems to be running pretty nicely so far. I haven't experienced any fragmentation or performance issues as any sized data that comes off the clients LAN side is fully taken care of now through the VPN side and onto the WAN and server side. If this is something you are not interested in I can understand that, I can stop posting here and the most I can do is at least submit a pull request in case anyone in the future is indeed interested in such work. It'd be nice to contribute to a good quality open source project that I have used for many many years and something which may help solve other community member's issues with regards to the small sized MTU + UDP problem which does exist in practice and really hampers connections along the way in a network design. I also don't mind explaining my code parts if you actually want, I just need to take time to write them out and describe what they are doing and why. As you can see, I am trying to achieve a very specific and exact design goal that the code base wasn't originally allowing for, so I had to make some modifications to be able to accomplish it. Thanks, Jon C On Fri, Aug 8, 2025 at 5:53 AM Arne Schwabe <arne@rfc2549.org> wrote: > Am 07.08.25 um 20:29 schrieb Jon Chiappetta via Openvpn-devel: > > Thanks to Gert's help on this, I was able to finally configure and > > compile and run and test the bulk mode changes against the latest git > > source code to ensure everything still works correctly. > > > > I also fixed up some other issues like properly freeing the extra buffer > > allocations and removing the unneeded batched data prefixes and > > converting a remaining while loop to a max limited for loop and properly > > resetting the outgoing tun buffer pointer at the end of the write method > > when finished. > > > It would still good to explain what you are trying to achieve here and > what the idea behind the patch is to be able to review and understand > your patch. > > The patch itself basically has no comments at all, so it is very hard to > decipher for me from the patch what it is trying to to do. Eg there is a > variable flag_ciph that fiddles with encryption of packets. > > You are talking and describing this bulk mode as if it was obvious but > it is not. The description on your blog says: > > > [...] read 8192 bytes off of the client’s TCP sockets directly and > > proxy them in one write call over TCP directly to the VPN server > > without needing a tunnel interface with a small sized MTU which > > bottlenecks reads+writes to <1500 bytes per function call. > > It also not helping as you talking about TCP write/reads, where I can > see some improvement by cutting down the number of reads/writes. But the > second part then talks about not using a tunnel with a small sized MTU. > But if you use a larger sized TUN interface with a larger MTU, then you > already have larger reads/writes to the TCP socket. > > Also your speedtest showing 562 is meaningless without having any > comparison without your patch. > > Arne >
[replying to my own update thread] I pushed an update to the PR with the following small changes: - rebased to the latest master commit - increased the data buffer size to be slightly bigger than the read buffer size - copied two more buf resets so that the additional bulk forward functions match the present logic checks Example updated pull request: https://github.com/OpenVPN/openvpn/pull/814/files $ cat 0001-bulk-mode.patch From 2ce0b023d105e7ecc289a414cd26f7ebc8bcbcaf Mon Sep 17 00:00:00 2001 From: Jon Chiappetta <root@fossjon.com> Date: Wed, 6 Aug 2025 16:33:18 -0400 Subject: [PATCH] bulk mode --- src/openvpn/forward.c | 228 ++++++++++++++++++++++++++++++++++++++++-- src/openvpn/forward.h | 4 + src/openvpn/init.c | 57 +++++++++++ src/openvpn/mtu.c | 10 +- src/openvpn/mtu.h | 13 +++ src/openvpn/multi.c | 7 +- src/openvpn/openvpn.h | 10 ++ src/openvpn/options.c | 8 ++ src/openvpn/options.h | 3 + 9 files changed, 328 insertions(+), 12 deletions(-) diff --git a/src/openvpn/forward.c b/src/openvpn/forward.c index 75ca9d5c..0af983e9 100644 --- a/src/openvpn/forward.c +++ b/src/openvpn/forward.c @@ -46,6 +46,9 @@ #include "mstats.h" +#include <sys/select.h> +#include <sys/time.h> + counter_type link_read_bytes_global; /* GLOBAL */ counter_type link_write_bytes_global; /* GLOBAL */ @@ -78,6 +81,32 @@ show_wait_status(struct context *c) #endif /* ifdef ENABLE_DEBUG */ +bool check_bulk_mode(struct context *c) +{ + if ((c->c2.frame.bulk_size > 0) && (c->c1.tuntap != NULL) && (c->c2.buffers != NULL)) + { + return true; + } + return false; +} + +void xfer_io(struct context *c, struct context *b) +{ + int plen = 0; + if (check_bulk_mode(b)) + { + int leng = (b->c2.buffers->bufs_indx + 1); + for (int x = 0; x < leng; ++x) + { + plen = BLEN(&b->c2.bufs[x]); + if (plen < 1) { c->c2.bufs[x].len = 0; } + else { c->c2.bufs[x] = b->c2.bufs[x]; } + } + c->c2.buffers->bufs_indx = b->c2.buffers->bufs_indx; + b->c2.buffers->bufs_indx = -1; + } +} + static void check_tls_errors_co(struct context *c) { @@ -605,6 +634,21 @@ buffer_turnover(const uint8_t *orig_buf, struct buffer *dest_stub, struct buffer } } +uint8_t *buff_prepsize(uint8_t *buff, int *size) +{ + buff[0] = ((*size >> 8) & 0xff); + buff[1] = ((*size >> 0) & 0xff); + buff += 2; + return buff; +} + +uint8_t *buff_postsize(uint8_t *buff, int *size) +{ + *size = ((buff[0] << 8) + (buff[1] << 0)); + buff += 2; + return buff; +} + /* * Compress, fragment, encrypt and HMAC-sign an outgoing packet. * Input: c->c2.buf @@ -1031,6 +1075,7 @@ process_incoming_link_part1(struct context *c, struct link_socket_info *lsi, boo fprintf(stderr, "R"); } #endif + msg(D_LINK_RW, "%s READ [%d] from %s: %s", proto2ascii(lsi->proto, lsi->af, true), BLEN(&c->c2.buf), print_link_socket_actual(&c->c2.from, &gc), PROTO_DUMP(&c->c2.buf, &gc)); @@ -1211,6 +1256,27 @@ process_incoming_link_part2(struct context *c, struct link_socket_info *lsi, } } +void process_incoming_link_part3(struct context *c) +{ + int leng = BLEN(&c->c2.buf); + if (leng > 0) + { + if (check_bulk_mode(c)) + { + c->c2.buffers->send_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->send_tun_max.len = leng; + bcopy(BPTR(&c->c2.buf), BPTR(&c->c2.buffers->send_tun_max), leng); + //dmsg(M_INFO, "FWD BAT LINK 0 [%d] [%d] [%d] [%d] [%d]", BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->send_tun_max)); + c->c2.to_tun.offset += 2; + c->c2.buf.offset += 2; + } + } + else + { + buf_reset(&c->c2.to_tun); + } +} + static void process_incoming_link(struct context *c, struct link_socket *sock) { @@ -1221,6 +1287,7 @@ process_incoming_link(struct context *c, struct link_socket *sock) process_incoming_link_part1(c, lsi, false); process_incoming_link_part2(c, lsi, orig_buf); + process_incoming_link_part3(c); perf_pop(); } @@ -1321,7 +1388,7 @@ process_incoming_dco(struct context *c) */ void -read_incoming_tun(struct context *c) +read_incoming_tun_part2(struct context *c) { /* * Setup for read() call on TUN/TAP device. @@ -1382,6 +1449,55 @@ read_incoming_tun(struct context *c) perf_pop(); } +void read_incoming_tun_part3(struct context *c) +{ + fd_set rfds; + struct timeval timo; + if (check_bulk_mode(c)) + { + int plen = 0; + int fdno = c->c1.tuntap->fd; + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + int leng = plen; + int indx = (c->c2.buffers->bufs_indx + 1); + if (indx >= TUN_BAT_MIN) { break; } + if (leng < 1) + { + FD_ZERO(&rfds); + FD_SET(fdno, &rfds); + timo.tv_sec = 0; + timo.tv_usec = 0; + select(fdno+1, &rfds, NULL, NULL, &timo); + if (FD_ISSET(fdno, &rfds)) + { + read_incoming_tun_part2(c); + plen = BLEN(&c->c2.buf); + } else { break; } + } + //dmsg(M_INFO, "FWD BAT READ 0 [%d] [%d] [%d] [%d] [%d]", c->c2.buffers->bufs_indx + 1, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); + leng = plen; + if (leng > 0) + { + c->c2.buffers->read_tun_bufs[indx].offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_bufs[indx].len = leng; + bcopy(BPTR(&c->c2.buf), BPTR(&c->c2.buffers->read_tun_bufs[indx]), leng); + c->c2.bufs[indx] = c->c2.buffers->read_tun_bufs[indx]; + c->c2.buffers->bufs_indx = indx; + } else { break; } + plen = 0; + } + } +} + +void read_incoming_tun(struct context *c) +{ + if (c->c2.frame.bulk_size <= 0) { + read_incoming_tun_part2(c); + } + read_incoming_tun_part3(c); +} + /** * Drops UDP packets which OS decided to route via tun. * @@ -1469,7 +1585,7 @@ drop_if_recursive_routing(struct context *c, struct buffer *buf) */ void -process_incoming_tun(struct context *c, struct link_socket *out_sock) +process_incoming_tun_part2(struct context *c, struct link_socket *out_sock) { struct gc_arena gc = gc_new(); @@ -1488,7 +1604,7 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) #endif /* Show packet content */ - dmsg(D_TUN_RW, "TUN READ [%d]", BLEN(&c->c2.buf)); + dmsg(D_TUN_RW, "TUN READ [%d] [%d]", BLEN(&c->c2.buf), c->c2.frame.buf.payload_size); if (c->c2.buf.len > 0) { @@ -1512,7 +1628,9 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) } if (c->c2.buf.len > 0) { + if ((c->c2.buffers == NULL) || (c->c2.buffers->flag_ciph != -2)) { encrypt_sign(c, true); + } } else { @@ -1522,6 +1640,67 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) gc_free(&gc); } +void process_incoming_tun_part3(struct context *c, struct link_socket *out_sock) +{ + if (c->c2.buf.len > 0) + { + if (check_bulk_mode(c)) + { + c->c2.buffers->flag_ciph = -2; + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_max.len = 0; + uint8_t *temp = BPTR(&c->c2.buffers->read_tun_max); + int plen = 0, fdno = c->c1.tuntap->fd; + int maxl = 0, leng = (c->c2.buffers->bufs_indx + 1); + if ((fdno > 0) && (leng > 0)) + { + for (int x = 0; x < leng; ++x) + { + c->c2.buf = c->c2.bufs[x]; + //dmsg(M_INFO, "FWD BAT INPT 0 [%d] [%d] [%d] [%d] [%d]", x, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.bufs[x])); + process_incoming_tun_part2(c, out_sock); + if (BLEN(&c->c2.buf) < 1) + { + c->c2.bufs[x].len = 0; + } + } + for (int x = 0; x < leng; ++x) + { + plen = c->c2.bufs[x].len; + if (plen > 0) + { + temp = buff_prepsize(temp, &plen); + bcopy(BPTR(&c->c2.bufs[x]), temp, plen); + temp += plen; maxl += (plen + 2); + } + } + if (maxl > 0) + { + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_max.len = maxl; + c->c2.buf = c->c2.buffers->read_tun_max; + //dmsg(M_INFO, "FWD BAT INPT 1 [%d] [%d] [%d] [%d] [%d]", maxl, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); + encrypt_sign(c, true); + } + } + c->c2.buffers->bufs_indx = -1; + c->c2.buffers->flag_ciph = -1; + } + } + else + { + buf_reset(&c->c2.to_link); + } +} + +void process_incoming_tun(struct context *c, struct link_socket *out_sock) +{ + if (c->c2.frame.bulk_size <= 0) { + process_incoming_tun_part2(c, out_sock); + } + process_incoming_tun_part3(c, out_sock); +} + /** * Forges a IPv6 ICMP packet with a no route to host error code from the * IPv6 packet in buf and sends it directly back to the client via the tun @@ -1748,7 +1927,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) perf_push(PERF_PROC_OUT_LINK); - if (c->c2.to_link.len > 0 && c->c2.to_link.len <= c->c2.frame.buf.payload_size) + if (c->c2.to_link.len > 0 && (c->c2.to_link.len <= c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0)) { /* * Setup for call to send/sendto which will send @@ -1793,6 +1972,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) fprintf(stderr, "W"); } #endif + msg(D_LINK_RW, "%s WRITE [%d] to %s: %s", proto2ascii(sock->info.proto, sock->info.af, true), BLEN(&c->c2.to_link), print_link_socket_actual(c->c2.to_link_addr, &gc), PROTO_DUMP(&c->c2.to_link, &gc)); @@ -1892,7 +2072,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) */ void -process_outgoing_tun(struct context *c, struct link_socket *in_sock) +process_outgoing_tun_part2(struct context *c, struct link_socket *in_sock) { /* * Set up for write() call to TUN/TAP @@ -1912,7 +2092,7 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) process_ip_header(c, PIP_MSSFIX | PIPV4_EXTRACT_DHCP_ROUTER | PIPV4_CLIENT_NAT | PIP_OUTGOING, &c->c2.to_tun, in_sock); - if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size) + if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0) { /* * Write to TUN/TAP device. @@ -1925,7 +2105,8 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) fprintf(stderr, "w"); } #endif - dmsg(D_TUN_RW, "TUN WRITE [%d]", BLEN(&c->c2.to_tun)); + + dmsg(D_TUN_RW, "TUN WRITE [%d] [%d]", BLEN(&c->c2.to_tun), c->c2.frame.buf.payload_size); #ifdef PACKET_TRUNCATION_CHECK ipv4_packet_size_verify(BPTR(&c->c2.to_tun), BLEN(&c->c2.to_tun), TUNNEL_TYPE(c->c1.tuntap), @@ -1981,6 +2162,39 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) perf_pop(); } +void process_outgoing_tun_part3(struct context *c, struct link_socket *in_sock) +{ + if (check_bulk_mode(c)) + { + int maxl = 0, plen = 0; + int leng = BLEN(&c->c2.buffers->send_tun_max); + uint8_t *temp = BPTR(&c->c2.buffers->send_tun_max); + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + temp = buff_postsize(temp, &plen); + if ((leng > 0) && (plen > 0) && ((maxl + plen) < leng)) + { + c->c2.to_tun = c->c2.buffers->to_tun_max; + c->c2.to_tun.offset = TUN_BAT_OFF; + c->c2.to_tun.len = plen; + bcopy(temp, BPTR(&c->c2.to_tun), plen); + temp += plen; maxl += (plen + 2); + //dmsg(M_INFO, "FWD BAT OUTP 1 [%d] [%d] [%d] [%d]", x, BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf)); + process_outgoing_tun_part2(c, in_sock); + } else { break; } + } + } + buf_reset(&c->c2.to_tun); +} + +void process_outgoing_tun(struct context *c, struct link_socket *in_sock) +{ + if (c->c2.frame.bulk_size <= 0) { + process_outgoing_tun_part2(c, in_sock); + } + process_outgoing_tun_part3(c, in_sock); +} + void pre_select(struct context *c) { diff --git a/src/openvpn/forward.h b/src/openvpn/forward.h index d5641491..9fda1583 100644 --- a/src/openvpn/forward.h +++ b/src/openvpn/forward.h @@ -79,6 +79,8 @@ void pre_select(struct context *c); void process_io(struct context *c, struct link_socket *sock); +void xfer_io(struct context *c, struct context *b); + /**********************************************************************/ /** @@ -196,6 +198,8 @@ bool process_incoming_link_part1(struct context *c, struct link_socket_info *lsi void process_incoming_link_part2(struct context *c, struct link_socket_info *lsi, const uint8_t *orig_buf); +void process_incoming_link_part3(struct context *c); + /** * Transfers \c float_sa data extracted from an incoming DCO * PEER_FLOAT_NTF to \c out_osaddr for later processing. diff --git a/src/openvpn/init.c b/src/openvpn/init.c index 40ae2c8c..bbdbad46 100644 --- a/src/openvpn/init.c +++ b/src/openvpn/init.c @@ -2971,6 +2971,10 @@ frame_finalize_options(struct context *c, const struct options *o) tailroom += COMP_EXTRA_BUFFER(payload_size); #endif + if (frame->bulk_size > 0) { + payload_size = frame->tun_mtu; + } + frame->buf.payload_size = payload_size; frame->buf.headroom = headroom; frame->buf.tailroom = tailroom; @@ -3473,6 +3477,9 @@ do_init_frame_tls(struct context *c) if (c->c2.tls_multi) { tls_multi_init_finalize(c->c2.tls_multi, c->options.ce.tls_mtu); + if (c->c2.frame.bulk_size > 0) { + c->c2.tls_multi->opt.frame.buf.payload_size = c->c2.frame.tun_mtu; + } ASSERT(c->c2.tls_multi->opt.frame.buf.payload_size <= c->c2.frame.buf.payload_size); frame_print(&c->c2.tls_multi->opt.frame, D_MTU_INFO, "Control Channel MTU parms"); @@ -3536,6 +3543,14 @@ do_init_frame(struct context *c) c->c2.frame.extra_tun += c->options.ce.tun_mtu_extra; } + /* + * Adjust bulk size based on the --bulk-mode parameter. + */ + if (c->options.ce.bulk_mode) + { + c->c2.frame.bulk_size = c->options.ce.tun_mtu; + } + /* * Fill in the blanks in the frame parameters structure, * make sure values are rational, etc. @@ -3676,9 +3691,41 @@ init_context_buffers(const struct frame *frame) size_t buf_size = BUF_SIZE(frame); + if (frame->bulk_size > 0) { + size_t off_size = (frame->buf.headroom + TUN_BAT_OFF + frame->buf.tailroom); + buf_size = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, off_size); + } + + dmsg(M_INFO, "MEM NEW [%ld] [%d+%d+%d]", buf_size, frame->buf.headroom, frame->buf.payload_size, frame->buf.tailroom); + b->read_link_buf = alloc_buf(buf_size); b->read_tun_buf = alloc_buf(buf_size); + if (frame->bulk_size > 0) { + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + size_t part_size = BUF_SIZE(frame); + b->read_tun_bufs[x] = alloc_buf(part_size); + b->read_tun_bufs[x].offset = TUN_BAT_OFF; + b->read_tun_bufs[x].len = 0; + } + + b->read_tun_max = alloc_buf(buf_size); + b->read_tun_max.offset = TUN_BAT_OFF; + b->read_tun_max.len = 0; + + b->send_tun_max = alloc_buf(buf_size); + b->send_tun_max.offset = TUN_BAT_OFF; + b->send_tun_max.len = 0; + + b->to_tun_max = alloc_buf(buf_size); + b->to_tun_max.offset = TUN_BAT_OFF; + b->to_tun_max.len = 0; + } + + b->bufs_indx = -1; + b->flag_ciph = -1; + b->aux_buf = alloc_buf(buf_size); b->encrypt_buf = alloc_buf(buf_size); @@ -3701,6 +3748,16 @@ free_context_buffers(struct context_buffers *b) free_buf(&b->read_tun_buf); free_buf(&b->aux_buf); + if (b->to_tun_max.data) { + free_buf(&b->to_tun_max); + free_buf(&b->send_tun_max); + free_buf(&b->read_tun_max); + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + free_buf(&b->read_tun_bufs[x]); + } + } + #ifdef USE_COMP free_buf(&b->compress_buf); free_buf(&b->decompress_buf); diff --git a/src/openvpn/mtu.c b/src/openvpn/mtu.c index a419e32d..7e35c837 100644 --- a/src/openvpn/mtu.c +++ b/src/openvpn/mtu.c @@ -41,9 +41,15 @@ void alloc_buf_sock_tun(struct buffer *buf, const struct frame *frame) { /* allocate buffer for overlapped I/O */ - *buf = alloc_buf(BUF_SIZE(frame)); + size_t alen = BUF_SIZE(frame); + size_t blen = frame->buf.payload_size; + if (frame->bulk_size > 0) { + alen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_OFF); + blen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_NOP); + } + *buf = alloc_buf(alen); ASSERT(buf_init(buf, frame->buf.headroom)); - buf->len = frame->buf.payload_size; + buf->len = blen; ASSERT(buf_safe(buf, 0)); } diff --git a/src/openvpn/mtu.h b/src/openvpn/mtu.h index 925ef0bf..eb799fb3 100644 --- a/src/openvpn/mtu.h +++ b/src/openvpn/mtu.h @@ -58,6 +58,14 @@ */ #define TUN_MTU_MIN 100 +/* + * Bulk mode static define values. + */ +#define TUN_BAT_MIN 6 +#define TUN_BAT_MAX 9 +#define TUN_BAT_OFF 256 +#define TUN_BAT_NOP 0 + /* * Default MTU of network over which tunnel data will pass by TCP/UDP. */ @@ -152,6 +160,10 @@ struct frame * which defaults to 0 for tun and 32 * (\c TAP_MTU_EXTRA_DEFAULT) for tap. * */ + + int bulk_size; /**< Signal to the init frame function + * to allow for bulk mode TCP transfers. + * */ }; /* Forward declarations, to prevent includes */ @@ -171,6 +183,7 @@ struct options; * larger than the headroom. */ #define BUF_SIZE(f) ((f)->buf.headroom + (f)->buf.payload_size + (f)->buf.tailroom) +#define BAT_SIZE(a, b, c) ((a * b) + c) /* * Function prototypes. diff --git a/src/openvpn/multi.c b/src/openvpn/multi.c index e1ce32ab..9e089703 100644 --- a/src/openvpn/multi.c +++ b/src/openvpn/multi.c @@ -3414,6 +3414,7 @@ multi_process_incoming_link(struct multi_context *m, struct multi_instance *inst } process_incoming_link_part2(c, lsi, orig_buf); + process_incoming_link_part3(c); } perf_pop(); @@ -3558,9 +3559,7 @@ multi_process_incoming_tun(struct multi_context *m, const unsigned int mpp_flags const int dev_type = TUNNEL_TYPE(m->top.c1.tuntap); int16_t vid = 0; -#ifdef MULTI_DEBUG_EVENT_LOOP - printf("TUN -> TCP/UDP [%d]\n", BLEN(&m->top.c2.buf)); -#endif + msg(D_MULTI_DEBUG, "TUN -> TCP/UDP [%d]", BLEN(&m->top.c2.buf)); if (m->pending) { @@ -3610,6 +3609,8 @@ multi_process_incoming_tun(struct multi_context *m, const unsigned int mpp_flags { /* transfer packet pointer from top-level context buffer to instance */ c->c2.buf = m->top.c2.buf; + /* todo determine if to call this (multi_process_incoming_tun) for each bulk item read? */ + xfer_io(c, &m->top); } else { diff --git a/src/openvpn/openvpn.h b/src/openvpn/openvpn.h index cd99cd40..21fa8967 100644 --- a/src/openvpn/openvpn.h +++ b/src/openvpn/openvpn.h @@ -112,6 +112,14 @@ struct context_buffers */ struct buffer read_link_buf; struct buffer read_tun_buf; + + struct buffer read_tun_bufs[TUN_BAT_MAX]; + struct buffer read_tun_max; + struct buffer send_tun_max; + struct buffer to_tun_max; + + int bufs_indx; + int flag_ciph; }; /* @@ -376,6 +384,8 @@ struct context_2 struct buffer to_tun; struct buffer to_link; + struct buffer bufs[TUN_BAT_MAX]; + /* should we print R|W|r|w to console on packet transfers? */ bool log_rw; diff --git a/src/openvpn/options.c b/src/openvpn/options.c index c54032d8..041d17d0 100644 --- a/src/openvpn/options.c +++ b/src/openvpn/options.c @@ -304,6 +304,7 @@ static const char usage_message[] = " 'maybe' -- Use per-route hints\n" " 'yes' -- Always DF (Don't Fragment)\n" "--mtu-test : Empirically measure and report MTU.\n" + "--bulk-mode : Use bulk TUN/TCP reads/writes.\n" #ifdef ENABLE_FRAGMENT "--fragment max : Enable internal datagram fragmentation so that no UDP\n" " datagrams are sent which are larger than max bytes.\n" @@ -3005,6 +3006,9 @@ options_postprocess_mutate_ce(struct options *o, struct connection_entry *ce) ce->tun_mtu_extra_defined = true; ce->tun_mtu_extra = TAP_MTU_EXTRA_DEFAULT; } + if (ce->proto != PROTO_TCP && ce->proto != PROTO_TCP_SERVER && ce->proto != PROTO_TCP_CLIENT) { + ce->bulk_mode = false; + } } /* @@ -9926,6 +9930,10 @@ add_option(struct options *options, char *p[], bool is_inline, const char *file, goto err; } } + else if (streq(p[0], "bulk-mode")) + { + options->ce.bulk_mode = true; + } else { int i; diff --git a/src/openvpn/options.h b/src/openvpn/options.h index 38e67c8d..d1b0586d 100644 --- a/src/openvpn/options.h +++ b/src/openvpn/options.h @@ -174,6 +174,9 @@ struct connection_entry /* Allow only client that support resending the wrapped client key */ bool tls_crypt_v2_force_cookie; + + /* Bulk mode allows for multiple tun reads + larger tcp writes */ + bool bulk_mode; }; struct remote_entry -- 2.39.5 (Apple Git-154) On Thu, Aug 7, 2025 at 2:29 PM Jon Chiappetta <root@fossjon.com> wrote: > Thanks to Gert's help on this, I was able to finally configure and compile > and run and test the bulk mode changes against the latest git source code > to ensure everything still works correctly. > > I also fixed up some other issues like properly freeing the extra buffer > allocations and removing the unneeded batched data prefixes and converting > a remaining while loop to a max limited for loop and properly resetting the > outgoing tun buffer pointer at the end of the write method when finished. > > Thanks, > Jon C > > Example updated pull request: > https://github.com/OpenVPN/openvpn/pull/814/files > > git formatted diff patch: > > From 985e88a9af26a39554f113f37ee18032a2f41c3e Mon Sep 17 00:00:00 2001 > From: Jon Chiappetta <root@fossjon.com> > Date: Wed, 6 Aug 2025 16:33:18 -0400 > Subject: [PATCH] bulk mode > > --- > src/openvpn/forward.c | 217 ++++++++++++++++++++++++++++++++++++++++-- > src/openvpn/forward.h | 4 + > src/openvpn/init.c | 56 +++++++++++ > src/openvpn/mtu.c | 10 +- > src/openvpn/mtu.h | 13 +++ > src/openvpn/multi.c | 7 +- > src/openvpn/openvpn.h | 10 ++ > src/openvpn/options.c | 8 ++ > src/openvpn/options.h | 3 + > 9 files changed, 316 insertions(+), 12 deletions(-) > > diff --git a/src/openvpn/forward.c b/src/openvpn/forward.c > index 75ca9d5c..d9a98607 100644 > --- a/src/openvpn/forward.c > +++ b/src/openvpn/forward.c > @@ -46,6 +46,9 @@ > > #include "mstats.h" > > +#include <sys/select.h> > +#include <sys/time.h> > + > counter_type link_read_bytes_global; /* GLOBAL */ > counter_type link_write_bytes_global; /* GLOBAL */ > > @@ -78,6 +81,32 @@ show_wait_status(struct context *c) > > #endif /* ifdef ENABLE_DEBUG */ > > +bool check_bulk_mode(struct context *c) > +{ > + if ((c->c2.frame.bulk_size > 0) && (c->c1.tuntap != NULL) && > (c->c2.buffers != NULL)) > + { > + return true; > + } > + return false; > +} > + > +void xfer_io(struct context *c, struct context *b) > +{ > + int plen = 0; > + if (check_bulk_mode(b)) > + { > + int leng = (b->c2.buffers->bufs_indx + 1); > + for (int x = 0; x < leng; ++x) > + { > + plen = BLEN(&b->c2.bufs[x]); > + if (plen < 1) { c->c2.bufs[x].len = 0; } > + else { c->c2.bufs[x] = b->c2.bufs[x]; } > + } > + c->c2.buffers->bufs_indx = b->c2.buffers->bufs_indx; > + b->c2.buffers->bufs_indx = -1; > + } > +} > + > static void > check_tls_errors_co(struct context *c) > { > @@ -605,6 +634,21 @@ buffer_turnover(const uint8_t *orig_buf, struct > buffer *dest_stub, struct buffer > } > } > > +uint8_t *buff_prepsize(uint8_t *buff, int *size) > +{ > + buff[0] = ((*size >> 8) & 0xff); > + buff[1] = ((*size >> 0) & 0xff); > + buff += 2; > + return buff; > +} > + > +uint8_t *buff_postsize(uint8_t *buff, int *size) > +{ > + *size = ((buff[0] << 8) + (buff[1] << 0)); > + buff += 2; > + return buff; > +} > + > /* > * Compress, fragment, encrypt and HMAC-sign an outgoing packet. > * Input: c->c2.buf > @@ -1031,6 +1075,7 @@ process_incoming_link_part1(struct context *c, > struct link_socket_info *lsi, boo > fprintf(stderr, "R"); > } > #endif > + > msg(D_LINK_RW, "%s READ [%d] from %s: %s", proto2ascii(lsi->proto, > lsi->af, true), > BLEN(&c->c2.buf), print_link_socket_actual(&c->c2.from, &gc), > PROTO_DUMP(&c->c2.buf, &gc)); > > @@ -1211,6 +1256,23 @@ process_incoming_link_part2(struct context *c, > struct link_socket_info *lsi, > } > } > > +void process_incoming_link_part3(struct context *c) > +{ > + int leng = BLEN(&c->c2.to_tun); > + if (leng > 0) > + { > + if (check_bulk_mode(c)) > + { > + c->c2.buffers->send_tun_max.offset = TUN_BAT_OFF; > + c->c2.buffers->send_tun_max.len = leng; > + bcopy(BPTR(&c->c2.to_tun), > BPTR(&c->c2.buffers->send_tun_max), leng); > + //dmsg(M_INFO, "FWD BAT LINK 0 [%d] [%d] [%d] [%d] [%d]", > BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf), > BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->send_tun_max)); > + c->c2.to_tun.offset += 2; > + c->c2.buf.offset += 2; > + } > + } > +} > + > static void > process_incoming_link(struct context *c, struct link_socket *sock) > { > @@ -1221,6 +1283,7 @@ process_incoming_link(struct context *c, struct > link_socket *sock) > > process_incoming_link_part1(c, lsi, false); > process_incoming_link_part2(c, lsi, orig_buf); > + process_incoming_link_part3(c); > > perf_pop(); > } > @@ -1321,7 +1384,7 @@ process_incoming_dco(struct context *c) > */ > > void > -read_incoming_tun(struct context *c) > +read_incoming_tun_part2(struct context *c) > { > /* > * Setup for read() call on TUN/TAP device. > @@ -1382,6 +1445,55 @@ read_incoming_tun(struct context *c) > perf_pop(); > } > > +void read_incoming_tun_part3(struct context *c) > +{ > + fd_set rfds; > + struct timeval timo; > + if (check_bulk_mode(c)) > + { > + int plen = 0; > + int fdno = c->c1.tuntap->fd; > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + int leng = plen; > + int indx = (c->c2.buffers->bufs_indx + 1); > + if (indx >= TUN_BAT_MIN) { break; } > + if (leng < 1) > + { > + FD_ZERO(&rfds); > + FD_SET(fdno, &rfds); > + timo.tv_sec = 0; > + timo.tv_usec = 0; > + select(fdno+1, &rfds, NULL, NULL, &timo); > + if (FD_ISSET(fdno, &rfds)) > + { > + read_incoming_tun_part2(c); > + plen = BLEN(&c->c2.buf); > + } else { break; } > + } > + //dmsg(M_INFO, "FWD BAT READ 0 [%d] [%d] [%d] [%d] [%d]", > c->c2.buffers->bufs_indx + 1, fdno, BLEN(&c->c2.buf), > BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); > + leng = plen; > + if (leng > 0) > + { > + c->c2.buffers->read_tun_bufs[indx].offset = TUN_BAT_OFF; > + c->c2.buffers->read_tun_bufs[indx].len = leng; > + bcopy(BPTR(&c->c2.buf), > BPTR(&c->c2.buffers->read_tun_bufs[indx]), leng); > + c->c2.bufs[indx] = c->c2.buffers->read_tun_bufs[indx]; > + c->c2.buffers->bufs_indx = indx; > + } else { break; } > + plen = 0; > + } > + } > +} > + > +void read_incoming_tun(struct context *c) > +{ > + if (c->c2.frame.bulk_size <= 0) { > + read_incoming_tun_part2(c); > + } > + read_incoming_tun_part3(c); > +} > + > /** > * Drops UDP packets which OS decided to route via tun. > * > @@ -1469,7 +1581,7 @@ drop_if_recursive_routing(struct context *c, struct > buffer *buf) > */ > > void > -process_incoming_tun(struct context *c, struct link_socket *out_sock) > +process_incoming_tun_part2(struct context *c, struct link_socket > *out_sock) > { > struct gc_arena gc = gc_new(); > > @@ -1488,7 +1600,7 @@ process_incoming_tun(struct context *c, struct > link_socket *out_sock) > #endif > > /* Show packet content */ > - dmsg(D_TUN_RW, "TUN READ [%d]", BLEN(&c->c2.buf)); > + dmsg(D_TUN_RW, "TUN READ [%d] [%d]", BLEN(&c->c2.buf), > c->c2.frame.buf.payload_size); > > if (c->c2.buf.len > 0) > { > @@ -1512,7 +1624,9 @@ process_incoming_tun(struct context *c, struct > link_socket *out_sock) > } > if (c->c2.buf.len > 0) > { > + if ((c->c2.buffers == NULL) || (c->c2.buffers->flag_ciph != -2)) { > encrypt_sign(c, true); > + } > } > else > { > @@ -1522,6 +1636,60 @@ process_incoming_tun(struct context *c, struct > link_socket *out_sock) > gc_free(&gc); > } > > +void process_incoming_tun_part3(struct context *c, struct link_socket > *out_sock) > +{ > + if (check_bulk_mode(c)) > + { > + c->c2.buffers->flag_ciph = -2; > + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; > + c->c2.buffers->read_tun_max.len = 0; > + uint8_t *temp = BPTR(&c->c2.buffers->read_tun_max); > + int plen = 0, fdno = c->c1.tuntap->fd; > + int maxl = 0, leng = (c->c2.buffers->bufs_indx + 1); > + if ((fdno > 0) && (leng > 0)) > + { > + for (int x = 0; x < leng; ++x) > + { > + c->c2.buf = c->c2.bufs[x]; > + //dmsg(M_INFO, "FWD BAT INPT 0 [%d] [%d] [%d] [%d] [%d]", > x, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), > BLEN(&c->c2.bufs[x])); > + process_incoming_tun_part2(c, out_sock); > + if (BLEN(&c->c2.buf) < 1) > + { > + c->c2.bufs[x].len = 0; > + } > + } > + for (int x = 0; x < leng; ++x) > + { > + plen = c->c2.bufs[x].len; > + if (plen > 0) > + { > + temp = buff_prepsize(temp, &plen); > + bcopy(BPTR(&c->c2.bufs[x]), temp, plen); > + temp += plen; maxl += (plen + 2); > + } > + } > + if (maxl > 0) > + { > + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; > + c->c2.buffers->read_tun_max.len = maxl; > + c->c2.buf = c->c2.buffers->read_tun_max; > + //dmsg(M_INFO, "FWD BAT INPT 1 [%d] [%d] [%d] [%d] [%d]", > maxl, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), > BLEN(&c->c2.buffers->read_tun_max)); > + encrypt_sign(c, true); > + } > + } > + c->c2.buffers->bufs_indx = -1; > + c->c2.buffers->flag_ciph = -1; > + } > +} > + > +void process_incoming_tun(struct context *c, struct link_socket *out_sock) > +{ > + if (c->c2.frame.bulk_size <= 0) { > + process_incoming_tun_part2(c, out_sock); > + } > + process_incoming_tun_part3(c, out_sock); > +} > + > /** > * Forges a IPv6 ICMP packet with a no route to host error code from the > * IPv6 packet in buf and sends it directly back to the client via the tun > @@ -1748,7 +1916,7 @@ process_outgoing_link(struct context *c, struct > link_socket *sock) > > perf_push(PERF_PROC_OUT_LINK); > > - if (c->c2.to_link.len > 0 && c->c2.to_link.len <= > c->c2.frame.buf.payload_size) > + if (c->c2.to_link.len > 0 && (c->c2.to_link.len <= > c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0)) > { > /* > * Setup for call to send/sendto which will send > @@ -1793,6 +1961,7 @@ process_outgoing_link(struct context *c, struct > link_socket *sock) > fprintf(stderr, "W"); > } > #endif > + > msg(D_LINK_RW, "%s WRITE [%d] to %s: %s", > proto2ascii(sock->info.proto, sock->info.af, true), > BLEN(&c->c2.to_link), > print_link_socket_actual(c->c2.to_link_addr, &gc), > PROTO_DUMP(&c->c2.to_link, &gc)); > @@ -1892,7 +2061,7 @@ process_outgoing_link(struct context *c, struct > link_socket *sock) > */ > > void > -process_outgoing_tun(struct context *c, struct link_socket *in_sock) > +process_outgoing_tun_part2(struct context *c, struct link_socket *in_sock) > { > /* > * Set up for write() call to TUN/TAP > @@ -1912,7 +2081,7 @@ process_outgoing_tun(struct context *c, struct > link_socket *in_sock) > process_ip_header(c, PIP_MSSFIX | PIPV4_EXTRACT_DHCP_ROUTER | > PIPV4_CLIENT_NAT | PIP_OUTGOING, > &c->c2.to_tun, in_sock); > > - if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size) > + if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size || > c->c2.frame.bulk_size > 0) > { > /* > * Write to TUN/TAP device. > @@ -1925,7 +2094,8 @@ process_outgoing_tun(struct context *c, struct > link_socket *in_sock) > fprintf(stderr, "w"); > } > #endif > - dmsg(D_TUN_RW, "TUN WRITE [%d]", BLEN(&c->c2.to_tun)); > + > + dmsg(D_TUN_RW, "TUN WRITE [%d] [%d]", BLEN(&c->c2.to_tun), > c->c2.frame.buf.payload_size); > > #ifdef PACKET_TRUNCATION_CHECK > ipv4_packet_size_verify(BPTR(&c->c2.to_tun), BLEN(&c->c2.to_tun), > TUNNEL_TYPE(c->c1.tuntap), > @@ -1981,6 +2151,39 @@ process_outgoing_tun(struct context *c, struct > link_socket *in_sock) > perf_pop(); > } > > +void process_outgoing_tun_part3(struct context *c, struct link_socket > *in_sock) > +{ > + if (check_bulk_mode(c)) > + { > + int maxl = 0, plen = 0; > + int leng = BLEN(&c->c2.buffers->send_tun_max); > + uint8_t *temp = BPTR(&c->c2.buffers->send_tun_max); > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + temp = buff_postsize(temp, &plen); > + if ((leng > 0) && (plen > 0) && ((maxl + plen) < leng)) > + { > + c->c2.to_tun = c->c2.buffers->to_tun_max; > + c->c2.to_tun.offset = TUN_BAT_OFF; > + c->c2.to_tun.len = plen; > + bcopy(temp, BPTR(&c->c2.to_tun), plen); > + temp += plen; maxl += (plen + 2); > + //dmsg(M_INFO, "FWD BAT OUTP 1 [%d] [%d] [%d] [%d]", x, > BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf)); > + process_outgoing_tun_part2(c, in_sock); > + } else { break; } > + } > + buf_reset(&c->c2.to_tun); > + } > +} > + > +void process_outgoing_tun(struct context *c, struct link_socket *in_sock) > +{ > + if (c->c2.frame.bulk_size <= 0) { > + process_outgoing_tun_part2(c, in_sock); > + } > + process_outgoing_tun_part3(c, in_sock); > +} > + > void > pre_select(struct context *c) > { > diff --git a/src/openvpn/forward.h b/src/openvpn/forward.h > index d5641491..9fda1583 100644 > --- a/src/openvpn/forward.h > +++ b/src/openvpn/forward.h > @@ -79,6 +79,8 @@ void pre_select(struct context *c); > > void process_io(struct context *c, struct link_socket *sock); > > +void xfer_io(struct context *c, struct context *b); > + > > /**********************************************************************/ > /** > @@ -196,6 +198,8 @@ bool process_incoming_link_part1(struct context *c, > struct link_socket_info *lsi > void process_incoming_link_part2(struct context *c, struct > link_socket_info *lsi, > const uint8_t *orig_buf); > > +void process_incoming_link_part3(struct context *c); > + > /** > * Transfers \c float_sa data extracted from an incoming DCO > * PEER_FLOAT_NTF to \c out_osaddr for later processing. > diff --git a/src/openvpn/init.c b/src/openvpn/init.c > index 40ae2c8c..0849dfce 100644 > --- a/src/openvpn/init.c > +++ b/src/openvpn/init.c > @@ -2971,6 +2971,10 @@ frame_finalize_options(struct context *c, const > struct options *o) > tailroom += COMP_EXTRA_BUFFER(payload_size); > #endif > > + if (frame->bulk_size > 0) { > + payload_size = frame->tun_mtu; > + } > + > frame->buf.payload_size = payload_size; > frame->buf.headroom = headroom; > frame->buf.tailroom = tailroom; > @@ -3473,6 +3477,9 @@ do_init_frame_tls(struct context *c) > if (c->c2.tls_multi) > { > tls_multi_init_finalize(c->c2.tls_multi, c->options.ce.tls_mtu); > + if (c->c2.frame.bulk_size > 0) { > + c->c2.tls_multi->opt.frame.buf.payload_size = > c->c2.frame.tun_mtu; > + } > ASSERT(c->c2.tls_multi->opt.frame.buf.payload_size <= > c->c2.frame.buf.payload_size); > frame_print(&c->c2.tls_multi->opt.frame, D_MTU_INFO, "Control > Channel MTU parms"); > > @@ -3536,6 +3543,14 @@ do_init_frame(struct context *c) > c->c2.frame.extra_tun += c->options.ce.tun_mtu_extra; > } > > + /* > + * Adjust bulk size based on the --bulk-mode parameter. > + */ > + if (c->options.ce.bulk_mode) > + { > + c->c2.frame.bulk_size = c->options.ce.tun_mtu; > + } > + > /* > * Fill in the blanks in the frame parameters structure, > * make sure values are rational, etc. > @@ -3676,9 +3691,40 @@ init_context_buffers(const struct frame *frame) > > size_t buf_size = BUF_SIZE(frame); > > + if (frame->bulk_size > 0) { > + buf_size = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, > frame->buf.headroom + frame->buf.tailroom); > + } > + > + dmsg(M_INFO, "MEM NEW [%ld] [%d+%d+%d]", buf_size, > frame->buf.headroom, frame->buf.payload_size, frame->buf.tailroom); > + > b->read_link_buf = alloc_buf(buf_size); > b->read_tun_buf = alloc_buf(buf_size); > > + if (frame->bulk_size > 0) { > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + size_t part_size = BUF_SIZE(frame); > + b->read_tun_bufs[x] = alloc_buf(part_size); > + b->read_tun_bufs[x].offset = TUN_BAT_OFF; > + b->read_tun_bufs[x].len = 0; > + } > + > + b->read_tun_max = alloc_buf(buf_size); > + b->read_tun_max.offset = TUN_BAT_OFF; > + b->read_tun_max.len = 0; > + > + b->send_tun_max = alloc_buf(buf_size); > + b->send_tun_max.offset = TUN_BAT_OFF; > + b->send_tun_max.len = 0; > + > + b->to_tun_max = alloc_buf(buf_size); > + b->to_tun_max.offset = TUN_BAT_OFF; > + b->to_tun_max.len = 0; > + } > + > + b->bufs_indx = -1; > + b->flag_ciph = -1; > + > b->aux_buf = alloc_buf(buf_size); > > b->encrypt_buf = alloc_buf(buf_size); > @@ -3701,6 +3747,16 @@ free_context_buffers(struct context_buffers *b) > free_buf(&b->read_tun_buf); > free_buf(&b->aux_buf); > > + if (b->to_tun_max.data) { > + free_buf(&b->to_tun_max); > + free_buf(&b->send_tun_max); > + free_buf(&b->read_tun_max); > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + free_buf(&b->read_tun_bufs[x]); > + } > + } > + > #ifdef USE_COMP > free_buf(&b->compress_buf); > free_buf(&b->decompress_buf); > diff --git a/src/openvpn/mtu.c b/src/openvpn/mtu.c > index a419e32d..7e35c837 100644 > --- a/src/openvpn/mtu.c > +++ b/src/openvpn/mtu.c > @@ -41,9 +41,15 @@ void > alloc_buf_sock_tun(struct buffer *buf, const struct frame *frame) > { > /* allocate buffer for overlapped I/O */ > - *buf = alloc_buf(BUF_SIZE(frame)); > + size_t alen = BUF_SIZE(frame); > + size_t blen = frame->buf.payload_size; > + if (frame->bulk_size > 0) { > + alen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_OFF); > + blen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_NOP); > + } > + *buf = alloc_buf(alen); > ASSERT(buf_init(buf, frame->buf.headroom)); > - buf->len = frame->buf.payload_size; > + buf->len = blen; > ASSERT(buf_safe(buf, 0)); > } > > diff --git a/src/openvpn/mtu.h b/src/openvpn/mtu.h > index 925ef0bf..eb799fb3 100644 > --- a/src/openvpn/mtu.h > +++ b/src/openvpn/mtu.h > @@ -58,6 +58,14 @@ > */ > #define TUN_MTU_MIN 100 > > +/* > + * Bulk mode static define values. > + */ > +#define TUN_BAT_MIN 6 > +#define TUN_BAT_MAX 9 > +#define TUN_BAT_OFF 256 > +#define TUN_BAT_NOP 0 > + > /* > * Default MTU of network over which tunnel data will pass by TCP/UDP. > */ > @@ -152,6 +160,10 @@ struct frame > * which defaults to 0 for tun and 32 > * (\c TAP_MTU_EXTRA_DEFAULT) for tap. > * */ > + > + int bulk_size; /**< Signal to the init frame function > + * to allow for bulk mode TCP transfers. > + * */ > }; > > /* Forward declarations, to prevent includes */ > @@ -171,6 +183,7 @@ struct options; > * larger than the headroom. > */ > #define BUF_SIZE(f) ((f)->buf.headroom + (f)->buf.payload_size + > (f)->buf.tailroom) > +#define BAT_SIZE(a, b, c) ((a * b) + c) > > /* > * Function prototypes. > diff --git a/src/openvpn/multi.c b/src/openvpn/multi.c > index e1ce32ab..9e089703 100644 > --- a/src/openvpn/multi.c > +++ b/src/openvpn/multi.c > @@ -3414,6 +3414,7 @@ multi_process_incoming_link(struct multi_context *m, > struct multi_instance *inst > } > > process_incoming_link_part2(c, lsi, orig_buf); > + process_incoming_link_part3(c); > } > perf_pop(); > > @@ -3558,9 +3559,7 @@ multi_process_incoming_tun(struct multi_context *m, > const unsigned int mpp_flags > const int dev_type = TUNNEL_TYPE(m->top.c1.tuntap); > int16_t vid = 0; > > -#ifdef MULTI_DEBUG_EVENT_LOOP > - printf("TUN -> TCP/UDP [%d]\n", BLEN(&m->top.c2.buf)); > -#endif > + msg(D_MULTI_DEBUG, "TUN -> TCP/UDP [%d]", BLEN(&m->top.c2.buf)); > > if (m->pending) > { > @@ -3610,6 +3609,8 @@ multi_process_incoming_tun(struct multi_context *m, > const unsigned int mpp_flags > { > /* transfer packet pointer from top-level > context buffer to instance */ > c->c2.buf = m->top.c2.buf; > + /* todo determine if to call this > (multi_process_incoming_tun) for each bulk item read? */ > + xfer_io(c, &m->top); > } > else > { > diff --git a/src/openvpn/openvpn.h b/src/openvpn/openvpn.h > index cd99cd40..21fa8967 100644 > --- a/src/openvpn/openvpn.h > +++ b/src/openvpn/openvpn.h > @@ -112,6 +112,14 @@ struct context_buffers > */ > struct buffer read_link_buf; > struct buffer read_tun_buf; > + > + struct buffer read_tun_bufs[TUN_BAT_MAX]; > + struct buffer read_tun_max; > + struct buffer send_tun_max; > + struct buffer to_tun_max; > + > + int bufs_indx; > + int flag_ciph; > }; > > /* > @@ -376,6 +384,8 @@ struct context_2 > struct buffer to_tun; > struct buffer to_link; > > + struct buffer bufs[TUN_BAT_MAX]; > + > /* should we print R|W|r|w to console on packet transfers? */ > bool log_rw; > > diff --git a/src/openvpn/options.c b/src/openvpn/options.c > index c54032d8..041d17d0 100644 > --- a/src/openvpn/options.c > +++ b/src/openvpn/options.c > @@ -304,6 +304,7 @@ static const char usage_message[] = > " 'maybe' -- Use per-route hints\n" > " 'yes' -- Always DF (Don't Fragment)\n" > "--mtu-test : Empirically measure and report MTU.\n" > + "--bulk-mode : Use bulk TUN/TCP reads/writes.\n" > #ifdef ENABLE_FRAGMENT > "--fragment max : Enable internal datagram fragmentation so that no > UDP\n" > " datagrams are sent which are larger than max > bytes.\n" > @@ -3005,6 +3006,9 @@ options_postprocess_mutate_ce(struct options *o, > struct connection_entry *ce) > ce->tun_mtu_extra_defined = true; > ce->tun_mtu_extra = TAP_MTU_EXTRA_DEFAULT; > } > + if (ce->proto != PROTO_TCP && ce->proto != PROTO_TCP_SERVER && > ce->proto != PROTO_TCP_CLIENT) { > + ce->bulk_mode = false; > + } > } > > /* > @@ -9926,6 +9930,10 @@ add_option(struct options *options, char *p[], bool > is_inline, const char *file, > goto err; > } > } > + else if (streq(p[0], "bulk-mode")) > + { > + options->ce.bulk_mode = true; > + } > else > { > int i; > diff --git a/src/openvpn/options.h b/src/openvpn/options.h > index 38e67c8d..d1b0586d 100644 > --- a/src/openvpn/options.h > +++ b/src/openvpn/options.h > @@ -174,6 +174,9 @@ struct connection_entry > > /* Allow only client that support resending the wrapped client key */ > bool tls_crypt_v2_force_cookie; > + > + /* Bulk mode allows for multiple tun reads + larger tcp writes */ > + bool bulk_mode; > }; > > struct remote_entry > -- > 2.39.5 (Apple Git-154) > >
[one last self reply] I did some work today to fix a potential issue with my read tun method where it may not read from the file descriptor upon reconnection. I believe it should be fixed now just in case anyone is still interested in a change like this. Example POC Pull Request Link: https://github.com/OpenVPN/openvpn/pull/814/files $ cat 0001-bulk-mode.patch From 28d43ff49984f2d3c787fa791d0a7a548ac80fad Mon Sep 17 00:00:00 2001 From: Jon Chiappetta <root@fossjon.com> Date: Wed, 6 Aug 2025 16:33:18 -0400 Subject: [PATCH] bulk mode --- src/openvpn/forward.c | 225 ++++++++++++++++++++++++++++++++++++++++-- src/openvpn/forward.h | 4 + src/openvpn/init.c | 57 +++++++++++ src/openvpn/mtu.c | 10 +- src/openvpn/mtu.h | 13 +++ src/openvpn/multi.c | 7 +- src/openvpn/openvpn.h | 10 ++ src/openvpn/options.c | 8 ++ src/openvpn/options.h | 3 + 9 files changed, 325 insertions(+), 12 deletions(-) diff --git a/src/openvpn/forward.c b/src/openvpn/forward.c index 75ca9d5c..b9d96482 100644 --- a/src/openvpn/forward.c +++ b/src/openvpn/forward.c @@ -46,6 +46,9 @@ #include "mstats.h" +#include <sys/select.h> +#include <sys/time.h> + counter_type link_read_bytes_global; /* GLOBAL */ counter_type link_write_bytes_global; /* GLOBAL */ @@ -78,6 +81,34 @@ show_wait_status(struct context *c) #endif /* ifdef ENABLE_DEBUG */ +bool check_bulk_mode(struct context *c) +{ + if ((c->c2.frame.bulk_size > 0) && (c->c1.tuntap != NULL) && (c->c2.buffers != NULL)) + { + return true; + } + return false; +} + +void xfer_io(struct context *c, struct context *b) +{ + //dmsg(M_INFO, "BULK MODE xfer_io c [%d] [%p] [%p] [%d] [%d] [%d]", c->c2.frame.bulk_size, c->c1.tuntap, c->c2.buffers, BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.to_link)); + //dmsg(M_INFO, "BULK MODE xfer_io b [%d] [%p] [%p] [%d] [%d] [%d]", b->c2.frame.bulk_size, b->c1.tuntap, b->c2.buffers, BLEN(&b->c2.buf), BLEN(&b->c2.to_tun), BLEN(&b->c2.to_link)); + int plen = 0; + if (check_bulk_mode(b)) + { + int leng = (b->c2.buffers->bufs_indx + 1); + for (int x = 0; x < leng; ++x) + { + plen = BLEN(&b->c2.bufs[x]); + if (plen < 1) { c->c2.bufs[x].len = 0; } + else { c->c2.bufs[x] = b->c2.bufs[x]; } + } + c->c2.buffers->bufs_indx = b->c2.buffers->bufs_indx; + b->c2.buffers->bufs_indx = -1; + } +} + static void check_tls_errors_co(struct context *c) { @@ -605,6 +636,21 @@ buffer_turnover(const uint8_t *orig_buf, struct buffer *dest_stub, struct buffer } } +uint8_t *buff_prepsize(uint8_t *buff, int *size) +{ + buff[0] = ((*size >> 8) & 0xff); + buff[1] = ((*size >> 0) & 0xff); + buff += 2; + return buff; +} + +uint8_t *buff_postsize(uint8_t *buff, int *size) +{ + *size = ((buff[0] << 8) + (buff[1] << 0)); + buff += 2; + return buff; +} + /* * Compress, fragment, encrypt and HMAC-sign an outgoing packet. * Input: c->c2.buf @@ -1031,6 +1077,7 @@ process_incoming_link_part1(struct context *c, struct link_socket_info *lsi, boo fprintf(stderr, "R"); } #endif + msg(D_LINK_RW, "%s READ [%d] from %s: %s", proto2ascii(lsi->proto, lsi->af, true), BLEN(&c->c2.buf), print_link_socket_actual(&c->c2.from, &gc), PROTO_DUMP(&c->c2.buf, &gc)); @@ -1211,6 +1258,26 @@ process_incoming_link_part2(struct context *c, struct link_socket_info *lsi, } } +void process_incoming_link_part3(struct context *c) +{ + int leng = BLEN(&c->c2.buf); + if (leng > 0) + { + if (check_bulk_mode(c)) + { + c->c2.buffers->send_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->send_tun_max.len = leng; + bcopy(BPTR(&c->c2.buf), BPTR(&c->c2.buffers->send_tun_max), leng); + c->c2.to_tun.offset += 2; + c->c2.buf.offset += 2; + } + } + else + { + buf_reset(&c->c2.to_tun); + } +} + static void process_incoming_link(struct context *c, struct link_socket *sock) { @@ -1221,6 +1288,7 @@ process_incoming_link(struct context *c, struct link_socket *sock) process_incoming_link_part1(c, lsi, false); process_incoming_link_part2(c, lsi, orig_buf); + process_incoming_link_part3(c); perf_pop(); } @@ -1321,7 +1389,7 @@ process_incoming_dco(struct context *c) */ void -read_incoming_tun(struct context *c) +read_incoming_tun_part2(struct context *c) { /* * Setup for read() call on TUN/TAP device. @@ -1382,6 +1450,54 @@ read_incoming_tun(struct context *c) perf_pop(); } +void read_incoming_tun_part3(struct context *c) +{ + fd_set rfds; + struct timeval timo; + if (check_bulk_mode(c)) + { + int plen = 0, pidx = -1; + int fdno = c->c1.tuntap->fd; + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + int leng = plen, indx = (pidx + 1); + if (indx >= TUN_BAT_MIN) { break; } + if (leng < 1) + { + FD_ZERO(&rfds); + FD_SET(fdno, &rfds); + timo.tv_sec = 0; + timo.tv_usec = 0; + select(fdno+1, &rfds, NULL, NULL, &timo); + if (FD_ISSET(fdno, &rfds)) + { + read_incoming_tun_part2(c); + plen = BLEN(&c->c2.buf); + } else { break; } + } + leng = plen; + if (leng > 0) + { + c->c2.buffers->read_tun_bufs[indx].offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_bufs[indx].len = leng; + bcopy(BPTR(&c->c2.buf), BPTR(&c->c2.buffers->read_tun_bufs[indx]), leng); + c->c2.bufs[indx] = c->c2.buffers->read_tun_bufs[indx]; + pidx = indx; + } else { break; } + plen = 0; + } + c->c2.buffers->bufs_indx = pidx; + } +} + +void read_incoming_tun(struct context *c) +{ + if (c->c2.frame.bulk_size <= 0) { + read_incoming_tun_part2(c); + } + read_incoming_tun_part3(c); +} + /** * Drops UDP packets which OS decided to route via tun. * @@ -1469,7 +1585,7 @@ drop_if_recursive_routing(struct context *c, struct buffer *buf) */ void -process_incoming_tun(struct context *c, struct link_socket *out_sock) +process_incoming_tun_part2(struct context *c, struct link_socket *out_sock) { struct gc_arena gc = gc_new(); @@ -1488,7 +1604,7 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) #endif /* Show packet content */ - dmsg(D_TUN_RW, "TUN READ [%d]", BLEN(&c->c2.buf)); + dmsg(D_TUN_RW, "TUN READ [%d] [%d]", BLEN(&c->c2.buf), c->c2.frame.buf.payload_size); if (c->c2.buf.len > 0) { @@ -1512,7 +1628,9 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) } if (c->c2.buf.len > 0) { + if ((c->c2.buffers == NULL) || (c->c2.buffers->flag_ciph != -2)) { encrypt_sign(c, true); + } } else { @@ -1522,6 +1640,65 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) gc_free(&gc); } +void process_incoming_tun_part3(struct context *c, struct link_socket *out_sock) +{ + if (c->c2.buf.len > 0) + { + if (check_bulk_mode(c)) + { + c->c2.buffers->flag_ciph = -2; + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_max.len = 0; + uint8_t *temp = BPTR(&c->c2.buffers->read_tun_max); + int plen = 0, fdno = c->c1.tuntap->fd; + int maxl = 0, leng = (c->c2.buffers->bufs_indx + 1); + if ((fdno > 0) && (leng > 0)) + { + for (int x = 0; x < leng; ++x) + { + c->c2.buf = c->c2.bufs[x]; + process_incoming_tun_part2(c, out_sock); + if (BLEN(&c->c2.buf) < 1) + { + c->c2.bufs[x].len = 0; + } + } + for (int x = 0; x < leng; ++x) + { + plen = c->c2.bufs[x].len; + if (plen > 0) + { + temp = buff_prepsize(temp, &plen); + bcopy(BPTR(&c->c2.bufs[x]), temp, plen); + temp += plen; maxl += (plen + 2); + } + } + if (maxl > 0) + { + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_max.len = maxl; + c->c2.buf = c->c2.buffers->read_tun_max; + encrypt_sign(c, true); + } + } + c->c2.buffers->bufs_indx = -1; + c->c2.buffers->flag_ciph = -1; + } + } + else + { + buf_reset(&c->c2.to_link); + } +} + +void process_incoming_tun(struct context *c, struct link_socket *out_sock) +{ + if (c->c2.frame.bulk_size <= 0) { + process_incoming_tun_part2(c, out_sock); + } + process_incoming_tun_part3(c, out_sock); +} + /** * Forges a IPv6 ICMP packet with a no route to host error code from the * IPv6 packet in buf and sends it directly back to the client via the tun @@ -1748,7 +1925,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) perf_push(PERF_PROC_OUT_LINK); - if (c->c2.to_link.len > 0 && c->c2.to_link.len <= c->c2.frame.buf.payload_size) + if (c->c2.to_link.len > 0 && (c->c2.to_link.len <= c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0)) { /* * Setup for call to send/sendto which will send @@ -1793,6 +1970,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) fprintf(stderr, "W"); } #endif + msg(D_LINK_RW, "%s WRITE [%d] to %s: %s", proto2ascii(sock->info.proto, sock->info.af, true), BLEN(&c->c2.to_link), print_link_socket_actual(c->c2.to_link_addr, &gc), PROTO_DUMP(&c->c2.to_link, &gc)); @@ -1892,7 +2070,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) */ void -process_outgoing_tun(struct context *c, struct link_socket *in_sock) +process_outgoing_tun_part2(struct context *c, struct link_socket *in_sock) { /* * Set up for write() call to TUN/TAP @@ -1912,7 +2090,7 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) process_ip_header(c, PIP_MSSFIX | PIPV4_EXTRACT_DHCP_ROUTER | PIPV4_CLIENT_NAT | PIP_OUTGOING, &c->c2.to_tun, in_sock); - if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size) + if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0) { /* * Write to TUN/TAP device. @@ -1925,7 +2103,8 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) fprintf(stderr, "w"); } #endif - dmsg(D_TUN_RW, "TUN WRITE [%d]", BLEN(&c->c2.to_tun)); + + dmsg(D_TUN_RW, "TUN WRITE [%d] [%d]", BLEN(&c->c2.to_tun), c->c2.frame.buf.payload_size); #ifdef PACKET_TRUNCATION_CHECK ipv4_packet_size_verify(BPTR(&c->c2.to_tun), BLEN(&c->c2.to_tun), TUNNEL_TYPE(c->c1.tuntap), @@ -1981,6 +2160,38 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) perf_pop(); } +void process_outgoing_tun_part3(struct context *c, struct link_socket *in_sock) +{ + if (check_bulk_mode(c)) + { + int maxl = 0, plen = 0; + int leng = BLEN(&c->c2.buffers->send_tun_max); + uint8_t *temp = BPTR(&c->c2.buffers->send_tun_max); + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + temp = buff_postsize(temp, &plen); + if ((leng > 0) && (plen > 0) && ((maxl + plen) < leng)) + { + c->c2.to_tun = c->c2.buffers->to_tun_max; + c->c2.to_tun.offset = TUN_BAT_OFF; + c->c2.to_tun.len = plen; + bcopy(temp, BPTR(&c->c2.to_tun), plen); + temp += plen; maxl += (plen + 2); + process_outgoing_tun_part2(c, in_sock); + } else { break; } + } + } + buf_reset(&c->c2.to_tun); +} + +void process_outgoing_tun(struct context *c, struct link_socket *in_sock) +{ + if (c->c2.frame.bulk_size <= 0) { + process_outgoing_tun_part2(c, in_sock); + } + process_outgoing_tun_part3(c, in_sock); +} + void pre_select(struct context *c) { diff --git a/src/openvpn/forward.h b/src/openvpn/forward.h index d5641491..9fda1583 100644 --- a/src/openvpn/forward.h +++ b/src/openvpn/forward.h @@ -79,6 +79,8 @@ void pre_select(struct context *c); void process_io(struct context *c, struct link_socket *sock); +void xfer_io(struct context *c, struct context *b); + /**********************************************************************/ /** @@ -196,6 +198,8 @@ bool process_incoming_link_part1(struct context *c, struct link_socket_info *lsi void process_incoming_link_part2(struct context *c, struct link_socket_info *lsi, const uint8_t *orig_buf); +void process_incoming_link_part3(struct context *c); + /** * Transfers \c float_sa data extracted from an incoming DCO * PEER_FLOAT_NTF to \c out_osaddr for later processing. diff --git a/src/openvpn/init.c b/src/openvpn/init.c index 40ae2c8c..bbdbad46 100644 --- a/src/openvpn/init.c +++ b/src/openvpn/init.c @@ -2971,6 +2971,10 @@ frame_finalize_options(struct context *c, const struct options *o) tailroom += COMP_EXTRA_BUFFER(payload_size); #endif + if (frame->bulk_size > 0) { + payload_size = frame->tun_mtu; + } + frame->buf.payload_size = payload_size; frame->buf.headroom = headroom; frame->buf.tailroom = tailroom; @@ -3473,6 +3477,9 @@ do_init_frame_tls(struct context *c) if (c->c2.tls_multi) { tls_multi_init_finalize(c->c2.tls_multi, c->options.ce.tls_mtu); + if (c->c2.frame.bulk_size > 0) { + c->c2.tls_multi->opt.frame.buf.payload_size = c->c2.frame.tun_mtu; + } ASSERT(c->c2.tls_multi->opt.frame.buf.payload_size <= c->c2.frame.buf.payload_size); frame_print(&c->c2.tls_multi->opt.frame, D_MTU_INFO, "Control Channel MTU parms"); @@ -3536,6 +3543,14 @@ do_init_frame(struct context *c) c->c2.frame.extra_tun += c->options.ce.tun_mtu_extra; } + /* + * Adjust bulk size based on the --bulk-mode parameter. + */ + if (c->options.ce.bulk_mode) + { + c->c2.frame.bulk_size = c->options.ce.tun_mtu; + } + /* * Fill in the blanks in the frame parameters structure, * make sure values are rational, etc. @@ -3676,9 +3691,41 @@ init_context_buffers(const struct frame *frame) size_t buf_size = BUF_SIZE(frame); + if (frame->bulk_size > 0) { + size_t off_size = (frame->buf.headroom + TUN_BAT_OFF + frame->buf.tailroom); + buf_size = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, off_size); + } + + dmsg(M_INFO, "MEM NEW [%ld] [%d+%d+%d]", buf_size, frame->buf.headroom, frame->buf.payload_size, frame->buf.tailroom); + b->read_link_buf = alloc_buf(buf_size); b->read_tun_buf = alloc_buf(buf_size); + if (frame->bulk_size > 0) { + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + size_t part_size = BUF_SIZE(frame); + b->read_tun_bufs[x] = alloc_buf(part_size); + b->read_tun_bufs[x].offset = TUN_BAT_OFF; + b->read_tun_bufs[x].len = 0; + } + + b->read_tun_max = alloc_buf(buf_size); + b->read_tun_max.offset = TUN_BAT_OFF; + b->read_tun_max.len = 0; + + b->send_tun_max = alloc_buf(buf_size); + b->send_tun_max.offset = TUN_BAT_OFF; + b->send_tun_max.len = 0; + + b->to_tun_max = alloc_buf(buf_size); + b->to_tun_max.offset = TUN_BAT_OFF; + b->to_tun_max.len = 0; + } + + b->bufs_indx = -1; + b->flag_ciph = -1; + b->aux_buf = alloc_buf(buf_size); b->encrypt_buf = alloc_buf(buf_size); @@ -3701,6 +3748,16 @@ free_context_buffers(struct context_buffers *b) free_buf(&b->read_tun_buf); free_buf(&b->aux_buf); + if (b->to_tun_max.data) { + free_buf(&b->to_tun_max); + free_buf(&b->send_tun_max); + free_buf(&b->read_tun_max); + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + free_buf(&b->read_tun_bufs[x]); + } + } + #ifdef USE_COMP free_buf(&b->compress_buf); free_buf(&b->decompress_buf); diff --git a/src/openvpn/mtu.c b/src/openvpn/mtu.c index a419e32d..7e35c837 100644 --- a/src/openvpn/mtu.c +++ b/src/openvpn/mtu.c @@ -41,9 +41,15 @@ void alloc_buf_sock_tun(struct buffer *buf, const struct frame *frame) { /* allocate buffer for overlapped I/O */ - *buf = alloc_buf(BUF_SIZE(frame)); + size_t alen = BUF_SIZE(frame); + size_t blen = frame->buf.payload_size; + if (frame->bulk_size > 0) { + alen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_OFF); + blen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_NOP); + } + *buf = alloc_buf(alen); ASSERT(buf_init(buf, frame->buf.headroom)); - buf->len = frame->buf.payload_size; + buf->len = blen; ASSERT(buf_safe(buf, 0)); } diff --git a/src/openvpn/mtu.h b/src/openvpn/mtu.h index 925ef0bf..eb799fb3 100644 --- a/src/openvpn/mtu.h +++ b/src/openvpn/mtu.h @@ -58,6 +58,14 @@ */ #define TUN_MTU_MIN 100 +/* + * Bulk mode static define values. + */ +#define TUN_BAT_MIN 6 +#define TUN_BAT_MAX 9 +#define TUN_BAT_OFF 256 +#define TUN_BAT_NOP 0 + /* * Default MTU of network over which tunnel data will pass by TCP/UDP. */ @@ -152,6 +160,10 @@ struct frame * which defaults to 0 for tun and 32 * (\c TAP_MTU_EXTRA_DEFAULT) for tap. * */ + + int bulk_size; /**< Signal to the init frame function + * to allow for bulk mode TCP transfers. + * */ }; /* Forward declarations, to prevent includes */ @@ -171,6 +183,7 @@ struct options; * larger than the headroom. */ #define BUF_SIZE(f) ((f)->buf.headroom + (f)->buf.payload_size + (f)->buf.tailroom) +#define BAT_SIZE(a, b, c) ((a * b) + c) /* * Function prototypes. diff --git a/src/openvpn/multi.c b/src/openvpn/multi.c index e1ce32ab..9e089703 100644 --- a/src/openvpn/multi.c +++ b/src/openvpn/multi.c @@ -3414,6 +3414,7 @@ multi_process_incoming_link(struct multi_context *m, struct multi_instance *inst } process_incoming_link_part2(c, lsi, orig_buf); + process_incoming_link_part3(c); } perf_pop(); @@ -3558,9 +3559,7 @@ multi_process_incoming_tun(struct multi_context *m, const unsigned int mpp_flags const int dev_type = TUNNEL_TYPE(m->top.c1.tuntap); int16_t vid = 0; -#ifdef MULTI_DEBUG_EVENT_LOOP - printf("TUN -> TCP/UDP [%d]\n", BLEN(&m->top.c2.buf)); -#endif + msg(D_MULTI_DEBUG, "TUN -> TCP/UDP [%d]", BLEN(&m->top.c2.buf)); if (m->pending) { @@ -3610,6 +3609,8 @@ multi_process_incoming_tun(struct multi_context *m, const unsigned int mpp_flags { /* transfer packet pointer from top-level context buffer to instance */ c->c2.buf = m->top.c2.buf; + /* todo determine if to call this (multi_process_incoming_tun) for each bulk item read? */ + xfer_io(c, &m->top); } else { diff --git a/src/openvpn/openvpn.h b/src/openvpn/openvpn.h index cd99cd40..21fa8967 100644 --- a/src/openvpn/openvpn.h +++ b/src/openvpn/openvpn.h @@ -112,6 +112,14 @@ struct context_buffers */ struct buffer read_link_buf; struct buffer read_tun_buf; + + struct buffer read_tun_bufs[TUN_BAT_MAX]; + struct buffer read_tun_max; + struct buffer send_tun_max; + struct buffer to_tun_max; + + int bufs_indx; + int flag_ciph; }; /* @@ -376,6 +384,8 @@ struct context_2 struct buffer to_tun; struct buffer to_link; + struct buffer bufs[TUN_BAT_MAX]; + /* should we print R|W|r|w to console on packet transfers? */ bool log_rw; diff --git a/src/openvpn/options.c b/src/openvpn/options.c index c54032d8..041d17d0 100644 --- a/src/openvpn/options.c +++ b/src/openvpn/options.c @@ -304,6 +304,7 @@ static const char usage_message[] = " 'maybe' -- Use per-route hints\n" " 'yes' -- Always DF (Don't Fragment)\n" "--mtu-test : Empirically measure and report MTU.\n" + "--bulk-mode : Use bulk TUN/TCP reads/writes.\n" #ifdef ENABLE_FRAGMENT "--fragment max : Enable internal datagram fragmentation so that no UDP\n" " datagrams are sent which are larger than max bytes.\n" @@ -3005,6 +3006,9 @@ options_postprocess_mutate_ce(struct options *o, struct connection_entry *ce) ce->tun_mtu_extra_defined = true; ce->tun_mtu_extra = TAP_MTU_EXTRA_DEFAULT; } + if (ce->proto != PROTO_TCP && ce->proto != PROTO_TCP_SERVER && ce->proto != PROTO_TCP_CLIENT) { + ce->bulk_mode = false; + } } /* @@ -9926,6 +9930,10 @@ add_option(struct options *options, char *p[], bool is_inline, const char *file, goto err; } } + else if (streq(p[0], "bulk-mode")) + { + options->ce.bulk_mode = true; + } else { int i; diff --git a/src/openvpn/options.h b/src/openvpn/options.h index 38e67c8d..d1b0586d 100644 --- a/src/openvpn/options.h +++ b/src/openvpn/options.h @@ -174,6 +174,9 @@ struct connection_entry /* Allow only client that support resending the wrapped client key */ bool tls_crypt_v2_force_cookie; + + /* Bulk mode allows for multiple tun reads + larger tcp writes */ + bool bulk_mode; }; struct remote_entry -- 2.39.5 (Apple Git-154) On Fri, Aug 8, 2025 at 10:23 AM Jon Chiappetta <root@fossjon.com> wrote: > [replying to my own update thread] > > I pushed an update to the PR with the following small changes: > > - rebased to the latest master commit > - increased the data buffer size to be slightly bigger than the read > buffer size > - copied two more buf resets so that the additional bulk forward functions > match the present logic checks > > Example updated pull request: > https://github.com/OpenVPN/openvpn/pull/814/files > > $ cat 0001-bulk-mode.patch > From 2ce0b023d105e7ecc289a414cd26f7ebc8bcbcaf Mon Sep 17 00:00:00 2001 > From: Jon Chiappetta <root@fossjon.com> > Date: Wed, 6 Aug 2025 16:33:18 -0400 > Subject: [PATCH] bulk mode > > --- > src/openvpn/forward.c | 228 ++++++++++++++++++++++++++++++++++++++++-- > src/openvpn/forward.h | 4 + > src/openvpn/init.c | 57 +++++++++++ > src/openvpn/mtu.c | 10 +- > src/openvpn/mtu.h | 13 +++ > src/openvpn/multi.c | 7 +- > src/openvpn/openvpn.h | 10 ++ > src/openvpn/options.c | 8 ++ > src/openvpn/options.h | 3 + > 9 files changed, 328 insertions(+), 12 deletions(-) > > diff --git a/src/openvpn/forward.c b/src/openvpn/forward.c > index 75ca9d5c..0af983e9 100644 > --- a/src/openvpn/forward.c > +++ b/src/openvpn/forward.c > @@ -46,6 +46,9 @@ > > #include "mstats.h" > > +#include <sys/select.h> > +#include <sys/time.h> > + > counter_type link_read_bytes_global; /* GLOBAL */ > counter_type link_write_bytes_global; /* GLOBAL */ > > @@ -78,6 +81,32 @@ show_wait_status(struct context *c) > > #endif /* ifdef ENABLE_DEBUG */ > > +bool check_bulk_mode(struct context *c) > +{ > + if ((c->c2.frame.bulk_size > 0) && (c->c1.tuntap != NULL) && > (c->c2.buffers != NULL)) > + { > + return true; > + } > + return false; > +} > + > +void xfer_io(struct context *c, struct context *b) > +{ > + int plen = 0; > + if (check_bulk_mode(b)) > + { > + int leng = (b->c2.buffers->bufs_indx + 1); > + for (int x = 0; x < leng; ++x) > + { > + plen = BLEN(&b->c2.bufs[x]); > + if (plen < 1) { c->c2.bufs[x].len = 0; } > + else { c->c2.bufs[x] = b->c2.bufs[x]; } > + } > + c->c2.buffers->bufs_indx = b->c2.buffers->bufs_indx; > + b->c2.buffers->bufs_indx = -1; > + } > +} > + > static void > check_tls_errors_co(struct context *c) > { > @@ -605,6 +634,21 @@ buffer_turnover(const uint8_t *orig_buf, struct > buffer *dest_stub, struct buffer > } > } > > +uint8_t *buff_prepsize(uint8_t *buff, int *size) > +{ > + buff[0] = ((*size >> 8) & 0xff); > + buff[1] = ((*size >> 0) & 0xff); > + buff += 2; > + return buff; > +} > + > +uint8_t *buff_postsize(uint8_t *buff, int *size) > +{ > + *size = ((buff[0] << 8) + (buff[1] << 0)); > + buff += 2; > + return buff; > +} > + > /* > * Compress, fragment, encrypt and HMAC-sign an outgoing packet. > * Input: c->c2.buf > @@ -1031,6 +1075,7 @@ process_incoming_link_part1(struct context *c, > struct link_socket_info *lsi, boo > fprintf(stderr, "R"); > } > #endif > + > msg(D_LINK_RW, "%s READ [%d] from %s: %s", proto2ascii(lsi->proto, > lsi->af, true), > BLEN(&c->c2.buf), print_link_socket_actual(&c->c2.from, &gc), > PROTO_DUMP(&c->c2.buf, &gc)); > > @@ -1211,6 +1256,27 @@ process_incoming_link_part2(struct context *c, > struct link_socket_info *lsi, > } > } > > +void process_incoming_link_part3(struct context *c) > +{ > + int leng = BLEN(&c->c2.buf); > + if (leng > 0) > + { > + if (check_bulk_mode(c)) > + { > + c->c2.buffers->send_tun_max.offset = TUN_BAT_OFF; > + c->c2.buffers->send_tun_max.len = leng; > + bcopy(BPTR(&c->c2.buf), BPTR(&c->c2.buffers->send_tun_max), > leng); > + //dmsg(M_INFO, "FWD BAT LINK 0 [%d] [%d] [%d] [%d] [%d]", > BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf), > BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->send_tun_max)); > + c->c2.to_tun.offset += 2; > + c->c2.buf.offset += 2; > + } > + } > + else > + { > + buf_reset(&c->c2.to_tun); > + } > +} > + > static void > process_incoming_link(struct context *c, struct link_socket *sock) > { > @@ -1221,6 +1287,7 @@ process_incoming_link(struct context *c, struct > link_socket *sock) > > process_incoming_link_part1(c, lsi, false); > process_incoming_link_part2(c, lsi, orig_buf); > + process_incoming_link_part3(c); > > perf_pop(); > } > @@ -1321,7 +1388,7 @@ process_incoming_dco(struct context *c) > */ > > void > -read_incoming_tun(struct context *c) > +read_incoming_tun_part2(struct context *c) > { > /* > * Setup for read() call on TUN/TAP device. > @@ -1382,6 +1449,55 @@ read_incoming_tun(struct context *c) > perf_pop(); > } > > +void read_incoming_tun_part3(struct context *c) > +{ > + fd_set rfds; > + struct timeval timo; > + if (check_bulk_mode(c)) > + { > + int plen = 0; > + int fdno = c->c1.tuntap->fd; > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + int leng = plen; > + int indx = (c->c2.buffers->bufs_indx + 1); > + if (indx >= TUN_BAT_MIN) { break; } > + if (leng < 1) > + { > + FD_ZERO(&rfds); > + FD_SET(fdno, &rfds); > + timo.tv_sec = 0; > + timo.tv_usec = 0; > + select(fdno+1, &rfds, NULL, NULL, &timo); > + if (FD_ISSET(fdno, &rfds)) > + { > + read_incoming_tun_part2(c); > + plen = BLEN(&c->c2.buf); > + } else { break; } > + } > + //dmsg(M_INFO, "FWD BAT READ 0 [%d] [%d] [%d] [%d] [%d]", > c->c2.buffers->bufs_indx + 1, fdno, BLEN(&c->c2.buf), > BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); > + leng = plen; > + if (leng > 0) > + { > + c->c2.buffers->read_tun_bufs[indx].offset = TUN_BAT_OFF; > + c->c2.buffers->read_tun_bufs[indx].len = leng; > + bcopy(BPTR(&c->c2.buf), > BPTR(&c->c2.buffers->read_tun_bufs[indx]), leng); > + c->c2.bufs[indx] = c->c2.buffers->read_tun_bufs[indx]; > + c->c2.buffers->bufs_indx = indx; > + } else { break; } > + plen = 0; > + } > + } > +} > + > +void read_incoming_tun(struct context *c) > +{ > + if (c->c2.frame.bulk_size <= 0) { > + read_incoming_tun_part2(c); > + } > + read_incoming_tun_part3(c); > +} > + > /** > * Drops UDP packets which OS decided to route via tun. > * > @@ -1469,7 +1585,7 @@ drop_if_recursive_routing(struct context *c, struct > buffer *buf) > */ > > void > -process_incoming_tun(struct context *c, struct link_socket *out_sock) > +process_incoming_tun_part2(struct context *c, struct link_socket > *out_sock) > { > struct gc_arena gc = gc_new(); > > @@ -1488,7 +1604,7 @@ process_incoming_tun(struct context *c, struct > link_socket *out_sock) > #endif > > /* Show packet content */ > - dmsg(D_TUN_RW, "TUN READ [%d]", BLEN(&c->c2.buf)); > + dmsg(D_TUN_RW, "TUN READ [%d] [%d]", BLEN(&c->c2.buf), > c->c2.frame.buf.payload_size); > > if (c->c2.buf.len > 0) > { > @@ -1512,7 +1628,9 @@ process_incoming_tun(struct context *c, struct > link_socket *out_sock) > } > if (c->c2.buf.len > 0) > { > + if ((c->c2.buffers == NULL) || (c->c2.buffers->flag_ciph != -2)) { > encrypt_sign(c, true); > + } > } > else > { > @@ -1522,6 +1640,67 @@ process_incoming_tun(struct context *c, struct > link_socket *out_sock) > gc_free(&gc); > } > > +void process_incoming_tun_part3(struct context *c, struct link_socket > *out_sock) > +{ > + if (c->c2.buf.len > 0) > + { > + if (check_bulk_mode(c)) > + { > + c->c2.buffers->flag_ciph = -2; > + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; > + c->c2.buffers->read_tun_max.len = 0; > + uint8_t *temp = BPTR(&c->c2.buffers->read_tun_max); > + int plen = 0, fdno = c->c1.tuntap->fd; > + int maxl = 0, leng = (c->c2.buffers->bufs_indx + 1); > + if ((fdno > 0) && (leng > 0)) > + { > + for (int x = 0; x < leng; ++x) > + { > + c->c2.buf = c->c2.bufs[x]; > + //dmsg(M_INFO, "FWD BAT INPT 0 [%d] [%d] [%d] [%d] > [%d]", x, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), > BLEN(&c->c2.bufs[x])); > + process_incoming_tun_part2(c, out_sock); > + if (BLEN(&c->c2.buf) < 1) > + { > + c->c2.bufs[x].len = 0; > + } > + } > + for (int x = 0; x < leng; ++x) > + { > + plen = c->c2.bufs[x].len; > + if (plen > 0) > + { > + temp = buff_prepsize(temp, &plen); > + bcopy(BPTR(&c->c2.bufs[x]), temp, plen); > + temp += plen; maxl += (plen + 2); > + } > + } > + if (maxl > 0) > + { > + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; > + c->c2.buffers->read_tun_max.len = maxl; > + c->c2.buf = c->c2.buffers->read_tun_max; > + //dmsg(M_INFO, "FWD BAT INPT 1 [%d] [%d] [%d] [%d] > [%d]", maxl, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), > BLEN(&c->c2.buffers->read_tun_max)); > + encrypt_sign(c, true); > + } > + } > + c->c2.buffers->bufs_indx = -1; > + c->c2.buffers->flag_ciph = -1; > + } > + } > + else > + { > + buf_reset(&c->c2.to_link); > + } > +} > + > +void process_incoming_tun(struct context *c, struct link_socket *out_sock) > +{ > + if (c->c2.frame.bulk_size <= 0) { > + process_incoming_tun_part2(c, out_sock); > + } > + process_incoming_tun_part3(c, out_sock); > +} > + > /** > * Forges a IPv6 ICMP packet with a no route to host error code from the > * IPv6 packet in buf and sends it directly back to the client via the tun > @@ -1748,7 +1927,7 @@ process_outgoing_link(struct context *c, struct > link_socket *sock) > > perf_push(PERF_PROC_OUT_LINK); > > - if (c->c2.to_link.len > 0 && c->c2.to_link.len <= > c->c2.frame.buf.payload_size) > + if (c->c2.to_link.len > 0 && (c->c2.to_link.len <= > c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0)) > { > /* > * Setup for call to send/sendto which will send > @@ -1793,6 +1972,7 @@ process_outgoing_link(struct context *c, struct > link_socket *sock) > fprintf(stderr, "W"); > } > #endif > + > msg(D_LINK_RW, "%s WRITE [%d] to %s: %s", > proto2ascii(sock->info.proto, sock->info.af, true), > BLEN(&c->c2.to_link), > print_link_socket_actual(c->c2.to_link_addr, &gc), > PROTO_DUMP(&c->c2.to_link, &gc)); > @@ -1892,7 +2072,7 @@ process_outgoing_link(struct context *c, struct > link_socket *sock) > */ > > void > -process_outgoing_tun(struct context *c, struct link_socket *in_sock) > +process_outgoing_tun_part2(struct context *c, struct link_socket *in_sock) > { > /* > * Set up for write() call to TUN/TAP > @@ -1912,7 +2092,7 @@ process_outgoing_tun(struct context *c, struct > link_socket *in_sock) > process_ip_header(c, PIP_MSSFIX | PIPV4_EXTRACT_DHCP_ROUTER | > PIPV4_CLIENT_NAT | PIP_OUTGOING, > &c->c2.to_tun, in_sock); > > - if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size) > + if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size || > c->c2.frame.bulk_size > 0) > { > /* > * Write to TUN/TAP device. > @@ -1925,7 +2105,8 @@ process_outgoing_tun(struct context *c, struct > link_socket *in_sock) > fprintf(stderr, "w"); > } > #endif > - dmsg(D_TUN_RW, "TUN WRITE [%d]", BLEN(&c->c2.to_tun)); > + > + dmsg(D_TUN_RW, "TUN WRITE [%d] [%d]", BLEN(&c->c2.to_tun), > c->c2.frame.buf.payload_size); > > #ifdef PACKET_TRUNCATION_CHECK > ipv4_packet_size_verify(BPTR(&c->c2.to_tun), BLEN(&c->c2.to_tun), > TUNNEL_TYPE(c->c1.tuntap), > @@ -1981,6 +2162,39 @@ process_outgoing_tun(struct context *c, struct > link_socket *in_sock) > perf_pop(); > } > > +void process_outgoing_tun_part3(struct context *c, struct link_socket > *in_sock) > +{ > + if (check_bulk_mode(c)) > + { > + int maxl = 0, plen = 0; > + int leng = BLEN(&c->c2.buffers->send_tun_max); > + uint8_t *temp = BPTR(&c->c2.buffers->send_tun_max); > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + temp = buff_postsize(temp, &plen); > + if ((leng > 0) && (plen > 0) && ((maxl + plen) < leng)) > + { > + c->c2.to_tun = c->c2.buffers->to_tun_max; > + c->c2.to_tun.offset = TUN_BAT_OFF; > + c->c2.to_tun.len = plen; > + bcopy(temp, BPTR(&c->c2.to_tun), plen); > + temp += plen; maxl += (plen + 2); > + //dmsg(M_INFO, "FWD BAT OUTP 1 [%d] [%d] [%d] [%d]", x, > BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf)); > + process_outgoing_tun_part2(c, in_sock); > + } else { break; } > + } > + } > + buf_reset(&c->c2.to_tun); > +} > + > +void process_outgoing_tun(struct context *c, struct link_socket *in_sock) > +{ > + if (c->c2.frame.bulk_size <= 0) { > + process_outgoing_tun_part2(c, in_sock); > + } > + process_outgoing_tun_part3(c, in_sock); > +} > + > void > pre_select(struct context *c) > { > diff --git a/src/openvpn/forward.h b/src/openvpn/forward.h > index d5641491..9fda1583 100644 > --- a/src/openvpn/forward.h > +++ b/src/openvpn/forward.h > @@ -79,6 +79,8 @@ void pre_select(struct context *c); > > void process_io(struct context *c, struct link_socket *sock); > > +void xfer_io(struct context *c, struct context *b); > + > > /**********************************************************************/ > /** > @@ -196,6 +198,8 @@ bool process_incoming_link_part1(struct context *c, > struct link_socket_info *lsi > void process_incoming_link_part2(struct context *c, struct > link_socket_info *lsi, > const uint8_t *orig_buf); > > +void process_incoming_link_part3(struct context *c); > + > /** > * Transfers \c float_sa data extracted from an incoming DCO > * PEER_FLOAT_NTF to \c out_osaddr for later processing. > diff --git a/src/openvpn/init.c b/src/openvpn/init.c > index 40ae2c8c..bbdbad46 100644 > --- a/src/openvpn/init.c > +++ b/src/openvpn/init.c > @@ -2971,6 +2971,10 @@ frame_finalize_options(struct context *c, const > struct options *o) > tailroom += COMP_EXTRA_BUFFER(payload_size); > #endif > > + if (frame->bulk_size > 0) { > + payload_size = frame->tun_mtu; > + } > + > frame->buf.payload_size = payload_size; > frame->buf.headroom = headroom; > frame->buf.tailroom = tailroom; > @@ -3473,6 +3477,9 @@ do_init_frame_tls(struct context *c) > if (c->c2.tls_multi) > { > tls_multi_init_finalize(c->c2.tls_multi, c->options.ce.tls_mtu); > + if (c->c2.frame.bulk_size > 0) { > + c->c2.tls_multi->opt.frame.buf.payload_size = > c->c2.frame.tun_mtu; > + } > ASSERT(c->c2.tls_multi->opt.frame.buf.payload_size <= > c->c2.frame.buf.payload_size); > frame_print(&c->c2.tls_multi->opt.frame, D_MTU_INFO, "Control > Channel MTU parms"); > > @@ -3536,6 +3543,14 @@ do_init_frame(struct context *c) > c->c2.frame.extra_tun += c->options.ce.tun_mtu_extra; > } > > + /* > + * Adjust bulk size based on the --bulk-mode parameter. > + */ > + if (c->options.ce.bulk_mode) > + { > + c->c2.frame.bulk_size = c->options.ce.tun_mtu; > + } > + > /* > * Fill in the blanks in the frame parameters structure, > * make sure values are rational, etc. > @@ -3676,9 +3691,41 @@ init_context_buffers(const struct frame *frame) > > size_t buf_size = BUF_SIZE(frame); > > + if (frame->bulk_size > 0) { > + size_t off_size = (frame->buf.headroom + TUN_BAT_OFF + > frame->buf.tailroom); > + buf_size = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, off_size); > + } > + > + dmsg(M_INFO, "MEM NEW [%ld] [%d+%d+%d]", buf_size, > frame->buf.headroom, frame->buf.payload_size, frame->buf.tailroom); > + > b->read_link_buf = alloc_buf(buf_size); > b->read_tun_buf = alloc_buf(buf_size); > > + if (frame->bulk_size > 0) { > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + size_t part_size = BUF_SIZE(frame); > + b->read_tun_bufs[x] = alloc_buf(part_size); > + b->read_tun_bufs[x].offset = TUN_BAT_OFF; > + b->read_tun_bufs[x].len = 0; > + } > + > + b->read_tun_max = alloc_buf(buf_size); > + b->read_tun_max.offset = TUN_BAT_OFF; > + b->read_tun_max.len = 0; > + > + b->send_tun_max = alloc_buf(buf_size); > + b->send_tun_max.offset = TUN_BAT_OFF; > + b->send_tun_max.len = 0; > + > + b->to_tun_max = alloc_buf(buf_size); > + b->to_tun_max.offset = TUN_BAT_OFF; > + b->to_tun_max.len = 0; > + } > + > + b->bufs_indx = -1; > + b->flag_ciph = -1; > + > b->aux_buf = alloc_buf(buf_size); > > b->encrypt_buf = alloc_buf(buf_size); > @@ -3701,6 +3748,16 @@ free_context_buffers(struct context_buffers *b) > free_buf(&b->read_tun_buf); > free_buf(&b->aux_buf); > > + if (b->to_tun_max.data) { > + free_buf(&b->to_tun_max); > + free_buf(&b->send_tun_max); > + free_buf(&b->read_tun_max); > + for (int x = 0; x < TUN_BAT_MAX; ++x) > + { > + free_buf(&b->read_tun_bufs[x]); > + } > + } > + > #ifdef USE_COMP > free_buf(&b->compress_buf); > free_buf(&b->decompress_buf); > diff --git a/src/openvpn/mtu.c b/src/openvpn/mtu.c > index a419e32d..7e35c837 100644 > --- a/src/openvpn/mtu.c > +++ b/src/openvpn/mtu.c > @@ -41,9 +41,15 @@ void > alloc_buf_sock_tun(struct buffer *buf, const struct frame *frame) > { > /* allocate buffer for overlapped I/O */ > - *buf = alloc_buf(BUF_SIZE(frame)); > + size_t alen = BUF_SIZE(frame); > + size_t blen = frame->buf.payload_size; > + if (frame->bulk_size > 0) { > + alen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_OFF); > + blen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_NOP); > + } > + *buf = alloc_buf(alen); > ASSERT(buf_init(buf, frame->buf.headroom)); > - buf->len = frame->buf.payload_size; > + buf->len = blen; > ASSERT(buf_safe(buf, 0)); > } > > diff --git a/src/openvpn/mtu.h b/src/openvpn/mtu.h > index 925ef0bf..eb799fb3 100644 > --- a/src/openvpn/mtu.h > +++ b/src/openvpn/mtu.h > @@ -58,6 +58,14 @@ > */ > #define TUN_MTU_MIN 100 > > +/* > + * Bulk mode static define values. > + */ > +#define TUN_BAT_MIN 6 > +#define TUN_BAT_MAX 9 > +#define TUN_BAT_OFF 256 > +#define TUN_BAT_NOP 0 > + > /* > * Default MTU of network over which tunnel data will pass by TCP/UDP. > */ > @@ -152,6 +160,10 @@ struct frame > * which defaults to 0 for tun and 32 > * (\c TAP_MTU_EXTRA_DEFAULT) for tap. > * */ > + > + int bulk_size; /**< Signal to the init frame function > + * to allow for bulk mode TCP transfers. > + * */ > }; > > /* Forward declarations, to prevent includes */ > @@ -171,6 +183,7 @@ struct options; > * larger than the headroom. > */ > #define BUF_SIZE(f) ((f)->buf.headroom + (f)->buf.payload_size + > (f)->buf.tailroom) > +#define BAT_SIZE(a, b, c) ((a * b) + c) > > /* > * Function prototypes. > diff --git a/src/openvpn/multi.c b/src/openvpn/multi.c > index e1ce32ab..9e089703 100644 > --- a/src/openvpn/multi.c > +++ b/src/openvpn/multi.c > @@ -3414,6 +3414,7 @@ multi_process_incoming_link(struct multi_context *m, > struct multi_instance *inst > } > > process_incoming_link_part2(c, lsi, orig_buf); > + process_incoming_link_part3(c); > } > perf_pop(); > > @@ -3558,9 +3559,7 @@ multi_process_incoming_tun(struct multi_context *m, > const unsigned int mpp_flags > const int dev_type = TUNNEL_TYPE(m->top.c1.tuntap); > int16_t vid = 0; > > -#ifdef MULTI_DEBUG_EVENT_LOOP > - printf("TUN -> TCP/UDP [%d]\n", BLEN(&m->top.c2.buf)); > -#endif > + msg(D_MULTI_DEBUG, "TUN -> TCP/UDP [%d]", BLEN(&m->top.c2.buf)); > > if (m->pending) > { > @@ -3610,6 +3609,8 @@ multi_process_incoming_tun(struct multi_context *m, > const unsigned int mpp_flags > { > /* transfer packet pointer from top-level > context buffer to instance */ > c->c2.buf = m->top.c2.buf; > + /* todo determine if to call this > (multi_process_incoming_tun) for each bulk item read? */ > + xfer_io(c, &m->top); > } > else > { > diff --git a/src/openvpn/openvpn.h b/src/openvpn/openvpn.h > index cd99cd40..21fa8967 100644 > --- a/src/openvpn/openvpn.h > +++ b/src/openvpn/openvpn.h > @@ -112,6 +112,14 @@ struct context_buffers > */ > struct buffer read_link_buf; > struct buffer read_tun_buf; > + > + struct buffer read_tun_bufs[TUN_BAT_MAX]; > + struct buffer read_tun_max; > + struct buffer send_tun_max; > + struct buffer to_tun_max; > + > + int bufs_indx; > + int flag_ciph; > }; > > /* > @@ -376,6 +384,8 @@ struct context_2 > struct buffer to_tun; > struct buffer to_link; > > + struct buffer bufs[TUN_BAT_MAX]; > + > /* should we print R|W|r|w to console on packet transfers? */ > bool log_rw; > > diff --git a/src/openvpn/options.c b/src/openvpn/options.c > index c54032d8..041d17d0 100644 > --- a/src/openvpn/options.c > +++ b/src/openvpn/options.c > @@ -304,6 +304,7 @@ static const char usage_message[] = > " 'maybe' -- Use per-route hints\n" > " 'yes' -- Always DF (Don't Fragment)\n" > "--mtu-test : Empirically measure and report MTU.\n" > + "--bulk-mode : Use bulk TUN/TCP reads/writes.\n" > #ifdef ENABLE_FRAGMENT > "--fragment max : Enable internal datagram fragmentation so that no > UDP\n" > " datagrams are sent which are larger than max > bytes.\n" > @@ -3005,6 +3006,9 @@ options_postprocess_mutate_ce(struct options *o, > struct connection_entry *ce) > ce->tun_mtu_extra_defined = true; > ce->tun_mtu_extra = TAP_MTU_EXTRA_DEFAULT; > } > + if (ce->proto != PROTO_TCP && ce->proto != PROTO_TCP_SERVER && > ce->proto != PROTO_TCP_CLIENT) { > + ce->bulk_mode = false; > + } > } > > /* > @@ -9926,6 +9930,10 @@ add_option(struct options *options, char *p[], bool > is_inline, const char *file, > goto err; > } > } > + else if (streq(p[0], "bulk-mode")) > + { > + options->ce.bulk_mode = true; > + } > else > { > int i; > diff --git a/src/openvpn/options.h b/src/openvpn/options.h > index 38e67c8d..d1b0586d 100644 > --- a/src/openvpn/options.h > +++ b/src/openvpn/options.h > @@ -174,6 +174,9 @@ struct connection_entry > > /* Allow only client that support resending the wrapped client key */ > bool tls_crypt_v2_force_cookie; > + > + /* Bulk mode allows for multiple tun reads + larger tcp writes */ > + bool bulk_mode; > }; > > struct remote_entry > -- > 2.39.5 (Apple Git-154) > > > > > On Thu, Aug 7, 2025 at 2:29 PM Jon Chiappetta <root@fossjon.com> wrote: > >> Thanks to Gert's help on this, I was able to finally configure and >> compile and run and test the bulk mode changes against the latest git >> source code to ensure everything still works correctly. >> >> I also fixed up some other issues like properly freeing the extra buffer >> allocations and removing the unneeded batched data prefixes and converting >> a remaining while loop to a max limited for loop and properly resetting the >> outgoing tun buffer pointer at the end of the write method when finished. >> >> Thanks, >> Jon C >> >> Example updated pull request: >> https://github.com/OpenVPN/openvpn/pull/814/files >> >> git formatted diff patch: >> >> From 985e88a9af26a39554f113f37ee18032a2f41c3e Mon Sep 17 00:00:00 2001 >> From: Jon Chiappetta <root@fossjon.com> >> Date: Wed, 6 Aug 2025 16:33:18 -0400 >> Subject: [PATCH] bulk mode >> >> --- >> src/openvpn/forward.c | 217 ++++++++++++++++++++++++++++++++++++++++-- >> src/openvpn/forward.h | 4 + >> src/openvpn/init.c | 56 +++++++++++ >> src/openvpn/mtu.c | 10 +- >> src/openvpn/mtu.h | 13 +++ >> src/openvpn/multi.c | 7 +- >> src/openvpn/openvpn.h | 10 ++ >> src/openvpn/options.c | 8 ++ >> src/openvpn/options.h | 3 + >> 9 files changed, 316 insertions(+), 12 deletions(-) >> >> diff --git a/src/openvpn/forward.c b/src/openvpn/forward.c >> index 75ca9d5c..d9a98607 100644 >> --- a/src/openvpn/forward.c >> +++ b/src/openvpn/forward.c >> @@ -46,6 +46,9 @@ >> >> #include "mstats.h" >> >> +#include <sys/select.h> >> +#include <sys/time.h> >> + >> counter_type link_read_bytes_global; /* GLOBAL */ >> counter_type link_write_bytes_global; /* GLOBAL */ >> >> @@ -78,6 +81,32 @@ show_wait_status(struct context *c) >> >> #endif /* ifdef ENABLE_DEBUG */ >> >> +bool check_bulk_mode(struct context *c) >> +{ >> + if ((c->c2.frame.bulk_size > 0) && (c->c1.tuntap != NULL) && >> (c->c2.buffers != NULL)) >> + { >> + return true; >> + } >> + return false; >> +} >> + >> +void xfer_io(struct context *c, struct context *b) >> +{ >> + int plen = 0; >> + if (check_bulk_mode(b)) >> + { >> + int leng = (b->c2.buffers->bufs_indx + 1); >> + for (int x = 0; x < leng; ++x) >> + { >> + plen = BLEN(&b->c2.bufs[x]); >> + if (plen < 1) { c->c2.bufs[x].len = 0; } >> + else { c->c2.bufs[x] = b->c2.bufs[x]; } >> + } >> + c->c2.buffers->bufs_indx = b->c2.buffers->bufs_indx; >> + b->c2.buffers->bufs_indx = -1; >> + } >> +} >> + >> static void >> check_tls_errors_co(struct context *c) >> { >> @@ -605,6 +634,21 @@ buffer_turnover(const uint8_t *orig_buf, struct >> buffer *dest_stub, struct buffer >> } >> } >> >> +uint8_t *buff_prepsize(uint8_t *buff, int *size) >> +{ >> + buff[0] = ((*size >> 8) & 0xff); >> + buff[1] = ((*size >> 0) & 0xff); >> + buff += 2; >> + return buff; >> +} >> + >> +uint8_t *buff_postsize(uint8_t *buff, int *size) >> +{ >> + *size = ((buff[0] << 8) + (buff[1] << 0)); >> + buff += 2; >> + return buff; >> +} >> + >> /* >> * Compress, fragment, encrypt and HMAC-sign an outgoing packet. >> * Input: c->c2.buf >> @@ -1031,6 +1075,7 @@ process_incoming_link_part1(struct context *c, >> struct link_socket_info *lsi, boo >> fprintf(stderr, "R"); >> } >> #endif >> + >> msg(D_LINK_RW, "%s READ [%d] from %s: %s", proto2ascii(lsi->proto, >> lsi->af, true), >> BLEN(&c->c2.buf), print_link_socket_actual(&c->c2.from, &gc), >> PROTO_DUMP(&c->c2.buf, &gc)); >> >> @@ -1211,6 +1256,23 @@ process_incoming_link_part2(struct context *c, >> struct link_socket_info *lsi, >> } >> } >> >> +void process_incoming_link_part3(struct context *c) >> +{ >> + int leng = BLEN(&c->c2.to_tun); >> + if (leng > 0) >> + { >> + if (check_bulk_mode(c)) >> + { >> + c->c2.buffers->send_tun_max.offset = TUN_BAT_OFF; >> + c->c2.buffers->send_tun_max.len = leng; >> + bcopy(BPTR(&c->c2.to_tun), >> BPTR(&c->c2.buffers->send_tun_max), leng); >> + //dmsg(M_INFO, "FWD BAT LINK 0 [%d] [%d] [%d] [%d] [%d]", >> BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf), >> BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->send_tun_max)); >> + c->c2.to_tun.offset += 2; >> + c->c2.buf.offset += 2; >> + } >> + } >> +} >> + >> static void >> process_incoming_link(struct context *c, struct link_socket *sock) >> { >> @@ -1221,6 +1283,7 @@ process_incoming_link(struct context *c, struct >> link_socket *sock) >> >> process_incoming_link_part1(c, lsi, false); >> process_incoming_link_part2(c, lsi, orig_buf); >> + process_incoming_link_part3(c); >> >> perf_pop(); >> } >> @@ -1321,7 +1384,7 @@ process_incoming_dco(struct context *c) >> */ >> >> void >> -read_incoming_tun(struct context *c) >> +read_incoming_tun_part2(struct context *c) >> { >> /* >> * Setup for read() call on TUN/TAP device. >> @@ -1382,6 +1445,55 @@ read_incoming_tun(struct context *c) >> perf_pop(); >> } >> >> +void read_incoming_tun_part3(struct context *c) >> +{ >> + fd_set rfds; >> + struct timeval timo; >> + if (check_bulk_mode(c)) >> + { >> + int plen = 0; >> + int fdno = c->c1.tuntap->fd; >> + for (int x = 0; x < TUN_BAT_MAX; ++x) >> + { >> + int leng = plen; >> + int indx = (c->c2.buffers->bufs_indx + 1); >> + if (indx >= TUN_BAT_MIN) { break; } >> + if (leng < 1) >> + { >> + FD_ZERO(&rfds); >> + FD_SET(fdno, &rfds); >> + timo.tv_sec = 0; >> + timo.tv_usec = 0; >> + select(fdno+1, &rfds, NULL, NULL, &timo); >> + if (FD_ISSET(fdno, &rfds)) >> + { >> + read_incoming_tun_part2(c); >> + plen = BLEN(&c->c2.buf); >> + } else { break; } >> + } >> + //dmsg(M_INFO, "FWD BAT READ 0 [%d] [%d] [%d] [%d] [%d]", >> c->c2.buffers->bufs_indx + 1, fdno, BLEN(&c->c2.buf), >> BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); >> + leng = plen; >> + if (leng > 0) >> + { >> + c->c2.buffers->read_tun_bufs[indx].offset = TUN_BAT_OFF; >> + c->c2.buffers->read_tun_bufs[indx].len = leng; >> + bcopy(BPTR(&c->c2.buf), >> BPTR(&c->c2.buffers->read_tun_bufs[indx]), leng); >> + c->c2.bufs[indx] = c->c2.buffers->read_tun_bufs[indx]; >> + c->c2.buffers->bufs_indx = indx; >> + } else { break; } >> + plen = 0; >> + } >> + } >> +} >> + >> +void read_incoming_tun(struct context *c) >> +{ >> + if (c->c2.frame.bulk_size <= 0) { >> + read_incoming_tun_part2(c); >> + } >> + read_incoming_tun_part3(c); >> +} >> + >> /** >> * Drops UDP packets which OS decided to route via tun. >> * >> @@ -1469,7 +1581,7 @@ drop_if_recursive_routing(struct context *c, struct >> buffer *buf) >> */ >> >> void >> -process_incoming_tun(struct context *c, struct link_socket *out_sock) >> +process_incoming_tun_part2(struct context *c, struct link_socket >> *out_sock) >> { >> struct gc_arena gc = gc_new(); >> >> @@ -1488,7 +1600,7 @@ process_incoming_tun(struct context *c, struct >> link_socket *out_sock) >> #endif >> >> /* Show packet content */ >> - dmsg(D_TUN_RW, "TUN READ [%d]", BLEN(&c->c2.buf)); >> + dmsg(D_TUN_RW, "TUN READ [%d] [%d]", BLEN(&c->c2.buf), >> c->c2.frame.buf.payload_size); >> >> if (c->c2.buf.len > 0) >> { >> @@ -1512,7 +1624,9 @@ process_incoming_tun(struct context *c, struct >> link_socket *out_sock) >> } >> if (c->c2.buf.len > 0) >> { >> + if ((c->c2.buffers == NULL) || (c->c2.buffers->flag_ciph != -2)) >> { >> encrypt_sign(c, true); >> + } >> } >> else >> { >> @@ -1522,6 +1636,60 @@ process_incoming_tun(struct context *c, struct >> link_socket *out_sock) >> gc_free(&gc); >> } >> >> +void process_incoming_tun_part3(struct context *c, struct link_socket >> *out_sock) >> +{ >> + if (check_bulk_mode(c)) >> + { >> + c->c2.buffers->flag_ciph = -2; >> + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; >> + c->c2.buffers->read_tun_max.len = 0; >> + uint8_t *temp = BPTR(&c->c2.buffers->read_tun_max); >> + int plen = 0, fdno = c->c1.tuntap->fd; >> + int maxl = 0, leng = (c->c2.buffers->bufs_indx + 1); >> + if ((fdno > 0) && (leng > 0)) >> + { >> + for (int x = 0; x < leng; ++x) >> + { >> + c->c2.buf = c->c2.bufs[x]; >> + //dmsg(M_INFO, "FWD BAT INPT 0 [%d] [%d] [%d] [%d] >> [%d]", x, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), >> BLEN(&c->c2.bufs[x])); >> + process_incoming_tun_part2(c, out_sock); >> + if (BLEN(&c->c2.buf) < 1) >> + { >> + c->c2.bufs[x].len = 0; >> + } >> + } >> + for (int x = 0; x < leng; ++x) >> + { >> + plen = c->c2.bufs[x].len; >> + if (plen > 0) >> + { >> + temp = buff_prepsize(temp, &plen); >> + bcopy(BPTR(&c->c2.bufs[x]), temp, plen); >> + temp += plen; maxl += (plen + 2); >> + } >> + } >> + if (maxl > 0) >> + { >> + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; >> + c->c2.buffers->read_tun_max.len = maxl; >> + c->c2.buf = c->c2.buffers->read_tun_max; >> + //dmsg(M_INFO, "FWD BAT INPT 1 [%d] [%d] [%d] [%d] >> [%d]", maxl, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), >> BLEN(&c->c2.buffers->read_tun_max)); >> + encrypt_sign(c, true); >> + } >> + } >> + c->c2.buffers->bufs_indx = -1; >> + c->c2.buffers->flag_ciph = -1; >> + } >> +} >> + >> +void process_incoming_tun(struct context *c, struct link_socket >> *out_sock) >> +{ >> + if (c->c2.frame.bulk_size <= 0) { >> + process_incoming_tun_part2(c, out_sock); >> + } >> + process_incoming_tun_part3(c, out_sock); >> +} >> + >> /** >> * Forges a IPv6 ICMP packet with a no route to host error code from the >> * IPv6 packet in buf and sends it directly back to the client via the >> tun >> @@ -1748,7 +1916,7 @@ process_outgoing_link(struct context *c, struct >> link_socket *sock) >> >> perf_push(PERF_PROC_OUT_LINK); >> >> - if (c->c2.to_link.len > 0 && c->c2.to_link.len <= >> c->c2.frame.buf.payload_size) >> + if (c->c2.to_link.len > 0 && (c->c2.to_link.len <= >> c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0)) >> { >> /* >> * Setup for call to send/sendto which will send >> @@ -1793,6 +1961,7 @@ process_outgoing_link(struct context *c, struct >> link_socket *sock) >> fprintf(stderr, "W"); >> } >> #endif >> + >> msg(D_LINK_RW, "%s WRITE [%d] to %s: %s", >> proto2ascii(sock->info.proto, sock->info.af, true), >> BLEN(&c->c2.to_link), >> print_link_socket_actual(c->c2.to_link_addr, &gc), >> PROTO_DUMP(&c->c2.to_link, &gc)); >> @@ -1892,7 +2061,7 @@ process_outgoing_link(struct context *c, struct >> link_socket *sock) >> */ >> >> void >> -process_outgoing_tun(struct context *c, struct link_socket *in_sock) >> +process_outgoing_tun_part2(struct context *c, struct link_socket >> *in_sock) >> { >> /* >> * Set up for write() call to TUN/TAP >> @@ -1912,7 +2081,7 @@ process_outgoing_tun(struct context *c, struct >> link_socket *in_sock) >> process_ip_header(c, PIP_MSSFIX | PIPV4_EXTRACT_DHCP_ROUTER | >> PIPV4_CLIENT_NAT | PIP_OUTGOING, >> &c->c2.to_tun, in_sock); >> >> - if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size) >> + if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size || >> c->c2.frame.bulk_size > 0) >> { >> /* >> * Write to TUN/TAP device. >> @@ -1925,7 +2094,8 @@ process_outgoing_tun(struct context *c, struct >> link_socket *in_sock) >> fprintf(stderr, "w"); >> } >> #endif >> - dmsg(D_TUN_RW, "TUN WRITE [%d]", BLEN(&c->c2.to_tun)); >> + >> + dmsg(D_TUN_RW, "TUN WRITE [%d] [%d]", BLEN(&c->c2.to_tun), >> c->c2.frame.buf.payload_size); >> >> #ifdef PACKET_TRUNCATION_CHECK >> ipv4_packet_size_verify(BPTR(&c->c2.to_tun), >> BLEN(&c->c2.to_tun), TUNNEL_TYPE(c->c1.tuntap), >> @@ -1981,6 +2151,39 @@ process_outgoing_tun(struct context *c, struct >> link_socket *in_sock) >> perf_pop(); >> } >> >> +void process_outgoing_tun_part3(struct context *c, struct link_socket >> *in_sock) >> +{ >> + if (check_bulk_mode(c)) >> + { >> + int maxl = 0, plen = 0; >> + int leng = BLEN(&c->c2.buffers->send_tun_max); >> + uint8_t *temp = BPTR(&c->c2.buffers->send_tun_max); >> + for (int x = 0; x < TUN_BAT_MAX; ++x) >> + { >> + temp = buff_postsize(temp, &plen); >> + if ((leng > 0) && (plen > 0) && ((maxl + plen) < leng)) >> + { >> + c->c2.to_tun = c->c2.buffers->to_tun_max; >> + c->c2.to_tun.offset = TUN_BAT_OFF; >> + c->c2.to_tun.len = plen; >> + bcopy(temp, BPTR(&c->c2.to_tun), plen); >> + temp += plen; maxl += (plen + 2); >> + //dmsg(M_INFO, "FWD BAT OUTP 1 [%d] [%d] [%d] [%d]", x, >> BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf)); >> + process_outgoing_tun_part2(c, in_sock); >> + } else { break; } >> + } >> + buf_reset(&c->c2.to_tun); >> + } >> +} >> + >> +void process_outgoing_tun(struct context *c, struct link_socket *in_sock) >> +{ >> + if (c->c2.frame.bulk_size <= 0) { >> + process_outgoing_tun_part2(c, in_sock); >> + } >> + process_outgoing_tun_part3(c, in_sock); >> +} >> + >> void >> pre_select(struct context *c) >> { >> diff --git a/src/openvpn/forward.h b/src/openvpn/forward.h >> index d5641491..9fda1583 100644 >> --- a/src/openvpn/forward.h >> +++ b/src/openvpn/forward.h >> @@ -79,6 +79,8 @@ void pre_select(struct context *c); >> >> void process_io(struct context *c, struct link_socket *sock); >> >> +void xfer_io(struct context *c, struct context *b); >> + >> >> /**********************************************************************/ >> /** >> @@ -196,6 +198,8 @@ bool process_incoming_link_part1(struct context *c, >> struct link_socket_info *lsi >> void process_incoming_link_part2(struct context *c, struct >> link_socket_info *lsi, >> const uint8_t *orig_buf); >> >> +void process_incoming_link_part3(struct context *c); >> + >> /** >> * Transfers \c float_sa data extracted from an incoming DCO >> * PEER_FLOAT_NTF to \c out_osaddr for later processing. >> diff --git a/src/openvpn/init.c b/src/openvpn/init.c >> index 40ae2c8c..0849dfce 100644 >> --- a/src/openvpn/init.c >> +++ b/src/openvpn/init.c >> @@ -2971,6 +2971,10 @@ frame_finalize_options(struct context *c, const >> struct options *o) >> tailroom += COMP_EXTRA_BUFFER(payload_size); >> #endif >> >> + if (frame->bulk_size > 0) { >> + payload_size = frame->tun_mtu; >> + } >> + >> frame->buf.payload_size = payload_size; >> frame->buf.headroom = headroom; >> frame->buf.tailroom = tailroom; >> @@ -3473,6 +3477,9 @@ do_init_frame_tls(struct context *c) >> if (c->c2.tls_multi) >> { >> tls_multi_init_finalize(c->c2.tls_multi, c->options.ce.tls_mtu); >> + if (c->c2.frame.bulk_size > 0) { >> + c->c2.tls_multi->opt.frame.buf.payload_size = >> c->c2.frame.tun_mtu; >> + } >> ASSERT(c->c2.tls_multi->opt.frame.buf.payload_size <= >> c->c2.frame.buf.payload_size); >> frame_print(&c->c2.tls_multi->opt.frame, D_MTU_INFO, "Control >> Channel MTU parms"); >> >> @@ -3536,6 +3543,14 @@ do_init_frame(struct context *c) >> c->c2.frame.extra_tun += c->options.ce.tun_mtu_extra; >> } >> >> + /* >> + * Adjust bulk size based on the --bulk-mode parameter. >> + */ >> + if (c->options.ce.bulk_mode) >> + { >> + c->c2.frame.bulk_size = c->options.ce.tun_mtu; >> + } >> + >> /* >> * Fill in the blanks in the frame parameters structure, >> * make sure values are rational, etc. >> @@ -3676,9 +3691,40 @@ init_context_buffers(const struct frame *frame) >> >> size_t buf_size = BUF_SIZE(frame); >> >> + if (frame->bulk_size > 0) { >> + buf_size = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, >> frame->buf.headroom + frame->buf.tailroom); >> + } >> + >> + dmsg(M_INFO, "MEM NEW [%ld] [%d+%d+%d]", buf_size, >> frame->buf.headroom, frame->buf.payload_size, frame->buf.tailroom); >> + >> b->read_link_buf = alloc_buf(buf_size); >> b->read_tun_buf = alloc_buf(buf_size); >> >> + if (frame->bulk_size > 0) { >> + for (int x = 0; x < TUN_BAT_MAX; ++x) >> + { >> + size_t part_size = BUF_SIZE(frame); >> + b->read_tun_bufs[x] = alloc_buf(part_size); >> + b->read_tun_bufs[x].offset = TUN_BAT_OFF; >> + b->read_tun_bufs[x].len = 0; >> + } >> + >> + b->read_tun_max = alloc_buf(buf_size); >> + b->read_tun_max.offset = TUN_BAT_OFF; >> + b->read_tun_max.len = 0; >> + >> + b->send_tun_max = alloc_buf(buf_size); >> + b->send_tun_max.offset = TUN_BAT_OFF; >> + b->send_tun_max.len = 0; >> + >> + b->to_tun_max = alloc_buf(buf_size); >> + b->to_tun_max.offset = TUN_BAT_OFF; >> + b->to_tun_max.len = 0; >> + } >> + >> + b->bufs_indx = -1; >> + b->flag_ciph = -1; >> + >> b->aux_buf = alloc_buf(buf_size); >> >> b->encrypt_buf = alloc_buf(buf_size); >> @@ -3701,6 +3747,16 @@ free_context_buffers(struct context_buffers *b) >> free_buf(&b->read_tun_buf); >> free_buf(&b->aux_buf); >> >> + if (b->to_tun_max.data) { >> + free_buf(&b->to_tun_max); >> + free_buf(&b->send_tun_max); >> + free_buf(&b->read_tun_max); >> + for (int x = 0; x < TUN_BAT_MAX; ++x) >> + { >> + free_buf(&b->read_tun_bufs[x]); >> + } >> + } >> + >> #ifdef USE_COMP >> free_buf(&b->compress_buf); >> free_buf(&b->decompress_buf); >> diff --git a/src/openvpn/mtu.c b/src/openvpn/mtu.c >> index a419e32d..7e35c837 100644 >> --- a/src/openvpn/mtu.c >> +++ b/src/openvpn/mtu.c >> @@ -41,9 +41,15 @@ void >> alloc_buf_sock_tun(struct buffer *buf, const struct frame *frame) >> { >> /* allocate buffer for overlapped I/O */ >> - *buf = alloc_buf(BUF_SIZE(frame)); >> + size_t alen = BUF_SIZE(frame); >> + size_t blen = frame->buf.payload_size; >> + if (frame->bulk_size > 0) { >> + alen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_OFF); >> + blen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_NOP); >> + } >> + *buf = alloc_buf(alen); >> ASSERT(buf_init(buf, frame->buf.headroom)); >> - buf->len = frame->buf.payload_size; >> + buf->len = blen; >> ASSERT(buf_safe(buf, 0)); >> } >> >> diff --git a/src/openvpn/mtu.h b/src/openvpn/mtu.h >> index 925ef0bf..eb799fb3 100644 >> --- a/src/openvpn/mtu.h >> +++ b/src/openvpn/mtu.h >> @@ -58,6 +58,14 @@ >> */ >> #define TUN_MTU_MIN 100 >> >> +/* >> + * Bulk mode static define values. >> + */ >> +#define TUN_BAT_MIN 6 >> +#define TUN_BAT_MAX 9 >> +#define TUN_BAT_OFF 256 >> +#define TUN_BAT_NOP 0 >> + >> /* >> * Default MTU of network over which tunnel data will pass by TCP/UDP. >> */ >> @@ -152,6 +160,10 @@ struct frame >> * which defaults to 0 for tun and 32 >> * (\c TAP_MTU_EXTRA_DEFAULT) for tap. >> * */ >> + >> + int bulk_size; /**< Signal to the init frame function >> + * to allow for bulk mode TCP >> transfers. >> + * */ >> }; >> >> /* Forward declarations, to prevent includes */ >> @@ -171,6 +183,7 @@ struct options; >> * larger than the headroom. >> */ >> #define BUF_SIZE(f) ((f)->buf.headroom + (f)->buf.payload_size + >> (f)->buf.tailroom) >> +#define BAT_SIZE(a, b, c) ((a * b) + c) >> >> /* >> * Function prototypes. >> diff --git a/src/openvpn/multi.c b/src/openvpn/multi.c >> index e1ce32ab..9e089703 100644 >> --- a/src/openvpn/multi.c >> +++ b/src/openvpn/multi.c >> @@ -3414,6 +3414,7 @@ multi_process_incoming_link(struct multi_context >> *m, struct multi_instance *inst >> } >> >> process_incoming_link_part2(c, lsi, orig_buf); >> + process_incoming_link_part3(c); >> } >> perf_pop(); >> >> @@ -3558,9 +3559,7 @@ multi_process_incoming_tun(struct multi_context *m, >> const unsigned int mpp_flags >> const int dev_type = TUNNEL_TYPE(m->top.c1.tuntap); >> int16_t vid = 0; >> >> -#ifdef MULTI_DEBUG_EVENT_LOOP >> - printf("TUN -> TCP/UDP [%d]\n", BLEN(&m->top.c2.buf)); >> -#endif >> + msg(D_MULTI_DEBUG, "TUN -> TCP/UDP [%d]", BLEN(&m->top.c2.buf)); >> >> if (m->pending) >> { >> @@ -3610,6 +3609,8 @@ multi_process_incoming_tun(struct multi_context *m, >> const unsigned int mpp_flags >> { >> /* transfer packet pointer from top-level >> context buffer to instance */ >> c->c2.buf = m->top.c2.buf; >> + /* todo determine if to call this >> (multi_process_incoming_tun) for each bulk item read? */ >> + xfer_io(c, &m->top); >> } >> else >> { >> diff --git a/src/openvpn/openvpn.h b/src/openvpn/openvpn.h >> index cd99cd40..21fa8967 100644 >> --- a/src/openvpn/openvpn.h >> +++ b/src/openvpn/openvpn.h >> @@ -112,6 +112,14 @@ struct context_buffers >> */ >> struct buffer read_link_buf; >> struct buffer read_tun_buf; >> + >> + struct buffer read_tun_bufs[TUN_BAT_MAX]; >> + struct buffer read_tun_max; >> + struct buffer send_tun_max; >> + struct buffer to_tun_max; >> + >> + int bufs_indx; >> + int flag_ciph; >> }; >> >> /* >> @@ -376,6 +384,8 @@ struct context_2 >> struct buffer to_tun; >> struct buffer to_link; >> >> + struct buffer bufs[TUN_BAT_MAX]; >> + >> /* should we print R|W|r|w to console on packet transfers? */ >> bool log_rw; >> >> diff --git a/src/openvpn/options.c b/src/openvpn/options.c >> index c54032d8..041d17d0 100644 >> --- a/src/openvpn/options.c >> +++ b/src/openvpn/options.c >> @@ -304,6 +304,7 @@ static const char usage_message[] = >> " 'maybe' -- Use per-route hints\n" >> " 'yes' -- Always DF (Don't Fragment)\n" >> "--mtu-test : Empirically measure and report MTU.\n" >> + "--bulk-mode : Use bulk TUN/TCP reads/writes.\n" >> #ifdef ENABLE_FRAGMENT >> "--fragment max : Enable internal datagram fragmentation so that no >> UDP\n" >> " datagrams are sent which are larger than max >> bytes.\n" >> @@ -3005,6 +3006,9 @@ options_postprocess_mutate_ce(struct options *o, >> struct connection_entry *ce) >> ce->tun_mtu_extra_defined = true; >> ce->tun_mtu_extra = TAP_MTU_EXTRA_DEFAULT; >> } >> + if (ce->proto != PROTO_TCP && ce->proto != PROTO_TCP_SERVER && >> ce->proto != PROTO_TCP_CLIENT) { >> + ce->bulk_mode = false; >> + } >> } >> >> /* >> @@ -9926,6 +9930,10 @@ add_option(struct options *options, char *p[], >> bool is_inline, const char *file, >> goto err; >> } >> } >> + else if (streq(p[0], "bulk-mode")) >> + { >> + options->ce.bulk_mode = true; >> + } >> else >> { >> int i; >> diff --git a/src/openvpn/options.h b/src/openvpn/options.h >> index 38e67c8d..d1b0586d 100644 >> --- a/src/openvpn/options.h >> +++ b/src/openvpn/options.h >> @@ -174,6 +174,9 @@ struct connection_entry >> >> /* Allow only client that support resending the wrapped client key */ >> bool tls_crypt_v2_force_cookie; >> + >> + /* Bulk mode allows for multiple tun reads + larger tcp writes */ >> + bool bulk_mode; >> }; >> >> struct remote_entry >> -- >> 2.39.5 (Apple Git-154) >> >>
Am 08.08.2025 um 16:04 schrieb Jon Chiappetta: > Hi Arne, > > You are correct, I didn't do a very good job of explaining the code in > my blog post, I usually keep those short with more screen captures > because I figure that not many people would actually take the time to > read through it there. Also, I didn't really add many comments either > but I did try to copy the present style of the code to try and make it > match and be more consistent throughout, even if not perfect yet. I'm > still testing out the change myself in my own home setup here to see > if I run into any bad edge cases along the way. > > I can always try to explain the different code parts as I am indeed > modifying the core parts of the read and write operations for tun and > tcp so it's a big change to make to the code base. Basically this > change is important to me in particular because of my setup and > requirements in specific. I have WiFi LAN clients which all assume a > 1500 byte MTU on their side and I have a router WAN client which > enforces a 1500 byte MTU on the internet's side. In the middle of my > core network is a VPN box and almost every VPN software will operate > in UDP mode with a sub-1500 MTU in the middle of this network > pipeline. This is not a good design to have in general as I don't want > to waste cycles fragmenting and/or compressing the data into smaller > sized UDP packets. With the code change I am presenting, I am able to > specify a true 1500 byte VPN MTU interface with the exact matching > 1500 byte read calls to the TUN interface itself (the code base had to > be modified to allow for this because it was adding onto the > payload_size used in the read call which I didn't want as I am > operating on exact multiples of 1500 bytes in specific). OpenVPN already supports arbitrary MTU sizes just fine. The default is even MTU 1500. mssfix is also enabled by default to avoid fragmentation, which especially for UDP is bad. So I am not sure what you are modifying here because what you are describing is already well supported by OpenVPN. > > With this change, my network pipeline is a true 1500 byte MTU which > matches all the way from the client side to the vpn link to the > internet side and to the server side (end to end to end to end). In > addition, I also added the ability to batch together multiple 1500 > byte read calls (specifically 6 x 1500 bytes into 9000 bytes) into one > single encrypt sign call and one single TCP write call. So this needs a different wire format and some extra headers/framing as you need to split this jumbo packet again into 1500 bytes packets on the receiver. In order to justify introducing a new framing/packet format, a clear benefit should be shown. It would be for example to show what the performance benefit here actually is, e.g. by doing a simple test. do a simple client server OpenVPN connection and run iperf between client and server - without VPN - unmodified OpenVPN with --mtu 1500 - unmodified OpenVPN with --mtu 9000 - your approach That would give an indication what kind of gains we are talking here about. > This allows the encryption method to operate only once on a much > larger payload size as well as allow the linux kernel to efficiently > transfer the data with order and delivery guaranteed as fast as > possible. The code base had to be modified to allow for all of this as > well as it was preventing me from performing this much larger sized > ssl sign and encrypt + tcp read and write (the code base assumes you > are operating on only 1 tun read call worth of data at a time > everywhere). > > This is exactly why I prefer using TCP to tunnel encrypted network > data as my solution provided can properly set a full sized 1500 byte > MTU as well as perform an exact matching read call of 1500 bytes to > get the full amount of data from the interface and then bulk it > together to efficiently encrypt it and then use the magic of TCP to > transfer that data all at once as quickly as possible without any need > for fragmentation or compression. I don't think any other VPN product > on the market offers that kind of functionality as far as I am aware > as most other VPN products use a smaller sized MTU as well as the > packet size limitations of UDP. I believe that this could be a > distinguishing feature for OpenVPN as well as automatically solve some > of the issues that folks run into when inserting a VPN appliance into > the middle of their network setups. In TCP mode can you get away with a lot of thing you are doing. In UDP mode, having tunnel inside MTU of 1500 or pushing large fragmented IP/UDP packets will be very detrimental to VPN performance. > > I've been running this change on my own setup to at least make sure it > works and it seems to be running pretty nicely so far. I haven't > experienced any fragmentation or performance issues as any sized data > that comes off the clients LAN side is fully taken care of now through > the VPN side and onto the WAN and server side. This whole MTU fragment and tun must have 1500 MTU to not see these problems is not a thing that I observe and also not a logical conclusion for me. > > If this is something you are not interested in I can understand that, > I can stop posting here and the most I can do is at least submit a > pull request in case anyone in the future is indeed interested in such > work. It'd be nice to contribute to a good quality open source project > that I have used for many many years and something which may help > solve other community member's issues with regards to the small sized > MTU + UDP problem which does exist in practice and really hampers > connections along the way in a network design. It would be good to understand what you are actually referring to here. Small sized MTU + UDP problem is quite vague as there are multiple things that can cause probelems with an UDP VPN. > > I also don't mind explaining my code parts if you actually want, I > just need to take time to write them out and describe what they are > doing and why. As you can see, I am trying to achieve a very specific > and exact design goal that the code base wasn't originally allowing > for, so I had to make some modifications to be able to accomplish it. >
diff --git a/src/openvpn/forward.c b/src/openvpn/forward.c index 75ca9d5c..d9a98607 100644 --- a/src/openvpn/forward.c +++ b/src/openvpn/forward.c @@ -46,6 +46,9 @@ #include "mstats.h" +#include <sys/select.h> +#include <sys/time.h> + counter_type link_read_bytes_global; /* GLOBAL */ counter_type link_write_bytes_global; /* GLOBAL */ @@ -78,6 +81,32 @@ show_wait_status(struct context *c) #endif /* ifdef ENABLE_DEBUG */ +bool check_bulk_mode(struct context *c) +{ + if ((c->c2.frame.bulk_size > 0) && (c->c1.tuntap != NULL) && (c->c2.buffers != NULL)) + { + return true; + } + return false; +} + +void xfer_io(struct context *c, struct context *b) +{ + int plen = 0; + if (check_bulk_mode(b)) + { + int leng = (b->c2.buffers->bufs_indx + 1); + for (int x = 0; x < leng; ++x) + { + plen = BLEN(&b->c2.bufs[x]); + if (plen < 1) { c->c2.bufs[x].len = 0; } + else { c->c2.bufs[x] = b->c2.bufs[x]; } + } + c->c2.buffers->bufs_indx = b->c2.buffers->bufs_indx; + b->c2.buffers->bufs_indx = -1; + } +} + static void check_tls_errors_co(struct context *c) { @@ -605,6 +634,21 @@ buffer_turnover(const uint8_t *orig_buf, struct buffer *dest_stub, struct buffer } } +uint8_t *buff_prepsize(uint8_t *buff, int *size) +{ + buff[0] = ((*size >> 8) & 0xff); + buff[1] = ((*size >> 0) & 0xff); + buff += 2; + return buff; +} + +uint8_t *buff_postsize(uint8_t *buff, int *size) +{ + *size = ((buff[0] << 8) + (buff[1] << 0)); + buff += 2; + return buff; +} + /* * Compress, fragment, encrypt and HMAC-sign an outgoing packet. * Input: c->c2.buf @@ -1031,6 +1075,7 @@ process_incoming_link_part1(struct context *c, struct link_socket_info *lsi, boo fprintf(stderr, "R"); } #endif + msg(D_LINK_RW, "%s READ [%d] from %s: %s", proto2ascii(lsi->proto, lsi->af, true), BLEN(&c->c2.buf), print_link_socket_actual(&c->c2.from, &gc), PROTO_DUMP(&c->c2.buf, &gc)); @@ -1211,6 +1256,23 @@ process_incoming_link_part2(struct context *c, struct link_socket_info *lsi, } } +void process_incoming_link_part3(struct context *c) +{ + int leng = BLEN(&c->c2.to_tun); + if (leng > 0) + { + if (check_bulk_mode(c)) + { + c->c2.buffers->send_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->send_tun_max.len = leng; + bcopy(BPTR(&c->c2.to_tun), BPTR(&c->c2.buffers->send_tun_max), leng); + //dmsg(M_INFO, "FWD BAT LINK 0 [%d] [%d] [%d] [%d] [%d]", BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->read_link_buf), BLEN(&c->c2.buffers->send_tun_max)); + c->c2.to_tun.offset += 2; + c->c2.buf.offset += 2; + } + } +} + static void process_incoming_link(struct context *c, struct link_socket *sock) { @@ -1221,6 +1283,7 @@ process_incoming_link(struct context *c, struct link_socket *sock) process_incoming_link_part1(c, lsi, false); process_incoming_link_part2(c, lsi, orig_buf); + process_incoming_link_part3(c); perf_pop(); } @@ -1321,7 +1384,7 @@ process_incoming_dco(struct context *c) */ void -read_incoming_tun(struct context *c) +read_incoming_tun_part2(struct context *c) { /* * Setup for read() call on TUN/TAP device. @@ -1382,6 +1445,55 @@ read_incoming_tun(struct context *c) perf_pop(); } +void read_incoming_tun_part3(struct context *c) +{ + fd_set rfds; + struct timeval timo; + if (check_bulk_mode(c)) + { + int plen = 0; + int fdno = c->c1.tuntap->fd; + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + int leng = plen; + int indx = (c->c2.buffers->bufs_indx + 1); + if (indx >= TUN_BAT_MIN) { break; } + if (leng < 1) + { + FD_ZERO(&rfds); + FD_SET(fdno, &rfds); + timo.tv_sec = 0; + timo.tv_usec = 0; + select(fdno+1, &rfds, NULL, NULL, &timo); + if (FD_ISSET(fdno, &rfds)) + { + read_incoming_tun_part2(c); + plen = BLEN(&c->c2.buf); + } else { break; } + } + //dmsg(M_INFO, "FWD BAT READ 0 [%d] [%d] [%d] [%d] [%d]", c->c2.buffers->bufs_indx + 1, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); + leng = plen; + if (leng > 0) + { + c->c2.buffers->read_tun_bufs[indx].offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_bufs[indx].len = leng; + bcopy(BPTR(&c->c2.buf), BPTR(&c->c2.buffers->read_tun_bufs[indx]), leng); + c->c2.bufs[indx] = c->c2.buffers->read_tun_bufs[indx]; + c->c2.buffers->bufs_indx = indx; + } else { break; } + plen = 0; + } + } +} + +void read_incoming_tun(struct context *c) +{ + if (c->c2.frame.bulk_size <= 0) { + read_incoming_tun_part2(c); + } + read_incoming_tun_part3(c); +} + /** * Drops UDP packets which OS decided to route via tun. * @@ -1469,7 +1581,7 @@ drop_if_recursive_routing(struct context *c, struct buffer *buf) */ void -process_incoming_tun(struct context *c, struct link_socket *out_sock) +process_incoming_tun_part2(struct context *c, struct link_socket *out_sock) { struct gc_arena gc = gc_new(); @@ -1488,7 +1600,7 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) #endif /* Show packet content */ - dmsg(D_TUN_RW, "TUN READ [%d]", BLEN(&c->c2.buf)); + dmsg(D_TUN_RW, "TUN READ [%d] [%d]", BLEN(&c->c2.buf), c->c2.frame.buf.payload_size); if (c->c2.buf.len > 0) { @@ -1512,7 +1624,9 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) } if (c->c2.buf.len > 0) { + if ((c->c2.buffers == NULL) || (c->c2.buffers->flag_ciph != -2)) { encrypt_sign(c, true); + } } else { @@ -1522,6 +1636,60 @@ process_incoming_tun(struct context *c, struct link_socket *out_sock) gc_free(&gc); } +void process_incoming_tun_part3(struct context *c, struct link_socket *out_sock) +{ + if (check_bulk_mode(c)) + { + c->c2.buffers->flag_ciph = -2; + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_max.len = 0; + uint8_t *temp = BPTR(&c->c2.buffers->read_tun_max); + int plen = 0, fdno = c->c1.tuntap->fd; + int maxl = 0, leng = (c->c2.buffers->bufs_indx + 1); + if ((fdno > 0) && (leng > 0)) + { + for (int x = 0; x < leng; ++x) + { + c->c2.buf = c->c2.bufs[x]; + //dmsg(M_INFO, "FWD BAT INPT 0 [%d] [%d] [%d] [%d] [%d]", x, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.bufs[x])); + process_incoming_tun_part2(c, out_sock); + if (BLEN(&c->c2.buf) < 1) + { + c->c2.bufs[x].len = 0; + } + } + for (int x = 0; x < leng; ++x) + { + plen = c->c2.bufs[x].len; + if (plen > 0) + { + temp = buff_prepsize(temp, &plen); + bcopy(BPTR(&c->c2.bufs[x]), temp, plen); + temp += plen; maxl += (plen + 2); + } + } + if (maxl > 0) + { + c->c2.buffers->read_tun_max.offset = TUN_BAT_OFF; + c->c2.buffers->read_tun_max.len = maxl; + c->c2.buf = c->c2.buffers->read_tun_max; + //dmsg(M_INFO, "FWD BAT INPT 1 [%d] [%d] [%d] [%d] [%d]", maxl, fdno, BLEN(&c->c2.buf), BLEN(&c->c2.buffers->read_tun_buf), BLEN(&c->c2.buffers->read_tun_max)); + encrypt_sign(c, true); + } + } + c->c2.buffers->bufs_indx = -1; + c->c2.buffers->flag_ciph = -1; + } +} + +void process_incoming_tun(struct context *c, struct link_socket *out_sock) +{ + if (c->c2.frame.bulk_size <= 0) { + process_incoming_tun_part2(c, out_sock); + } + process_incoming_tun_part3(c, out_sock); +} + /** * Forges a IPv6 ICMP packet with a no route to host error code from the * IPv6 packet in buf and sends it directly back to the client via the tun @@ -1748,7 +1916,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) perf_push(PERF_PROC_OUT_LINK); - if (c->c2.to_link.len > 0 && c->c2.to_link.len <= c->c2.frame.buf.payload_size) + if (c->c2.to_link.len > 0 && (c->c2.to_link.len <= c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0)) { /* * Setup for call to send/sendto which will send @@ -1793,6 +1961,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) fprintf(stderr, "W"); } #endif + msg(D_LINK_RW, "%s WRITE [%d] to %s: %s", proto2ascii(sock->info.proto, sock->info.af, true), BLEN(&c->c2.to_link), print_link_socket_actual(c->c2.to_link_addr, &gc), PROTO_DUMP(&c->c2.to_link, &gc)); @@ -1892,7 +2061,7 @@ process_outgoing_link(struct context *c, struct link_socket *sock) */ void -process_outgoing_tun(struct context *c, struct link_socket *in_sock) +process_outgoing_tun_part2(struct context *c, struct link_socket *in_sock) { /* * Set up for write() call to TUN/TAP @@ -1912,7 +2081,7 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) process_ip_header(c, PIP_MSSFIX | PIPV4_EXTRACT_DHCP_ROUTER | PIPV4_CLIENT_NAT | PIP_OUTGOING, &c->c2.to_tun, in_sock); - if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size) + if (c->c2.to_tun.len <= c->c2.frame.buf.payload_size || c->c2.frame.bulk_size > 0) { /* * Write to TUN/TAP device. @@ -1925,7 +2094,8 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) fprintf(stderr, "w"); } #endif - dmsg(D_TUN_RW, "TUN WRITE [%d]", BLEN(&c->c2.to_tun)); + + dmsg(D_TUN_RW, "TUN WRITE [%d] [%d]", BLEN(&c->c2.to_tun), c->c2.frame.buf.payload_size); #ifdef PACKET_TRUNCATION_CHECK ipv4_packet_size_verify(BPTR(&c->c2.to_tun), BLEN(&c->c2.to_tun), TUNNEL_TYPE(c->c1.tuntap), @@ -1981,6 +2151,39 @@ process_outgoing_tun(struct context *c, struct link_socket *in_sock) perf_pop(); } +void process_outgoing_tun_part3(struct context *c, struct link_socket *in_sock) +{ + if (check_bulk_mode(c)) + { + int maxl = 0, plen = 0; + int leng = BLEN(&c->c2.buffers->send_tun_max); + uint8_t *temp = BPTR(&c->c2.buffers->send_tun_max); + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + temp = buff_postsize(temp, &plen); + if ((leng > 0) && (plen > 0) && ((maxl + plen) < leng)) + { + c->c2.to_tun = c->c2.buffers->to_tun_max; + c->c2.to_tun.offset = TUN_BAT_OFF; + c->c2.to_tun.len = plen; + bcopy(temp, BPTR(&c->c2.to_tun), plen); + temp += plen; maxl += (plen + 2); + //dmsg(M_INFO, "FWD BAT OUTP 1 [%d] [%d] [%d] [%d]", x, BLEN(&c->c2.buf), BLEN(&c->c2.to_tun), BLEN(&c->c2.buffers->read_link_buf)); + process_outgoing_tun_part2(c, in_sock); + } else { break; } + } + buf_reset(&c->c2.to_tun); + } +} + +void process_outgoing_tun(struct context *c, struct link_socket *in_sock) +{ + if (c->c2.frame.bulk_size <= 0) { + process_outgoing_tun_part2(c, in_sock); + } + process_outgoing_tun_part3(c, in_sock); +} + void pre_select(struct context *c) { diff --git a/src/openvpn/forward.h b/src/openvpn/forward.h index d5641491..9fda1583 100644 --- a/src/openvpn/forward.h +++ b/src/openvpn/forward.h @@ -79,6 +79,8 @@ void pre_select(struct context *c); void process_io(struct context *c, struct link_socket *sock); +void xfer_io(struct context *c, struct context *b); + /**********************************************************************/ /** @@ -196,6 +198,8 @@ bool process_incoming_link_part1(struct context *c, struct link_socket_info *lsi void process_incoming_link_part2(struct context *c, struct link_socket_info *lsi, const uint8_t *orig_buf); +void process_incoming_link_part3(struct context *c); + /** * Transfers \c float_sa data extracted from an incoming DCO * PEER_FLOAT_NTF to \c out_osaddr for later processing. diff --git a/src/openvpn/init.c b/src/openvpn/init.c index 40ae2c8c..0849dfce 100644 --- a/src/openvpn/init.c +++ b/src/openvpn/init.c @@ -2971,6 +2971,10 @@ frame_finalize_options(struct context *c, const struct options *o) tailroom += COMP_EXTRA_BUFFER(payload_size); #endif + if (frame->bulk_size > 0) { + payload_size = frame->tun_mtu; + } + frame->buf.payload_size = payload_size; frame->buf.headroom = headroom; frame->buf.tailroom = tailroom; @@ -3473,6 +3477,9 @@ do_init_frame_tls(struct context *c) if (c->c2.tls_multi) { tls_multi_init_finalize(c->c2.tls_multi, c->options.ce.tls_mtu); + if (c->c2.frame.bulk_size > 0) { + c->c2.tls_multi->opt.frame.buf.payload_size = c->c2.frame.tun_mtu; + } ASSERT(c->c2.tls_multi->opt.frame.buf.payload_size <= c->c2.frame.buf.payload_size); frame_print(&c->c2.tls_multi->opt.frame, D_MTU_INFO, "Control Channel MTU parms"); @@ -3536,6 +3543,14 @@ do_init_frame(struct context *c) c->c2.frame.extra_tun += c->options.ce.tun_mtu_extra; } + /* + * Adjust bulk size based on the --bulk-mode parameter. + */ + if (c->options.ce.bulk_mode) + { + c->c2.frame.bulk_size = c->options.ce.tun_mtu; + } + /* * Fill in the blanks in the frame parameters structure, * make sure values are rational, etc. @@ -3676,9 +3691,40 @@ init_context_buffers(const struct frame *frame) size_t buf_size = BUF_SIZE(frame); + if (frame->bulk_size > 0) { + buf_size = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, frame->buf.headroom + frame->buf.tailroom); + } + + dmsg(M_INFO, "MEM NEW [%ld] [%d+%d+%d]", buf_size, frame->buf.headroom, frame->buf.payload_size, frame->buf.tailroom); + b->read_link_buf = alloc_buf(buf_size); b->read_tun_buf = alloc_buf(buf_size); + if (frame->bulk_size > 0) { + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + size_t part_size = BUF_SIZE(frame); + b->read_tun_bufs[x] = alloc_buf(part_size); + b->read_tun_bufs[x].offset = TUN_BAT_OFF; + b->read_tun_bufs[x].len = 0; + } + + b->read_tun_max = alloc_buf(buf_size); + b->read_tun_max.offset = TUN_BAT_OFF; + b->read_tun_max.len = 0; + + b->send_tun_max = alloc_buf(buf_size); + b->send_tun_max.offset = TUN_BAT_OFF; + b->send_tun_max.len = 0; + + b->to_tun_max = alloc_buf(buf_size); + b->to_tun_max.offset = TUN_BAT_OFF; + b->to_tun_max.len = 0; + } + + b->bufs_indx = -1; + b->flag_ciph = -1; + b->aux_buf = alloc_buf(buf_size); b->encrypt_buf = alloc_buf(buf_size); @@ -3701,6 +3747,16 @@ free_context_buffers(struct context_buffers *b) free_buf(&b->read_tun_buf); free_buf(&b->aux_buf); + if (b->to_tun_max.data) { + free_buf(&b->to_tun_max); + free_buf(&b->send_tun_max); + free_buf(&b->read_tun_max); + for (int x = 0; x < TUN_BAT_MAX; ++x) + { + free_buf(&b->read_tun_bufs[x]); + } + } + #ifdef USE_COMP free_buf(&b->compress_buf); free_buf(&b->decompress_buf); diff --git a/src/openvpn/mtu.c b/src/openvpn/mtu.c index a419e32d..7e35c837 100644 --- a/src/openvpn/mtu.c +++ b/src/openvpn/mtu.c @@ -41,9 +41,15 @@ void alloc_buf_sock_tun(struct buffer *buf, const struct frame *frame) { /* allocate buffer for overlapped I/O */ - *buf = alloc_buf(BUF_SIZE(frame)); + size_t alen = BUF_SIZE(frame); + size_t blen = frame->buf.payload_size; + if (frame->bulk_size > 0) { + alen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_OFF); + blen = BAT_SIZE(TUN_BAT_MAX, frame->tun_mtu, TUN_BAT_NOP); + } + *buf = alloc_buf(alen); ASSERT(buf_init(buf, frame->buf.headroom)); - buf->len = frame->buf.payload_size; + buf->len = blen; ASSERT(buf_safe(buf, 0)); } diff --git a/src/openvpn/mtu.h b/src/openvpn/mtu.h index 925ef0bf..eb799fb3 100644 --- a/src/openvpn/mtu.h +++ b/src/openvpn/mtu.h @@ -58,6 +58,14 @@ */ #define TUN_MTU_MIN 100 +/* + * Bulk mode static define values. + */ +#define TUN_BAT_MIN 6 +#define TUN_BAT_MAX 9 +#define TUN_BAT_OFF 256 +#define TUN_BAT_NOP 0 + /* * Default MTU of network over which tunnel data will pass by TCP/UDP. */ @@ -152,6 +160,10 @@ struct frame * which defaults to 0 for tun and 32 * (\c TAP_MTU_EXTRA_DEFAULT) for tap. * */ + + int bulk_size; /**< Signal to the init frame function + * to allow for bulk mode TCP transfers. + * */ }; /* Forward declarations, to prevent includes */ @@ -171,6 +183,7 @@ struct options; * larger than the headroom. */ #define BUF_SIZE(f) ((f)->buf.headroom + (f)->buf.payload_size + (f)->buf.tailroom) +#define BAT_SIZE(a, b, c) ((a * b) + c) /* * Function prototypes. diff --git a/src/openvpn/multi.c b/src/openvpn/multi.c index e1ce32ab..9e089703 100644 --- a/src/openvpn/multi.c +++ b/src/openvpn/multi.c @@ -3414,6 +3414,7 @@ multi_process_incoming_link(struct multi_context *m, struct multi_instance *inst } process_incoming_link_part2(c, lsi, orig_buf); + process_incoming_link_part3(c); } perf_pop(); @@ -3558,9 +3559,7 @@ multi_process_incoming_tun(struct multi_context *m, const unsigned int mpp_flags const int dev_type = TUNNEL_TYPE(m->top.c1.tuntap); int16_t vid = 0; -#ifdef MULTI_DEBUG_EVENT_LOOP - printf("TUN -> TCP/UDP [%d]\n", BLEN(&m->top.c2.buf)); -#endif + msg(D_MULTI_DEBUG, "TUN -> TCP/UDP [%d]", BLEN(&m->top.c2.buf)); if (m->pending) { @@ -3610,6 +3609,8 @@ multi_process_incoming_tun(struct multi_context *m, const unsigned int mpp_flags { /* transfer packet pointer from top-level context buffer to instance */ c->c2.buf = m->top.c2.buf; + /* todo determine if to call this (multi_process_incoming_tun) for each bulk item read? */ + xfer_io(c, &m->top); } else { diff --git a/src/openvpn/openvpn.h b/src/openvpn/openvpn.h index cd99cd40..21fa8967 100644 --- a/src/openvpn/openvpn.h +++ b/src/openvpn/openvpn.h @@ -112,6 +112,14 @@ struct context_buffers */ struct buffer read_link_buf; struct buffer read_tun_buf; + + struct buffer read_tun_bufs[TUN_BAT_MAX]; + struct buffer read_tun_max; + struct buffer send_tun_max; + struct buffer to_tun_max; + + int bufs_indx; + int flag_ciph; }; /* @@ -376,6 +384,8 @@ struct context_2 struct buffer to_tun; struct buffer to_link; + struct buffer bufs[TUN_BAT_MAX]; + /* should we print R|W|r|w to console on packet transfers? */ bool log_rw; diff --git a/src/openvpn/options.c b/src/openvpn/options.c index c54032d8..041d17d0 100644 --- a/src/openvpn/options.c +++ b/src/openvpn/options.c @@ -304,6 +304,7 @@ static const char usage_message[] = " 'maybe' -- Use per-route hints\n" " 'yes' -- Always DF (Don't Fragment)\n" "--mtu-test : Empirically measure and report MTU.\n" + "--bulk-mode : Use bulk TUN/TCP reads/writes.\n" #ifdef ENABLE_FRAGMENT "--fragment max : Enable internal datagram fragmentation so that no UDP\n" " datagrams are sent which are larger than max bytes.\n" @@ -3005,6 +3006,9 @@ options_postprocess_mutate_ce(struct options *o, struct connection_entry *ce) ce->tun_mtu_extra_defined = true; ce->tun_mtu_extra = TAP_MTU_EXTRA_DEFAULT; } + if (ce->proto != PROTO_TCP && ce->proto != PROTO_TCP_SERVER && ce->proto != PROTO_TCP_CLIENT) { + ce->bulk_mode = false; + } } /* @@ -9926,6 +9930,10 @@ add_option(struct options *options, char *p[], bool is_inline, const char *file, goto err; } } + else if (streq(p[0], "bulk-mode")) + { + options->ce.bulk_mode = true; + } else { int i; diff --git a/src/openvpn/options.h b/src/openvpn/options.h index 38e67c8d..d1b0586d 100644 --- a/src/openvpn/options.h +++ b/src/openvpn/options.h @@ -174,6 +174,9 @@ struct connection_entry /* Allow only client that support resending the wrapped client key */ bool tls_crypt_v2_force_cookie; + + /* Bulk mode allows for multiple tun reads + larger tcp writes */ + bool bulk_mode; }; struct remote_entry
Thanks to Gert's help on this, I was able to finally configure and compile and run and test the bulk mode changes against the latest git source code to ensure everything still works correctly. I also fixed up some other issues like properly freeing the extra buffer allocations and removing the unneeded batched data prefixes and converting a remaining while loop to a max limited for loop and properly resetting the outgoing tun buffer pointer at the end of the write method when finished. Thanks, Jon C Example updated pull request: https://github.com/OpenVPN/openvpn/pull/814/files git formatted diff patch: From 985e88a9af26a39554f113f37ee18032a2f41c3e Mon Sep 17 00:00:00 2001 From: Jon Chiappetta <root@fossjon.com> Date: Wed, 6 Aug 2025 16:33:18 -0400 Subject: [PATCH] bulk mode --- src/openvpn/forward.c | 217 ++++++++++++++++++++++++++++++++++++++++-- src/openvpn/forward.h | 4 + src/openvpn/init.c | 56 +++++++++++ src/openvpn/mtu.c | 10 +- src/openvpn/mtu.h | 13 +++ src/openvpn/multi.c | 7 +- src/openvpn/openvpn.h | 10 ++ src/openvpn/options.c | 8 ++ src/openvpn/options.h | 3 + 9 files changed, 316 insertions(+), 12 deletions(-) -- 2.39.5 (Apple Git-154)