[Openvpn-devel,v2] tests: disable automake serial_tests

Message ID 20231214111635.237429-1-frank@lichtenheld.com
State Accepted
Headers show
Series [Openvpn-devel,v2] tests: disable automake serial_tests | expand

Commit Message

Frank Lichtenheld Dec. 14, 2023, 11:16 a.m. UTC
Serial mode is the old one and offers much less options for
running the tests. Generally our tests seem to work fine
with the newer parallel mode. The only reason we stuck with
serial_tests seems to be that we didn't like that it doesn't
output the test output by default. We could fix that with a
custom test driver. But will put that into a separate commit.

Change-Id: Ic7265d89142637b0963a6847c6beb06d9163bbb1
Signed-off-by: Frank Lichtenheld <frank@lichtenheld.com>
Acked-by: Arne Schwabe <arne-openvpn@rfc2549.org>
---

This change was reviewed on Gerrit and approved by at least one
developer. I request to merge it to master.

Gerrit URL: https://gerrit.openvpn.net/c/openvpn/+/477
This mail reflects revision 2 of this Change.
Acked-by according to Gerrit (reflected above):
Arne Schwabe <arne-openvpn@rfc2549.org>

Comments

Gert Doering Dec. 18, 2023, 6:10 p.m. UTC | #1
Simplification of the autoconf logic is definitely welcome...  tested
on FreeBSD 13.2 with autoconf 2.71 / automake 1.16, and it seems to
nicely run all the tests, with a nice and short summary

PASS: argv_testdriver
PASS: buffer_testdriver
PASS: crypto_testdriver
PASS: packet_id_testdriver
PASS: auth_token_testdriver
PASS: ncp_testdriver
PASS: misc_testdriver
PASS: pkt_testdriver
PASS: tls_crypt_testdriver
PASS: provider_testdriver
============================================================================
Testsuite summary for OpenVPN 2.7_git Unit-Tests
============================================================================
# TOTAL: 10
# PASS:  10

(for the auth_pam_testdriver, the output now explodes from "one test
run" to "and now a 10-line summary", but well...)


I look forward to the promised improvements for the test driver, though
- after

make  check-TESTS
SKIP: t_client.sh
PASS: t_lpback.sh

.. the thing just sits there for 2 minutes, with no indication what is
happening and why it's not finished yet (did it crash?) and then it
finally reaches

PASS: t_cltsrv.sh
============================================================================
Testsuite summary for OpenVPN 2.7_git System Tests
============================================================================
# TOTAL: 3
# PASS:  2
# SKIP:  1

.. and "make check V=99" does not improve things here, so this definitely
needs work.  Much worse if t_client.sh tests actually run, because then it
just stops at "make check-TESTS" for a looong time...


Your patch has been applied to the master branch.

commit 55d73959cc98241d0ef10dbd091ecf0061b175fe
Author: Frank Lichtenheld
Date:   Thu Dec 14 12:16:35 2023 +0100

     tests: disable automake serial_tests

     Signed-off-by: Frank Lichtenheld <frank@lichtenheld.com>
     Acked-by: Arne Schwabe <arne-openvpn@rfc2549.org>
     Message-Id: <20231214111635.237429-1-frank@lichtenheld.com>
     URL: https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg27812.html
     Signed-off-by: Gert Doering <gert@greenie.muc.de>


--
kind regards,

Gert Doering

Patch

diff --git a/configure.ac b/configure.ac
index 54f79ab..2823f04 100644
--- a/configure.ac
+++ b/configure.ac
@@ -41,20 +41,6 @@ 
 AC_CONFIG_SRCDIR([src/openvpn/syshead.h])
 AC_CONFIG_MACRO_DIR([m4])
 
-dnl Initialize automake.  automake < 1.12 didn't have serial-tests and
-dnl gives an error if it sees this, but for automake >= 1.13
-dnl serial-tests is required so we have to include it.  Solution is to
-dnl test for the version of automake (by running an external command)
-dnl and provide it if necessary.  Note we have to do this entirely using
-dnl m4 macros since automake queries this macro by running
-dnl 'autoconf --trace ...'.
-m4_define([serial_tests], [
-    m4_esyscmd([automake --version |
-                head -1 |
-                awk '{split ($NF,a,"."); if (a[1] == 1 && a[2] >= 12) { print "serial-tests" }}'
-    ])
-])
-
 dnl Automake 1.14+ warns if sources are in sub-directories but subdir-objects
 dnl options is not enabled. However, automake before 1.15a has a bug that causes
 dnl variable expansion to fail in foo_SOURCES when this option is used.
@@ -69,7 +55,7 @@ 
 
 # This foreign option prevents autoreconf from overriding our COPYING and
 # INSTALL targets:
-AM_INIT_AUTOMAKE(foreign serial_tests subdir_objects 1.9) dnl NB: Do not [quote] this parameter.
+AM_INIT_AUTOMAKE(foreign subdir_objects 1.9) dnl NB: Do not [quote] this parameter.
 AC_CANONICAL_HOST
 AC_USE_SYSTEM_EXTENSIONS
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 80673d5..6c71067 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -14,6 +14,8 @@ 
 
 SUBDIRS = unit_tests
 
+AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING) System Tests'
+
 if !WIN32
 test_scripts = t_client.sh t_lpback.sh t_cltsrv.sh
 if HAVE_SITNL
diff --git a/tests/unit_tests/example_test/Makefile.am b/tests/unit_tests/example_test/Makefile.am
index 24eb0ba..1ae80c8 100644
--- a/tests/unit_tests/example_test/Makefile.am
+++ b/tests/unit_tests/example_test/Makefile.am
@@ -1,5 +1,7 @@ 
 AUTOMAKE_OPTIONS = foreign
 
+AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING) example Unit-Tests'
+
 check_PROGRAMS = example_testdriver example2_testdriver
 
 if !CROSS_COMPILING
diff --git a/tests/unit_tests/openvpn/Makefile.am b/tests/unit_tests/openvpn/Makefile.am
index ef45b11..cecf4dc 100644
--- a/tests/unit_tests/openvpn/Makefile.am
+++ b/tests/unit_tests/openvpn/Makefile.am
@@ -1,5 +1,7 @@ 
 AUTOMAKE_OPTIONS = foreign
 
+AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING) Unit-Tests'
+
 test_binaries=
 
 if HAVE_LD_WRAP_SUPPORT
diff --git a/tests/unit_tests/plugins/auth-pam/Makefile.am b/tests/unit_tests/plugins/auth-pam/Makefile.am
index de5e96e..ba32013 100644
--- a/tests/unit_tests/plugins/auth-pam/Makefile.am
+++ b/tests/unit_tests/plugins/auth-pam/Makefile.am
@@ -1,5 +1,7 @@ 
 AUTOMAKE_OPTIONS = foreign
 
+AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING) auth_pam Plugin Unit-Tests'
+
 if ENABLE_PLUGIN_AUTH_PAM
 check_PROGRAMS = auth_pam_testdriver
 TESTS = $(check_PROGRAMS)