[PATCH v2] test: pytest: introduce pytest for network, test tftp notfound

[PATCH v2] test: pytest: introduce pytest for network, test tftp notfound

[Jonas Rebmann]

barebox already has a few pytest integration tests but none of them are
networking related.

This patch introduces a test that asserts on bringing up at least one
DHCP interface and, on any of these interfaces, a successful ICMP-ping
from the DUT to the test pytest-host as well as a basic "File not Found"
conversation between the DUT and the test pytest-host.

As the test requires a network connection to the barebox device, without
the "network" feature the test is marked xfail. For the case of a qemu
device we also assume DHCP to set serverip correctly. In any other case,
it is set via console to that IP of the pytest-host that has the route
to the barebox devices IP.

Signed-off-by: Jonas Rebmann <mail@schlaraffenlan.de>
---
Changes in v2:
- use random ephermal instead of hardcoded port for tftp test
- test any interface doing DHCP, expect one to test without issues
- fail on any tftp error code
- cleanup
- update commit message
- Link to v1: https://lore.barebox.org/barebox/20240605-test_tftp-v1-1-5752ea677b06@schlaraffenlan.de
---
 test/arm/virt@multi_v8_defconfig.yaml |   1 +
 test/openrisc/generic_defconfig.yaml  |   2 +
 test/py/test_network.py               | 106 ++++++++++++++++++++++++++++++++++
 3 files changed, 109 insertions(+)

diff --git a/test/arm/virt@multi_v8_defconfig.yaml b/test/arm/virt@multi_v8_defconfig.yaml
index d8f8ab5cbff35b5f9c414b0c7a66974a9a723794..42ce10328d7cc903010199364f5464d5146239e0 100644
--- a/test/arm/virt@multi_v8_defconfig.yaml
+++ b/test/arm/virt@multi_v8_defconfig.yaml
@@ -14,6 +14,7 @@ targets:
       BareboxTestStrategy: {}
     features:
       - virtio-mmio
+      - network
     runner:
       tuxmake_arch: arm64
 images:
diff --git a/test/openrisc/generic_defconfig.yaml b/test/openrisc/generic_defconfig.yaml
index 93ba9586c439d597c1a5e8deca44d1b99c902bed..56b70b8242f0e7f50f7ac732525a4857d8541744 100644
--- a/test/openrisc/generic_defconfig.yaml
+++ b/test/openrisc/generic_defconfig.yaml
@@ -12,6 +12,8 @@ targets:
         prompt: 'barebox@[^:]+:[^ ]+ '
         bootstring: 'commandline:'
       BareboxTestStrategy: {}
+    features:
+      - network
 images:
   barebox: !template "$LG_BUILDDIR/barebox"
 imports:
diff --git a/test/py/test_network.py b/test/py/test_network.py
new file mode 100644
index 0000000000000000000000000000000000000000..afaa259e2496740fabe61129e8e7d4190c5d4c50
--- /dev/null
+++ b/test/py/test_network.py
@@ -0,0 +1,106 @@
+import pytest
+
+from labgrid import driver,Environment
+import socket
+import threading
+import re
+import warnings
+
+# Setting the port to zero causes bind to choose a random ephermal port
+TFTP_TEST_PORT = 0
+
+def get_source_addr(destination_ip, destination_port):
+    udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    udp_socket.connect((destination_ip, destination_port))
+    source_ip, _ = udp_socket.getsockname()
+    return source_ip
+
+def tftp_setup_socket(listen_addr, listen_port=0):
+    udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    udp_socket.bind((listen_addr, listen_port))
+    udp_socket.settimeout(3)
+    return udp_socket
+
+
+def tftp_expect_notfound(filename, udp_socket):
+    uname = filename.encode("ascii")
+    messages = [
+        b"\000\001" + uname + b"\000octet\000timeout\0005\000blksize\000512\000tsize\0000\000",
+        b"\000\005\000\001File not found\000",
+    ]
+    data, addr = udp_socket.recvfrom(1024)
+    udp_socket.sendto(messages[1], addr)
+    udp_socket.close()
+    assert data == messages[0]
+
+def tftp_conversation(barebox, barebox_interface, guestaddr):
+
+    listen_addr = "127.0.0.1"
+    if not isinstance(barebox.console, driver.QEMUDriver):
+        # sending an arbitrary udp package to determine the IP towards DUT
+        listen_addr = get_source_addr(guestaddr, TFTP_TEST_PORT)
+        barebox.run_check(f"eth0.serverip={listen_addr}")
+
+    barebox.run_check("ping $eth0.serverip", timeout=2)
+
+    tftp_socket=tftp_setup_socket(listen_addr, TFTP_TEST_PORT)
+    tftp_port_used=tftp_socket.getsockname()[1]
+
+    tftp_thread = threading.Thread(
+        target=tftp_expect_notfound,
+        name="tftp_expect_notfound",
+        args=("a", tftp_socket),
+    )
+    tftp_thread.daemon = True
+    tftp_thread.start()
+
+    try:
+        stdout, _, returncode = barebox.run(f"tftp -P {tftp_port_used} a", timeout=3)
+        assert returncode != 0
+    finally:
+        # terminate a timed-out tftp
+        barebox.console.sendcontrol("c")
+        tftp_thread.join()
+        barebox.run_check("ifdown eth0")
+
+
+def test_barebox_network(barebox, env):
+    # on DUTs without network feature, this is expected to fail
+    # set xfail_strict=True to enforce specifying the network feature if available
+    if not 'network' in env.get_target_features():
+        pytest.xfail("network feature not specified")
+
+    stdout = barebox.run_check("ifup -a")
+
+    interfaces = []
+
+    for line in stdout:
+        # eth0: DHCP client bound to address 10.0.2.15
+        m = re.search('(eth[0-9]+): DHCP client bound to address (.*)', line)
+        if m is None:
+            continue
+        else:
+            interfaces.append((m.group(1), m.group(2)))
+
+    assert interfaces, "Network testing is only possible with at least one DHCP interface"
+
+    for iface in interfaces:
+        ifname = iface[0]
+        ifaddr = iface[1]
+
+        assert ifaddr != "0.0.0.0"
+        assert ifaddr == barebox.run_check(f"echo ${ifname}.ipaddr")[0]
+
+        # Attempt a conversation with the DUT, which needs to succeed only on one interface
+        success = False
+
+        try:
+            tftp_conversation(barebox, ifname, ifaddr)
+            success = True
+            break
+        except Exception as e:
+            warnings.warn(f"Could not connect to DUT on {ifname} ({ifaddr}): {e}")
+
+        if not success:
+            pytest.fail("Could not converse with DUT on any of the found DHCP interfaces!")
+

---
base-commit: 5a2866973d386ffe9b9f5c429a0cbcd58bcef69c
change-id: 20240605-test_tftp-9e12282d46a0

Best regards,
-- 
Jonas Rebmann <mail@schlaraffenlan.de>

Re: [PATCH v2] test: pytest: introduce pytest for network, test tftp notfound

[Ahmad Fatoum]

On 08.10.24 13:21, Jonas Rebmann wrote:
> barebox already has a few pytest integration tests but none of them are
> networking related.
> 
> This patch introduces a test that asserts on bringing up at least one
> DHCP interface and, on any of these interfaces, a successful ICMP-ping
> from the DUT to the test pytest-host as well as a basic "File not Found"
> conversation between the DUT and the test pytest-host.
> 
> As the test requires a network connection to the barebox device, without
> the "network" feature the test is marked xfail. For the case of a qemu
> device we also assume DHCP to set serverip correctly. In any other case,
> it is set via console to that IP of the pytest-host that has the route
> to the barebox devices IP.
> 
> Signed-off-by: Jonas Rebmann <mail@schlaraffenlan.de>

This test ran successfully against an i.MX8MP with a switch and only
eth2 having link up.

Tested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

> ---
> Changes in v2:
> - use random ephermal instead of hardcoded port for tftp test

I noticed an issue with this approach now: I had to disable my firewall,
because it's not known beforehand which port is going to be chosen...

Maybe there should also be a way to hardcode a user-supplied value,
but this shouldn't delay application of this patch.

Thanks!
Ahmad

> - test any interface doing DHCP, expect one to test without issues
> - fail on any tftp error code
> - cleanup
> - update commit message
> - Link to v1: https://lore.barebox.org/barebox/20240605-test_tftp-v1-1-5752ea677b06@schlaraffenlan.de
> ---
>  test/arm/virt@multi_v8_defconfig.yaml |   1 +
>  test/openrisc/generic_defconfig.yaml  |   2 +
>  test/py/test_network.py               | 106 ++++++++++++++++++++++++++++++++++
>  3 files changed, 109 insertions(+)
> 
> diff --git a/test/arm/virt@multi_v8_defconfig.yaml b/test/arm/virt@multi_v8_defconfig.yaml
> index d8f8ab5cbff35b5f9c414b0c7a66974a9a723794..42ce10328d7cc903010199364f5464d5146239e0 100644
> --- a/test/arm/virt@multi_v8_defconfig.yaml
> +++ b/test/arm/virt@multi_v8_defconfig.yaml
> @@ -14,6 +14,7 @@ targets:
>        BareboxTestStrategy: {}
>      features:
>        - virtio-mmio
> +      - network
>      runner:
>        tuxmake_arch: arm64
>  images:
> diff --git a/test/openrisc/generic_defconfig.yaml b/test/openrisc/generic_defconfig.yaml
> index 93ba9586c439d597c1a5e8deca44d1b99c902bed..56b70b8242f0e7f50f7ac732525a4857d8541744 100644
> --- a/test/openrisc/generic_defconfig.yaml
> +++ b/test/openrisc/generic_defconfig.yaml
> @@ -12,6 +12,8 @@ targets:
>          prompt: 'barebox@[^:]+:[^ ]+ '
>          bootstring: 'commandline:'
>        BareboxTestStrategy: {}
> +    features:
> +      - network
>  images:
>    barebox: !template "$LG_BUILDDIR/barebox"
>  imports:
> diff --git a/test/py/test_network.py b/test/py/test_network.py
> new file mode 100644
> index 0000000000000000000000000000000000000000..afaa259e2496740fabe61129e8e7d4190c5d4c50
> --- /dev/null
> +++ b/test/py/test_network.py
> @@ -0,0 +1,106 @@
> +import pytest
> +
> +from labgrid import driver,Environment
> +import socket
> +import threading
> +import re
> +import warnings
> +
> +# Setting the port to zero causes bind to choose a random ephermal port
> +TFTP_TEST_PORT = 0
> +
> +def get_source_addr(destination_ip, destination_port):
> +    udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> +    udp_socket.connect((destination_ip, destination_port))
> +    source_ip, _ = udp_socket.getsockname()
> +    return source_ip
> +
> +def tftp_setup_socket(listen_addr, listen_port=0):
> +    udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> +    udp_socket.bind((listen_addr, listen_port))
> +    udp_socket.settimeout(3)
> +    return udp_socket
> +
> +
> +def tftp_expect_notfound(filename, udp_socket):
> +    uname = filename.encode("ascii")
> +    messages = [
> +        b"\000\001" + uname + b"\000octet\000timeout\0005\000blksize\000512\000tsize\0000\000",
> +        b"\000\005\000\001File not found\000",
> +    ]
> +    data, addr = udp_socket.recvfrom(1024)
> +    udp_socket.sendto(messages[1], addr)
> +    udp_socket.close()
> +    assert data == messages[0]
> +
> +def tftp_conversation(barebox, barebox_interface, guestaddr):
> +
> +    listen_addr = "127.0.0.1"
> +    if not isinstance(barebox.console, driver.QEMUDriver):
> +        # sending an arbitrary udp package to determine the IP towards DUT
> +        listen_addr = get_source_addr(guestaddr, TFTP_TEST_PORT)
> +        barebox.run_check(f"eth0.serverip={listen_addr}")
> +
> +    barebox.run_check("ping $eth0.serverip", timeout=2)
> +
> +    tftp_socket=tftp_setup_socket(listen_addr, TFTP_TEST_PORT)
> +    tftp_port_used=tftp_socket.getsockname()[1]
> +
> +    tftp_thread = threading.Thread(
> +        target=tftp_expect_notfound,
> +        name="tftp_expect_notfound",
> +        args=("a", tftp_socket),
> +    )
> +    tftp_thread.daemon = True
> +    tftp_thread.start()
> +
> +    try:
> +        stdout, _, returncode = barebox.run(f"tftp -P {tftp_port_used} a", timeout=3)
> +        assert returncode != 0
> +    finally:
> +        # terminate a timed-out tftp
> +        barebox.console.sendcontrol("c")
> +        tftp_thread.join()
> +        barebox.run_check("ifdown eth0")
> +
> +
> +def test_barebox_network(barebox, env):
> +    # on DUTs without network feature, this is expected to fail
> +    # set xfail_strict=True to enforce specifying the network feature if available
> +    if not 'network' in env.get_target_features():
> +        pytest.xfail("network feature not specified")
> +
> +    stdout = barebox.run_check("ifup -a")
> +
> +    interfaces = []
> +
> +    for line in stdout:
> +        # eth0: DHCP client bound to address 10.0.2.15
> +        m = re.search('(eth[0-9]+): DHCP client bound to address (.*)', line)
> +        if m is None:
> +            continue
> +        else:
> +            interfaces.append((m.group(1), m.group(2)))
> +
> +    assert interfaces, "Network testing is only possible with at least one DHCP interface"
> +
> +    for iface in interfaces:
> +        ifname = iface[0]
> +        ifaddr = iface[1]
> +
> +        assert ifaddr != "0.0.0.0"
> +        assert ifaddr == barebox.run_check(f"echo ${ifname}.ipaddr")[0]
> +
> +        # Attempt a conversation with the DUT, which needs to succeed only on one interface
> +        success = False
> +
> +        try:
> +            tftp_conversation(barebox, ifname, ifaddr)
> +            success = True
> +            break
> +        except Exception as e:
> +            warnings.warn(f"Could not connect to DUT on {ifname} ({ifaddr}): {e}")
> +
> +        if not success:
> +            pytest.fail("Could not converse with DUT on any of the found DHCP interfaces!")
> +
> 
> ---
> base-commit: 5a2866973d386ffe9b9f5c429a0cbcd58bcef69c
> change-id: 20240605-test_tftp-9e12282d46a0
> 
> Best regards,


-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [PATCH v2] test: pytest: introduce pytest for network, test tftp notfound

[Sascha Hauer]

On Tue, 08 Oct 2024 13:21:47 +0200, Jonas Rebmann wrote:
> barebox already has a few pytest integration tests but none of them are
> networking related.
> 
> This patch introduces a test that asserts on bringing up at least one
> DHCP interface and, on any of these interfaces, a successful ICMP-ping
> from the DUT to the test pytest-host as well as a basic "File not Found"
> conversation between the DUT and the test pytest-host.
> 
> [...]

Applied, thanks!

[1/1] test: pytest: introduce pytest for network, test tftp notfound
      https://git.pengutronix.de/cgit/barebox/commit/?id=2240367552f3 (link may not be stable)

Best regards,
-- 
Sascha Hauer <s.hauer@pengutronix.de>