Issues with Tools on clab

[bendergm]

Can someone help me out and confirm if this should work?
I cannot seem to get Tools such as SuzieQ or Graphite to connect to the management interfaces for the iol routers.
Even while both Host system to iol device and between iol devices on the clab-mgmt vrf works.

A simple example topology,

---
defaults.device: iol
defaults.devices.iol.image: vrnetlab/cisco_iol:17.15.01
provider: clab

nodes:
  r1:
  r2:

links:
- r1:
  r2:

tools:
  graphite:
  suzieq:

I have been trying to figure it out on my own for the past few days, without luck. I have tried both Debian and Ubuntu on both a Virtual Machine and within Windows WSL.

I have tried docker exec -it into each container to verify ping does work properly, but the SSH traffic does not complete.
When I connect other containers, such as wbitt/network-multitool, to the netlab_mgmt bridge network, I can get PCAPs showing a partial SSH connection. I even tested with all the settings from the command output from netlab connect r1 -vvv with the ssh config files.

• Ping between any devices on the bridge does work ✔️
• SSH from Host machine works ✔️
• SSH between network devices works ✔️
• SSH from suzieq or graphite container to a network devices gets a Connection Timeout ❌
• SSH from another container added onto the bridge network also gets a Connection Timeout ❌
• SSH from a network devices to another container added onto the bridge (network-multitool) DOES see an SSH SYN connection attempt on a pcap. The multitool responds with a SYN ACK, but then I get TCP Retransmissions. Oddly both directions. As if only traffic FROM a non-network device (or host) TO the network devices is getting dropped. ⁉️
• SSH from graphite to the multitool container does work ✔️

Also tested device type frr without success, though I did not test too thoroughly.
Been loving netlab for the past year and would like some new ways to interact with larger topologies.

[ipspace]

Can someone help me out and confirm if this should work? I cannot seem to get Tools such as SuzieQ or Graphite to connect to the management interfaces for the iol routers.

I had the same problem. The last time I checked, using debug ip tcp something on Cisco IOL, it reported checksum errors when external tools tried to establish TCP connection with it.

I documented it in https://netlab.tools/caveats/#cisco-ios-on-linux-iol-and-iol-layer-2-image and moved on.

Never looked into what exactly is causing it, could be TCP offload on some Linux interface.

Also tested device type frr without success, though I did not test too thoroughly.

FRR containers do not run an SSH server. You could use FRR virtual machines, but that consumes way more memory.

If you want to test tools like SuzieQ or Graphite with containers, use Arista cEOS.

Been loving netlab for the past year

Thanks a million.

[ipspace]

I am going to keep working at, as it is just too valuable of a feature to give up on.

Thank you. Please let me know if you find something that could be fixed in netlab.

[bendergm]

Just like you said, its related to a checksum error because docker enables some Tx checksum offloading and never recomputes the value properly.

Temporary Fix

One can temporarily fix the issue by Disabling Tx offload on each container management veth. The container veth interface can easily be found with sudo ip link since the elevated execution includes the container name in the output. The ethtool command requires sudo anyways.

$ sudo ip link
...
114: veth46d3091@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2ece48482ead state UP mode DEFAULT group default
    link/ether 5a:21:82:3e:e0:09 brd ff:ff:ff:ff:ff:ff link-netns clab-TestNet-r1
...

$ sudo ethtool -K vethf97dc0d tx off
Actual changes:
tx-checksumming: off
        tx-checksum-ip-generic: off
        tx-checksum-sctp: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [requested on]
        tx-tcp-mangleid-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]

Permanent Fix Thoughts

It feels like anything done in netlab would just be duct tape. I am sure there would be a place to insert this Temporary Fix for each container on a conditional for device type, but I spent more time digging down into Containerlab.

Containerlab has certainly ran across this, and offers a tool aptly called disable-tx-offload command. Unfortunately, this does not work for IOL since the device mgmt interface is not named "eth0". The tool command bakes in the "eth0" string.

The underlying command util function seems to have the ability to support a Disable Tx Offload for any interface, see ethtool.go function for NSEthtoolTXOff. Digging around, I saw SROS uses another utility function in an odd parent function that is not present in the IOL kind. But the linux kind has a good example that calls same NDEthtoolTXOff function called in a PostDeploy step. There is already an IOL PostDeploy step I feel like this can be slotted into.

If you made it this far, thank you again, and let me know what you think. I will plan to create an issue in Containerlab that includes this detail.

[ipspace]

Excellent analysis, thanks a million! Totally agree with you, it would be best if this could be solved in Containerlab.

They are usually pretty responsive, so let's start with a Containerlab issue, and if needed, we can always apply another layer of duct tape ;)

[bendergm]

I may have jumped the gun and submitted a pull request before submitting an issue.
But it was merged!
srl-labs/containerlab#3077

I will keep an eye on it to see if it will be functional in the next release.

[ipspace]

But it was merged! srl-labs/containerlab#3077

Yeah, I've seen that. Congratulations

I will keep an eye on it to see if it will be functional in the next release.

Please let me know if it works and I'll update caveats.md (or, even better, submit a PR ;)

[DanPartelly]

Please note that ssh works. If all those checksum errors are induced by emulated hw offloading, please tell me by what mechanisms you think we can actually have hours long sessions of ssh into iol without any issues , but suzieq has issues. How does one works over those wires and the other one not

This needs more analysis - at least find exactly when/why it happens and what is the exact type of packet which triggers it.

[ipspace]

Host-originated SSH is originated from the bridge interface:

$ sudo ethtool -k br-0301014b8db9 | grep segmen
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [requested on]
	tx-tcp-ecn-segmentation: off [requested on]
	tx-tcp-mangleid-segmentation: off [requested on]
	tx-tcp6-segmentation: off [requested on]
generic-segmentation-offload: on
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-tunnel-remcsum-segmentation: on
tx-sctp-segmentation: on
tx-esp-segmentation: on
tx-udp-segmentation: on

Container TCP sessions are originated from the container veth interface connected to that bridge interface.

$ sudo ethtool -k vethf671597 | grep segmen
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
generic-segmentation-offload: on
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on

Spot the difference ;)

[DanPartelly]

And the veth from iol ? To get into iol, you have to take the second veth, which it set how ? RX/TX path ? Or why does aRista work ? Is any of those wires -- suzieq / arista -- setup differently with Arista ?

[ipspace]

And the veth from iol ? TO get into iol, you have to take the second veth, which it set how ?

Obviously IOL has problems with incoming TCP packets that were fragmented (or whatever) by the Linux kernel. It wouldn't report checksum errors on outgoing packets, would it? Also, no TCP packets are fragmented when sent from IOL because IOL TCP stack fragments them.

Also, due to the errors being checksum errors (and SSH sessions not being established at all), I'd suspect it's not really fragmentation but checksum offload. But hey, what do I know about Linux interfaces?

The really fun part: SSH session from a Linux container that is part of topology to IOL over the management interface works. SSH session from a tool connected to the same management interface does not. In the end, we might have to be the ones doing the twiddling.

[DanPartelly]

I do not disagree with you, I just find it pretty weird. Why does Suqieq works with Arista then ? I doubt veths are differently setup Anyways, maybe doesn't worth learning why.

[ipspace]

• Arista cEOS is a pure container using Linux TCP stack
• Most other things are VMs running in containers and have their interfaces somehow connected to the Linux bridge. IIRC those could be tap interfaces.
• IOL is a Linux process running a TCP stack internally

Let me just say that after seeing everything I had to see in the last few years, I wouldn't be surprised if the TCP checksum would be invalid on a packet going from one Linux process to another for optimization reasons. Time to run a packet capture?

[ipspace]

Ran a few more checks and tried out all the possible options. The only thing that solves the issue is disabling offloads (tx off) on the host side of the veth pair connecting IOL to the management network.

Disabling offloads on container interfaces or on the host side of the tool container interface does not help.

[DanPartelly]

Thanks for this.