Throughput drop with newer Linux / Openwrt version.

[pparent76]

Hello,

On an Edge-router X (mt7621 processor), I observe the following difference in throughput with the exact same configuration and with Coova-Chilli 1.9, with xt-coova enabled.

Openwrt 23.05.3, linux 5.15.189 : 350-550 mbps
Openwrt 24.10.6, linux 6.6.127 : 50-110 mbps

I'm not sure where this could come from. I will try to inspect the code of the xt-coova kernel module. Do you think it has been properly optimized, especially in the context of newer kernels?

[sevan]

Do you think it has been properly optimized, especially in the context of newer kernels?

Most certainly not, but why do you not suspect a regression on the kernel side?

[pparent76]

Most certainly not, but why do you not suspect a regression on the kernel side?

Because, now I can't explain the difference in Throughput between those two setups. And the major difference between them (at least from my current point of view and knowledge) is the kernel version. Though of course it is possible that the cause is something else. (Edit another hypothesis I have is that there is a change in the way tunnels are handled between kernel versions)

[pparent76]

Ps: It could be related to the version of iptables but I don't think it is the most probable

Openwrt 23.05.3: iptables v1.8.8 (legacy)
Openwrt 24.10.6, linux 6.6.127 : iptables v1.8.10 (legacy)

[sevan]

Put the kernel from OpenWRT 24.10.6 on a device running OpenWRT 23.05.3 and see if the performance drops then? that will rule out iptables.

[pparent76]

I really don't know how to Put the kernel from OpenWRT 24.10.6 in a OpenWRT 23.05.3 system. I don't know if it is even possible because there are many things in Openwrt that depend on a specific kernel version.

I think I will rather test various changes to the xt-coova module to see if anything has an impact on performance.

[pparent76]

Well I think it does not come from Xt-coova module because I've tried to add the following just for testing:

static bool
coova_mt(const struct sk_buff *skb, struct xt_action_param *par) 
{
>      return true;

And it did not allow to get back to trhoughput obeserved in Openwrt 23.05.3 . Would you come to the conclusion that the xt-coova module is not at fault? @sevan

My next hypothesis, is that it is a problem with the tunel, and Linux has changed the way it handles tunnels, and it's less efficient to do the operations that Chilli does. Do you think it is possible?

Thank you very much!

[sevan]

I really don't know how to Put the kernel from OpenWRT 24.10.6 in a OpenWRT 23.05.3 system. I don't know if it is even possible because there are many things in Openwrt that depend on a specific kernel version.

Copy the kernel, associated modules directory, and initrd ramdisk image from the newer OS to the older OS version & manually boot the system from the newer kernel. (don't worry about working out how to convince the OS tools to install a kernel version using the regular method if that's where you're stuck).

[pparent76]

Copy the kernel, associated modules directory, and initrd ramdisk image from the newer OS to the older OS version & manually boot the system from the newer kernel.

I'm not sure to follow you in openwrt the filesystem is read-only, the only simple possibility that I am aware of, to change the kernel, is to build a new firmware and flash it. Building a new firmware based on Openwrt 23.05 and a newer kernel seems a complex task. ( There might be also the possibility with UART+Tftp on some devices to run a specific kernel, but no grantee that it be trivial ).

Any thought on the result of what I posted above? Do you think that it's more likely to be tun handling, or xt-coova kernel module, that is the bottleneck in terms of CPU usage?

[pparent76]

By the way here is what the CPU consumption looks like on a Quad core processor:

CPU:   4% usr  21% sys   0% nic  45% idle   0% io   0% irq  27% sirq
Load average: 0.45 0.67 0.68 3/137 28953
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
   25     2 root     RW       0   0%  25% [ksoftirqd/2]
21460     1 root     S     6548   3%  20% chilli --mtu=65000 -c /etc/chilli.conf

So the load is shared by the kernel process and chilli process, but it seems that the kernel process consumes slightly more and is the limiting factor.

Then about the question of whether this CPU consumption comes from the tunnel handling or iptables/xt_coova handling, right now I would tend towards the hypothesis of the tunnel being the culprit. Because if I modify the xt-coova module to shortcut everything with

static bool
coova_mt(const struct sk_buff *skb, struct xt_action_param *par) 
{
>      return true;

Then it does not have any impact. And if I add iptables -I FORWARD -j ACCEPT, which should prevent to even reach the iptable rule with xt-coova module, it does not have any impact either.

I will try to experiment some changes in the code related to tunnel management to see if it has any impact.

Any feedback or guidance will be appreciated.

Ps: is there any reason that that the tunnel handling is mono-core and mono-thread in coova?

[pparent76]

I want to mention 2 things:

I notice that on a regular Nat network on the same router I've also had throughput drop between linux 5.15.189 / Openwrt 23.05.3 and linux 6.6.127/ Openwrt 24.10.6, linux 6.6.127 . When reaching the max throughput I seen [ksoftirqd/2] reach 25% CPU ( or 100% of one core ). I will test if the use of nftables instead of iptables (legacay), can improve the situation?

I've also experimented performance with a few different variations of the coova-chilli, code using AI to prototype fast, and I've succeeded to improve a bit the performance adding a BPF filter, but it could not compensate for the difference in Linux versions.

By the way the router I test on, is based on Mt7621/Mips.