Roaming client speed issue

[TriAndy]

Hi, we have several staff working remotely as a result of covid but as part of our compliance requirements all traffic from these machines needs to run through a hardware firewall. We were using OpenVPN but switched to WireGuard full tunnel for better speeds. Everything works fine but we are not seeing the speeds I would have hoped through it. I suspect there is a fairly simple fundamental reason but would love some helping in working out what it is! The setup looks like this:

Datacentre
1Gbps each way fibre internet connection
Untangle running 16.5.0 as a VM on Hyper-V - 8GB RAM, 4vCPUs on a Xeon E5-2430, 500GB disk on a SAS RAID 10 array - no hardware resources look remotely loaded when running a speed test on this
MTU set to 1500 and tested to confirm
WG set to use 8.8.8.8 as the DNS server with

Office
Starlink internet - 120-150MB down and 20-30MB up
MTU set to 1500 and tested to confirm
Untangle Firewall running 16.5.0 on NUC (i5, 8GB, 250GB SSD)
Various laptops that work from here or from home\out on the road, hence no site to site VPN but instead all setup as roaming full tunnel clients

Using a speedtest website such as fast.com with no VPN we see the Office speeds, with the VPN I can't break 30MB down and 7MB up. This is true on alternative internet connections, some staff have fibre internet at home (300MB down and 30MB+ up) and still can't break the 30MB down on an online speed test.

I've tried playing with QoS rules to priortise WG traffic (based on IP ranges on each side, port number and interface) on both UT's in the above example but it made no difference. Also set both UT's to bypass the WG traffic and adjusted QoS accordingly again to no avail.

Where do I go next with this?

Thanks

Andy

[dashpuppy]

Hi, we have several staff working remotely as a result of covid but as part of our compliance requirements all traffic from these machines needs to run through a hardware firewall. We were using OpenVPN but switched to WireGuard full tunnel for better speeds. Everything works fine but we are not seeing the speeds I would have hoped through it. I suspect there is a fairly simple fundamental reason but would love some helping in working out what it is! The setup looks like this:

Datacentre
1Gbps each way fibre internet connection
Untangle running 16.5.0 as a VM on Hyper-V - 8GB RAM, 4vCPUs on a Xeon E5-2430, 500GB disk on a SAS RAID 10 array - no hardware resources look remotely loaded when running a speed test on this
MTU set to 1500 and tested to confirm
WG set to use 8.8.8.8 as the DNS server with

Office
Starlink internet - 120-150MB down and 20-30MB up
MTU set to 1500 and tested to confirm
Untangle Firewall running 16.5.0 on NUC (i5, 8GB, 250GB SSD)
Various laptops that work from here or from home\out on the road, hence no site to site VPN but instead all setup as roaming full tunnel clients

Using a speedtest website such as fast.com with no VPN we see the Office speeds, with the VPN I can't break 30MB down and 7MB up. This is true on alternative internet connections, some staff have fibre internet at home (300MB down and 30MB+ up) and still can't break the 30MB down on an online speed test.

I've tried playing with QoS rules to priortise WG traffic (based on IP ranges on each side, port number and interface) on both UT's in the above example but it made no difference. Also set both UT's to bypass the WG traffic and adjusted QoS accordingly again to no avail.

Where do I go next with this?

Thanks

Andy

What about a Iperf test ? ( through the vpn tunnel )

[Garrett Brown]

I'm interested to see this post. We're seeing similar behavior across several systems. I've been unable to identify the source of the slowdowns. We're seeing artificially slow and inconsistent Wireguard connections speeds for roaming clients across multiple Untangle systems. Given the number of variables, it's very hard to track down the issue. It would be nice to get clarification from Untangle about how resources are allocated and possibly limited for these tunnels.

[sky-knight]

Wireguard is baked into the Linux kernel. Untangle has extremely limited tools to manage things that are running on the Untangle server itself. Basically the only things that work are the filter, I suppose QoS itself might impact that stuff, but I doubt it.

If Untangle's wireguard is slow, and a dedicated WG service behind Untangle is not... That would prove it's something in Untangle. But my experience with VPNs on Untangle is they operate at the speed that the platform provides. I'm not a huge fan of virtual untangle in this space because you need to allocate dedicated CPU and RAM resources to the device to offset for the software networking load.

Worse, the above load will NOT APPEAR in TOP or any other performance monitoring tool. Because it's the kernel and it cannot monitor itself.

[TriAndy]

Thanks all for the replies, will try and iperf. @sky-knight good shout re physical server resources but having checked again it doesn't appear to be relevant. Run more speed tests watching just the physical server resouces and CPU less 15%, RAM 50% free, NIC and Disks don't show any issues either. Any other suggestions, for the record I'm not suggesting WG on Untangle is slow more that it feels like there is a setting wrong\needs amending on this to get better speed.

[sky-knight]

Do you get expected performance outside of the VPN tunnel?

Also, do please monitor your CPU loads with your hypervisor's tools, not Untangle's... Because again you cannot see kernel loads from within the kernel that's doing the load!

Something about finding the fiend first before you can use the fiend finder.

[Garrett Brown]

One thing we've documented is that a lot of things can impact the performance from a UDP traffic standpoint. If we connect to a Wireguard VPN from behind an Untangle, performance drops dramatically and gets slowed to predictable rate. If we bypass that traffic from Untangle inspection, performance resumes normally. So there could be a lot of variables for users depending on where they're connecting from including on the carrier side. We've experimented with different UDP ports without much luck.

[sky-knight]

That's fascinating... because Untangle automatically bypasses UDP sessions after the initial packets are inspected.

So perhaps that initial inspection is slowing things down and setting some sort of cap during negotiation?

Regardless that behavior wouldn't impact the services running on Untangle itself, because all traffic terminating on Untangle itself is bypassed by design and default.

So I would expect a performance improvement of a VPN server running behind Untangle if bypassed, and possibly a VPN client behind Untangle too. But never the VPN server running on Untangle itself, because again bypassing things twice doesn't change anything.

[TriAndy]

Hi all, quick update on this. Bypassing the VPN traffic does help and with that in place my colleagues on FTTP connections are now getting speeds that are more than acceptable. My issue appears to be the Starlink connection, best I can tell that is not getting used for the VPN. We have two internet connections at the Office, Starlink which is alot faster and a normal business broadband FTTC connection at 35MB down and 5MB up. The office VPN connections appear to only work via the FTTC and the speeds then reflect that. I could play with the WAN balancing app to push all traffic through the Starlink connection but for now I have my answer.

[sky-knight]

WAN Balancer has nothing to do with how ingress services connect.

Untangle will list all IP addresses or hostnames available in the VPN configurations it generates. If you want to force VPN users to a specific WAN, you'll need to edit those configuration files to make that happen. Or... even easier, use config -> hostname to set Untangle to use a hostname, and configure aforementioned DNS record publicly to only resolve to the IP address on the connection you want to use. That way you can easily change that address later.