Tweet
PPPoE/VLAN, MTU and Bypass Rule (an Eternal Golden Braid?)
Hello.
I'm not sure this is a question, a heads-up or simply a topic for those rainy days.
I had some issues with my custom appliance and I reinstalled it, though I only could find a 16.6.2 ISO while my last backup was from a 16.5.x. It seemed to have restored everything fine (a few glitch here and there, but more or less all seems ok). I then changed my WAN to a new PPPoE addressed vlan subinterface (as it is needed by the new provider) and it correctly got an IP. Then it all went upside down.
I've been replicating what follows since yesterday night, so it does not seem something related to an internet provider fluctuation. The only thing that changed along the restore process (which I now know is not related with it at all) is that before I had a ppp0 on the addressed external interface with a modem doing the VLAN connection and acting as passthrough to UT (so no VLAN in the interface configuration). Now I have the addressed ppp0 with VLAN on a subif of the disabled external.
Most of the traffic was not completely functional: some websites worked, some didn't. Some seemed to work better in Firefox, others in Chromium or Safari. SSHing to a remote server did not work at all, all speed test websites showed an upload of ZERO megabits. So I thought: If I set a bypass rule for my PC ip, everything will work. And it did work. I then tried to disable EVERY app in the policy chain my PC is assigned to, as I thought it might be like bypassing it, but that did not work and I had the same symptoms I described.
While tcpdumping ppp0 and internal during the SSH attempt, I saw that, while bypassed, the TCP window of the client kept incrementing till 2000+ values and I correctly received ACKs for my PSHed data. If the client was not bypassed, the window was limited to around 500 and I didn't receive ACKs from the SSH server and UT then kept retransmitting its ACKs (but then the connection was almost desyncronized). While I did not tcpdump it, the end user symptoms while attempting to browse some web sites were the same: somehow they stayed stuck till a 504 or until the browser stopped the request (whatever happened first).
As soon as the client was bypassed in UT, everything worked. If it was not, even if all policy apps was disabled, this intermittent behavior happened immediately. I then tried the SSH session from UT shell itself and it showed the same symptoms (connection attempt stuck, TCP window and missed or not received ACKs).
Then I had a look at ppp0 MTU and changed it from 1492 to 1480 and all started working as it should. 1492 was the automatically configured value - and while chatting with a tech guy from the internet provider support, asking if it was them announcing the 1492 it through pppoe (and it was) I saw that the same value is configured in /etc/ppp/peers/dsl-provider.
PPPoE can be disturbing and is deprecated in many places, but that's what I get from all FttH providers in my country. Now, though, the interesting summary:
- with my FttH provider and a ppp0 MTU of 1492 SSH from UT box to outside does not work
- with my FttH provider and a ppp0 MTU of 1480 SSH from UT box to outside works
- if the client is bypassed it works with both 1480 and 1492 but with the latter it will dynamically adjust at least the TCP window to work properly
- if the client is not bypassed and UT processes its traffic, then it won't work with 1492 but will work with 1480
- if I disable every app in the policy chain the client is assigned to, if there is no bypass rules for it, then it still won't work with a 1492 MTU
With xDSL and FttH there is a great history of compatibility issues (not all of them critical of course) between different firmwares/equipments from the client modem to the ISP core machines. So the first question might be: is there a Debian networking ninja that knows what sys parameters to change in this scenario? Or would it be useful, although rarely, to have a MTU option in the interface configuration for PPPoE? And what is UT doing to clients' traffic when every app is disabled or, better yet, do really disabled app remain silent or is some processing done anyway (you can read it as: what's the difference between disabling all policy apps and setting a bypass rule)?
I would really love, for personal knowledge, listening to somebody from UT/Arista regarding this, since it is at least interesting (I suppose). Thank you.
I'm not sure this is a question, a heads-up or simply a topic for those rainy days.
I had some issues with my custom appliance and I reinstalled it, though I only could find a 16.6.2 ISO while my last backup was from a 16.5.x. It seemed to have restored everything fine (a few glitch here and there, but more or less all seems ok). I then changed my WAN to a new PPPoE addressed vlan subinterface (as it is needed by the new provider) and it correctly got an IP. Then it all went upside down.
I've been replicating what follows since yesterday night, so it does not seem something related to an internet provider fluctuation. The only thing that changed along the restore process (which I now know is not related with it at all) is that before I had a ppp0 on the addressed external interface with a modem doing the VLAN connection and acting as passthrough to UT (so no VLAN in the interface configuration). Now I have the addressed ppp0 with VLAN on a subif of the disabled external.
Most of the traffic was not completely functional: some websites worked, some didn't. Some seemed to work better in Firefox, others in Chromium or Safari. SSHing to a remote server did not work at all, all speed test websites showed an upload of ZERO megabits. So I thought: If I set a bypass rule for my PC ip, everything will work. And it did work. I then tried to disable EVERY app in the policy chain my PC is assigned to, as I thought it might be like bypassing it, but that did not work and I had the same symptoms I described.
While tcpdumping ppp0 and internal during the SSH attempt, I saw that, while bypassed, the TCP window of the client kept incrementing till 2000+ values and I correctly received ACKs for my PSHed data. If the client was not bypassed, the window was limited to around 500 and I didn't receive ACKs from the SSH server and UT then kept retransmitting its ACKs (but then the connection was almost desyncronized). While I did not tcpdump it, the end user symptoms while attempting to browse some web sites were the same: somehow they stayed stuck till a 504 or until the browser stopped the request (whatever happened first).
As soon as the client was bypassed in UT, everything worked. If it was not, even if all policy apps was disabled, this intermittent behavior happened immediately. I then tried the SSH session from UT shell itself and it showed the same symptoms (connection attempt stuck, TCP window and missed or not received ACKs).
Then I had a look at ppp0 MTU and changed it from 1492 to 1480 and all started working as it should. 1492 was the automatically configured value - and while chatting with a tech guy from the internet provider support, asking if it was them announcing the 1492 it through pppoe (and it was) I saw that the same value is configured in /etc/ppp/peers/dsl-provider.
PPPoE can be disturbing and is deprecated in many places, but that's what I get from all FttH providers in my country. Now, though, the interesting summary:
- with my FttH provider and a ppp0 MTU of 1492 SSH from UT box to outside does not work
- with my FttH provider and a ppp0 MTU of 1480 SSH from UT box to outside works
- if the client is bypassed it works with both 1480 and 1492 but with the latter it will dynamically adjust at least the TCP window to work properly
- if the client is not bypassed and UT processes its traffic, then it won't work with 1492 but will work with 1480
- if I disable every app in the policy chain the client is assigned to, if there is no bypass rules for it, then it still won't work with a 1492 MTU
With xDSL and FttH there is a great history of compatibility issues (not all of them critical of course) between different firmwares/equipments from the client modem to the ISP core machines. So the first question might be: is there a Debian networking ninja that knows what sys parameters to change in this scenario? Or would it be useful, although rarely, to have a MTU option in the interface configuration for PPPoE? And what is UT doing to clients' traffic when every app is disabled or, better yet, do really disabled app remain silent or is some processing done anyway (you can read it as: what's the difference between disabling all policy apps and setting a bypass rule)?
I would really love, for personal knowledge, listening to somebody from UT/Arista regarding this, since it is at least interesting (I suppose). Thank you.
Comment