Friday 26th September 2014: 1.43am. Getting towards the end of my week long vacation ... and the chores are getting harder! There has been something subtly wrong with our internet here for a long time, but I could never quite put my finger on it. A few days ago I got annoyed that incoming phone calls to the house phone didn't present audio - stupid double NAT strikes again - so I resolved to actually do something about it, which culminated in me declaring defeat in persuading that crap residential vDSL modem from Magnet Broadband to behave itself, and simply ordering a Business Broadband connection not encumbered by all that crap instead. It did however also highlight more wrongness with the internet connection, stuff like Facebook or Google not loading properly sometimes, but not other times. I knew it was MTU related, because the big problems began when I dropped the MTU to 1400, though they had always been there before, just infrequently.
Anyway, I finally figured it out after much trial and error - path MTU discovery had got broken! MTU stands for Maximum Transmission Unit, and it is the largest amount of data the internet sends at any given time for some connection. How large that should be is anywhere between 576 bytes and 65535 bytes, and how it is determined is by programs sending a larger amount marked as "don't send if it's too big", and if it can't get through because it's too big, you get back a ICMP message saying it is too big with the largest value that particular stage on the route to destination can handle. You then scale down to that size and try again, and eventually you will find the lowest common denominator which lets you reach your destination. You from then on send packets at that size (ideally), but if you exceed the MTU then the intervening hardware may break up packets into smaller ones, forcing your recipient to reassemble the fragments.
In my case, I have 9000 byte MTUs here local to the house as it's all gigabit networking here. The vDSL enforces a 1464 byte MTU due to my ISP using PPPoE, so all default i.e. non-path-MTUed packets heading towards the internet may get broken up into 7 fragments at the point of leaving the house. Ordinarily these get reassembled at the destination, and no one is the wiser.
Unfortunately, encrypted packets (any website with a https prefix) are marked as "don't send if they are too big" for security reasons. This therefore requires path MTU discovery to be working correctly, because otherwise your encrypted connection packets will get dropped when they exceed MTU, then the implementation will note the dropped packets and back off for a while so your encrypted connection works again, but then start increasing packet size again because path MTU discovery says you should. Hence unreliable encrypted connections, and a very unhappy Megan when she fails to upload photos of Clara to Facebook!
The cause of path MTU discovery failing was quite subtle. My pfsense firewall is based on FreeBSD 8 and runs inside a virtual machine which has, of course, virtual network cards using the virtio drivers. These, it turns out, silently fail to return the ICMP failed to deliver message for packet sizes between 1400 and 1500 bytes if and only if hardware segmentation send and receive offload is disabled - yes, that is a very subtle bug. And I had turned hardware offload off purely because the docs for pfsense with virtio drivers had suggested that routing between VMs on the hypervisor fails with hardware checksum offload turned on, so I had simply disabled all hardware offload - I had thought that hardware offloading to a virtualised network adaptor seemed pointless as it's all software anyway. It seems, in fact, I was switching on a poorly tested code path instead.
Anyway, turning hardware offload back on fixes the loss of ICMP replies, and with that Google Play on our Android devices suddenly started working again and a ton of updates came down. So there you go - that's what one calls a subtle bug eh? And I might add that internet responsiveness in general has hugely improved - before every single unencrypted packet sent was getting fragmented and some percentage of encrypted packets were needing to be resent, which led to annoying pauses when browsing websites etc. That's all gone now - it simply runs at full speed. It's quite noticeable.
#pfsense #packetloss #virtualization #kvm
| Go to previous entry | Go to next entry | Go back to the archive index | Go back to the latest entries |