This post will be mainly about testing the seventh generation of my public server infrastructure. I previously discussed last December the current economics of the market driving me towards a colocated server solution for the first time ever, which showed quite the shift in exigencies in recent years. As you will see, this new solution has big gains in some areas, but a few regressions in others.
Firstly, to summarise past posts a little, what has shifted is that obsolete servers offered as budget dedicated servers have been rising in price as datacenter electricity has risen in price. This is because obsolete hardware whilst nowadays very good at idle power consumption, they can still consume a fair whack of power if doing anything, so their peak power consumption makes them expensive. If you can reduce your space footprint down to two credit cards and your power consumption down to fifteen watts or especially ten watts or less, there are colocation options available nowadays far cheaper than renting a budget obsolete server.
I gave a list of those I could find in previous posts, and I ended up choosing the cheapest which was FinalTek in the Czech Republic at €1.97 inc VAT per server per month if you buy three years of colocation at once. This, as my noted in earlier posts, is a 100 Mbit ‘unlimited’ shared service on a 1 Gbit switch, so you get 1 Gbit between the two Pis but up to 100 Mbit to the public internet. I’ll quote their setup email at this point for requirements:
The device must meet the following parameters:
- it must be in a box (not a bare board with exposed circuitry)
- the power adapter to the device must be for Czech power grid (EU power plug)
- dimensions must not exceed 15 x 10 x 5 cm
- must not have a power consumption greater than 5V / 3A
- must be set to a static IP address
- send the device to the address belowIf the device uses an SD card for operation, it is advisable to pack spare cards with a copy of the OS in case the primary card fails.
As mentioned in previous posts, it is the 5v @ 3a requirement which filters out Intel N97 mini PCs which probably can be configured to run under 15 watts (and definitely under 20 watts), but they generally need a 12v input. They’re far better value for money than a Raspberry Pi based solution which is around one third more expensive for a much worse specification. You can of course fit a higher spec Banana Pi or any other Single Board Computer (SBC) also able to take 5v power, but none of those have the effortless software ecosystem maturity of the Raspberry Pi i.e. an officially supported Ubuntu Server LTS edition which will ‘just work’ over the next four years. So, to put it simply, the one single compelling use case I have ever found for a high end Raspberry Pi is cheap dedicated server colocation. For this one use case, they are currently best in class with current market dynamics.
Even with the very low monthly colocation fees, this hasn’t been an especially cheap exercise. Each Raspberry Pi 5 with case cost €150 inc VAT or so. Add power supply €10 and used NVMe SSD off eBay €35 and you’re talking €200 inc VAT per server. Over three years, that’s equivalent to €7.64 inc VAT per server per month which is similarly priced to my existing rented Intel Atom C2338 servers (~€7.38 inc VAT per server per month). So this solution overall is not cheaper, but as previous posts recounted you get >= 2x the performance, memory and storage across the board. And, the next cheapest non-Atom rented server is €21 inc VAT per month, and this is one third the cost of that all-in.
Assuming market dynamics continue to shift along their current trajectories, in 2032 when I am next likely to look at new server hardware it’ll be interesting to see if wattage per specification will let me reduce watts for good enough hardware for a 2030’s software stack. In the list of colocation providers I gave in previous posts, many capped power to ten watts max or price went shooting up quickly. That’s enough for a Raspberry Pi 4, but they’re as slow as my existing Intel Atom C2338 rented servers plus they can’t take a NVMe SSD. Seven years from now, I would assume there will be a Raspberry Pi 6 and/or cheap colocation for 12v ten watt max mini PCs might be now affordable. It’ll be interesting to see how trends play out.
In any case, server capability per inflation adjusted dollar continues its exponential improvement over time. The fact I can colocate a server for the cost of a cup of coffee per month is quite astounding given I grew up in a time where colocation space cost at least €10k per 1U rack per month. I reckon they’re fitting ten to twelve SBCs per 1U rack, so that’s ~€20-24 per rack slot per month. Which is 99.8% cheaper than in the 1990s! In case you’re wondering, a 1U slot with max 100 watts power currently costs about €30-40 ex VAT per month, so I guess FinalTek are relying on those Pis to not draw all of their fifteen watt power budget to make a profit!
Raw storage device performance
The Raspberry Pi 5 has a single PCIe 2.0 lane available to be connected to a NVMe adapter. Much debugging and tweaking has been done by RPI engineers in the past year to get that PCIe lane running at 3.0 speed and working without issue over a wide range of NVMe SSDs. The most recent significant compatibility improvement was only in December 2024’s firmware, so this is an ongoing process since the Pi 5 was launched in Autum 2023.
Most – but not all as we shall see – original RPI NVMe SSD compatibility
issues have been worked around such that compatibility is now very good.
Just make sure you install a year 2025 or newer EEPROM firmware and you
should be good to go with running PCI 3.0 on any of the after market NVMe expansion kits.
I ended up fitting a 256 Gb well used Samsung SM961 off eBay to
europe7a and an official Raspberry Pi 512 Gb NVMe SSD to europe7b
after wasting a lot of time on a Samsung PM9B1 SSD which nearly works.
It turns out that the Samsung PM9B1 actually has a Marvell controller,
and that is very finickety: it doesn’t like the RPI, it also doesn’t
like one of the USB3 NVMe enclosures I have, but it’s happy in the other
USB3 NVMe enclosure I have. I think there’s nothing wrong with the SSD
apart from limited compatibility, and as the PM9B1 was an OEM only
model they only needed it to work well in the OEM’s specific hardware.
The official Raspberry Pi NVMe used to be a rebadged Samsung PM991a which is a superb SSD. Unfortunately, at some point they silently swapped it for a rebadged Biwin AP425 which is an industrial SSD. The Biwin is fast in smoke testing, but it doesn’t implement TRIM so I’m unsure how either its performance or longevity would hold up over extended use. It is also a RAMless design, and having a RAM cache on the SSD particularly benefits random reads for the Pi from testing. So the used Samsung SSD with about 80k hours and ~25Tb written (i.e. 100 total drive writes, which the firmware thinks is 9% spare threshold used) ended up going into the primary server, and the brand new Biwin SSD into the failover server.
For the 256 Gb Samsung SM961
dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=10000 oflag=direct
10485760000 bytes (10 GB, 9.8 GiB) copied, 15.1569 s, 692 MB/s
Idle is 0.3 watts, write load is +3.6 watts.
dd of=/dev/null if=/dev/nvme0n1 bs=1M count=10000 iflag=direct
10485760000 bytes (10 GB, 9.8 GiB) copied, 12.6483 s, 829 MB/s
Idle is 0.3 watts, read load is +3.3 watts.
For the 512 Gb Biwin AP425 (official RPI NVMe SSD)
dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=10000 oflag=direct
10485760000 bytes (10 GB, 9.8 GiB) copied, 14.5891 s, 719 MB/s
Idle is near zero, write load is +0.9 watts.
dd of=/dev/null if=/dev/nvme0n1 bs=1M count=10000 iflag=direct
10485760000 bytes (10 GB, 9.8 GiB) copied, 13.4262 s, 781 MB/s
Idle is near zero, read load is +0.8 watts.
Remarks
Raw bandwidth is approx 2x that of the SATA SSD on my Atom C2338 servers. As it’s NVMe instead of SATA, latency will be orders of magnitude lower too, but milliseconds to microseconds won’t matter much for a web server.
RAM does consume power, and you see it in the idle power consumption above. The Samsung SSD is approx 4x less power efficient during reads and writes than the Biwin SSD which in fairness uses very, very little power in smoke testing. Obviously the amount of time your internet server spends doing sustained reads or writes will generally be minimal, so apart from peak power consumption calculations to ensure you fit inside the max colocation power limit, the idle power consumption will be what is used almost all of the time.
I tried battering the Biwin SSD with sustained writes and yes after a while you start seeing power consumption spikes of about four watts while write performance nose dives. After you leave it for a while it recovers. This suggests that it does have some form of SLC cache to enable fast burst writes at low power consumption. If so, why on earth it doesn’t also implement TRIM is beyond me as a SLC cache implies a block storage emulation layer internally.
Filesystem storage performance
I only run ZFS on my public servers which has been the case for many years now, principally for ZFS’s awesome auto-replication feature whereby ZFS filesystems can be mirrored very efficiently across multiple machines. I have ZFS replication running between the two public servers every few minutes, and from public servers to home and from home to public servers. I therefore have multiple offsite backups of everything down to a few minutes of lag, which I learned after the year 2020 two week outage is very wise.
The slow Intel Atom C2338 with LZ4 compressed unencrypted ZFS was surprisingly okay at about 348 Mb/sec reads and 80 Mb/sec writes. It did rather less well with encrypted storage at 43 Mb/sec read and 35 Mb/sec write, this is because the ZFS in kernel 5.4 wasn’t able to use the Intel AES-NI hardware acceleration functions so everything is done in software.
Ubuntu 24.04 LTS comes with kernel 6.8, and if I was on Intel its ZFS would now use AES-NI hardware acceleration. Unfortunately, as https://github.com/openzfs/zfs/issues/12171 which adds AArch64 hardware crypto acceleration to ZFS is unmerged, we still fall back to software cryptography. LZ4 compressed unencrypted ZFS reads at about 448 Mb/sec and writes at 144 Mb/sec – a 28% and 80% performance improvement - but encrypted reads are 84 Mb/sec and encrypted writes are 51 Mb/sec which whilst still a nice improvement, they are not the ~500 Mb/sec rates which that ZFS patch if merged would produce.
Still, it is only the email storage and a few other bits which use encrypted ZFS. Most of the system uses unencrypted LZ4 compressed ZFS and it uses highly compressible data, not the random bytes I used for the testing. So in practice you’ll get much better performance than the above.
Indeed, this RPI can saturate its 1 Gbit NIC from the ZFS filesystem with ease,
something the Intel Atom C2338 wasn’t quite strong enough to do.
I suspect kernel sendfile() improvements are also at work as the
Intel Atom C2338 + ZFS could only push 59 Mb/sec down its NIC if
the file content had to be read from the SSD or 80 Mb/sec if the file
content was in ZFS RAM cache. A quick google search just there seems
to confirm that ZFS copy_file_range() support was merged in Ubuntu 24.04,
so kernel zero copy byte range splice support is indeed just fresh
out of the oven as it were.
But aren’t we on a 100 Mbit capped public internet connection, so does it matter if the board can saturate 1 Gbit?
FinalTek network performance
I performed these benchmarks at 1am BST 2am CET to ensure a relatively
quiet public internet. Firstly, between adjacent Pi servers I get
936 Mbit, which has since been confirmed many times by scp that yes
the switch is 1 Gbit as promised.
The sixth generation server has a ‘up to 1Gbit’ NIC. It is located in Amsterdam, whereas the seven gen is in the Czech Republic. There is 22 milliseconds of latency between them which is surprisingly high given the distance between them is under 1000 km (approx 5 ms latency). It turns out that traffic routes via Vodafone’s backhaul who are a British company, so I suspect traffic takes an elongated route far from a geographical straight line. That said, I measured 96.6 Mbit from dedi6 to dedi7, and 887 Mbit from dedi7 to dedi6.
Yes, that did make me think I had made a mistake too. So I ran the same test from my site’s fibre internet, which has a 1 Gbit downstream and 100 Mbit upstream. There is a 38 millisecond latency between my site and FinalTek, with 61 Mbit from site to dedi7, and 226 Mbit from dedi7 to site. Here is how traffic is routed:
traceroute -4 dedi7.nedprod.com
traceroute to dedi7.nedprod.com (46.167.244.57), 30 hops max, 46 byte packets
1 217-183-226-1-dynamic.agg3.mlw.lmk-mlw.eircom.net (217.183.226.1) 2.395 ms 3.295 ms 0.691 ms
2 eth-trunk137.hcore1.lmk.core.eircom.net (86.43.253.148) 9.729 ms 5.629 ms 4.578 ms
3 eth-trunk13.hcore1.dbn.core.eircom.net (159.134.123.9) 13.517 ms 10.842 ms 8.655 ms
4 * * *
5 dln-b3-link.ip.twelve99.net (62.115.32.200) 9.360 ms 7.846 ms 8.852 ms
6 vodafonepeering-ic-356964.ip.twelve99-cust.net (213.248.98.129) 8.622 ms 9.161 ms 7.572 ms
7 ae25-xcr1.ltw.cw.net (195.2.3.129) 38.989 ms 37.176 ms 37.839 ms
8 ae37-pcr1.fnt.cw.net (195.2.2.74) 32.413 ms 29.676 ms 30.388 ms
9 ae4-ucr1.czs.cw.net (195.2.10.233) 36.986 ms 36.364 ms 36.577 ms
10 vodafonecz-gw3.czs.cw.net (195.2.12.42) 38.170 ms 39.950 ms 41.302 ms
11 ip-81-27-200-59.net.vodafone.cz (81.27.200.59) 37.554 ms 36.569 ms 44.476 ms
12 tachyon.finaltek.net (77.48.106.250) 40.717 ms 38.654 ms 39.420 ms
13 europe7a.nedproductions.biz (46.167.244.57) 40.171 ms 39.042 ms 39.527 ms
(cw.net is Cable & Wireless, the internet backhaul subsiduary of Vodafone.
Between Eircom and them is twelve99 which used to be the familiar TeliaSonera
of yore, now called Arelion who are Swedish)
I did a bit more testing, and it looks like Finaltek only throttle inbound to 100 Mbit, not outbound. That means the public can download from my website at up to 1 Gbit. This was quite unexpected for the monthly cost – I had assumed a straight 100 Mbit throttle per MAC address with maybe a minimum bandwidth guarantee of 5 Mbit, like you might get at OVH et al. Apparently not.
Obviously FinalTek have promised nothing other than ‘up to 100 Mbit shared’ so network performance can absolutely worsen in the future and they’re still within claims. But consider me very impressed for the money!
I run CloudFlare on front of the website in any case, so it doesn’t actually matter if the NIC is faster than claimed except when CF is fetching uncached content, which is for two thirds of all requests according to its dashboard (I am on the free tier). I have no idea if there is any DDoS protection from FinalTek – I would assume not, but again CloudFlare should take of that too at least for the web site part of things.
Final words
I’m not sure what more there is to say other than that I am very pleased with the considerable performance improvements in the seventh generation of my server infrastructure. It should be good for at least seven years once I’m fully migrated over, then I’ll be releasing the sixth generation from rental (I have already released all the VPSs I used to rent to provide redundant DNS now that CloudFlare and Hurricane Electric provide redundant DNS for free with far lower latencies than any VPS could). Despite the nearly five hundred euro outlay on new hardware, over four years it will be cheaper than the present rental costs and over seven years it should be far cheaper. Let’s hope that the hardware is reliable and trouble free during that time!
Re: house build, that is still stuck in joist design as it has been since November last year. I don’t know when it’ll get free of that, it isn’t for a lack of me nagging people. My next post might be on the eighteen solar panels I’ve had installed on the site since Oct 2023, as I now have two winters of measurements to analyse as the inverter captures lots of data every ten minutes or so, so I have a large time series database now. I think it worth a post here trying to figure out what I might be able to infer from all that data.
Until the next time, be happy!
| Go to previous entry | Go to next entry | Go back to the archive index | Go back to the latest entries |