Sunday 1 June 2025: 21:12. Last post when I said what my future posts will be I forget to mention this post which is the annual update to my periodic comparison of storage bytes per inflation adjusted dollar for magnetic hard drives, flash SSDs, and Intel Optane XPoint devices (you can find all the past posts here), which I have done every June since 2012:

Raw data: http://www.nedprod.com/studystuff/SSDsVsHardDrives.xlsx

You will note that I have stopped collecting prices for Optane drives – historical inventory for them has been sold, and you can’t really get them new anymore. Nor is there any close substitute. I’ll keep the historical data on the graph for now in case somebody does launch new storage with affordably priced non-volatile memory based storage. It’s a real shame how Optane went, had Intel not messed it up so badly, it should have replaced flash for storage.

This year is a normal looking year – spinning rust storage has become quite a bit cheaper, while flash storage has become quite a bit more expensive. It is, however, anything but a normal year because this year’s prices include the effects of the 10% import tariff currently being applied within the United States.

I ummed and awed about whether to try removing the effects of those taxes to keep this year’s figures comparable to previous years. But then this graph is in inflation adjusted US dollars, so we are by definition heavily tied into US economic specifics in any case. And they now have an import tariff making electronics more expensive. So I decided to not attempt to adjust anything. This graph is for the US economy and everything about that economy in each year.

It turns out SSDs are almost exactly 9% more expensive than last year. If you include inflation, it is 7.5% in real terms. Not quite the 10% tariff tax, but undoubtedly a fair price increase.

Hard drives, on the other hand are 28% cheaper including the tariff and excluding inflation. If you include those, they’re more like 40% cheaper than last year, which is getting towards half price. That’s an enormous price decrease in a single year, and most of it I think can be explained by a huge slump in demand for hard drives.

This time two years ago I predicted a recession would cause storage prices to tumble. Here looks like that recession, but so far it hasn’t appeared in the wider US economy, though it has in the wider European and Asian economies.

As regular readers here will remember, I recently picked up a factory recertified 28Tb hard drive recently for a dedicated AI inferencing machine for the site’s security cameras. I got that drive delivered for €400. That might seem a lot, but minus sales taxes (23%) and delivery (maybe €30) that 28Tb enterprise hard drive cost about €300. That same drive cost nearer a grand after taxes this time last year. It’s madness just how much hard drive prices have fallen in a single year. I can’t remember anything like it in recent memory.

#ssdsvsharddrives

Friday 30 May 2025: 20:52. I am now officially unemployed for the first time in many many years! My last full time day with Category Labs was today. As described last post, I shall be on retainer with them until Autumn, which equals a summer of freedom for the first time in a very long while.

During the past month that I’ve known a period of unemployment was coming, I was very busy planning what I’d be doing, getting my affairs and finances in order, and wrapping things up at work so there would be a good hand off. I was very much looking forward to being free of this two metre squared space I have been tethered to for so many years now. However, now unemployment is actually here … I must admit feeling a certain bittersweetness.

At the time, I worked very hard to get into Monad, more than perhaps any startup I’ve ever campaigned to get hired into. I was very keen on the proposition (a better implemented Ethereum), I thought the founders were the right mix to make it happen and therefore a good bet, and I wasn’t unaware that my participation in this project if successful would slightly change the future of all mankind, so my work there would have genuine true real long lived impact.

I had hoped to be contributing until mainnet launch, but after the Series A round the founders decided to localise my part of the workforce into New York and they now had the means to do so. I interviewed my replacements, and I was well aware that I was likely putting myself out of a job at the time. They’re the people which will take this to the end.

As I’ve been part of half a dozen startups by now and therefore more experienced than most startup founders, I’ve often said to startup founders that the people you need to bring you from tiny to small are often not the people you need to bring you from small to big. In my previous roles in startups, I tended to be retained across that transition as many others were let go. This time round, it was not the case, I ended up being one of those surplus to requirements. I am grateful that they kept me on as long as they did, and that they have retained me until the Autumn to hopefully see this thing through from the coalface.

I wish Monad/Category Labs the very best of luck with their mainnet launch, and my genuine gratitude to their founders for taking a chance on me – who was their first international remote hire of many since. I hope I was useful as I pitched to them at the time that I would be.

---

For the past eighteen months there has been headcount reduction in my industry, and in the past six months that rate has been increasing. When I go on LinkedIn right now, I see many high end devs I know personally who are unemployed, and they are not finding new work at whatever pay rates they’ve been willing to currently accept. A few weeks ago I began noticing lay offs even in AI, which until now has been insulated from the general tech industry recession.

I certainly think another year of tech industry downsizing remains. They over-hired during and after the pandemic, and the revenues just aren’t there to support that much workforce at those pandemic elevated pay rates. They therefore will keep downsizing until average pay levels get low enough that it makes business sense. We have a while to go on that I think.

Tech tends to precede the wider economy, so tech gets a recession first and the wider economy follows after. This has been the longest delay between tech recession and everybody else recession by far in my lifetime, but only from the perspective of the Anglo-Saxon countries. In Central Europe, the everybody else recession followed the tech recession within the traditional delay. What’s different this time is that the everybody else recession has not begun yet in the US, and therefore not in Ireland (much of whose economy is US multinationals).

I think much of why has to do with the US running such a massive deficit – they are borrowing at wartime levels to juice their economy and have been doing so for so many years now that the situation has become detached from normal economic fundamentals. Historically that will turn, eventually, into an especially deep economic collapse along the lines of Argentina which was once one of the richest countries in the world.

I’ve already talked about all that here before, and I don’t have much new to say about it. At some point the debt will need to be paid. Between now and then, it’s entirely possible that the US can carry on borrowing 8-10% of GDP per year for several more years to come. So detachment from economic fundamentals may continue for a while yet, indeed in the Anglo-Saxon countries the tech industry may exit recession into some sort of very weak recovery before the general economy enters recession.

The Irish economy may contain an unusual proportion relative to its size of the US economy, but it is more strongly tied to the European economy than the US economy overall. There are big market shifts happening in Europe. We’re about to build an absolute ton load of weapons all of which will require lots and lots of system engineers like me. The sixth generation European strike fighter is very noticeably completely independent from the US. It likely will be developed by a consortium of the major European powers and Japan and Australia (Canada currently wants to join after US threats upon its sovereignty, but I don’t think that will survive long term after the current US administration exits – after all, such a project has a half century long time horizon). Like the competing US, Chinese and Russian sixth generation strike fighters, it will be more an nVidia CUDA based supercomputer than airplane. As the programming languages well supported by CUDA is very limited (and does not currently include Rust), that could turn into stable long term employment for people like me who specialise in C, C++ and Python.

As this is Europe, pay rates will be a fraction of those elsewhere. But stable – or any – income is better than none. So I think Europe may be an exception to the wider pan-tech industry malaise if you’re a systems engineer. Ireland not being in NATO nor having any security clearances will mean Irish citizens won’t be allowed to work on anything secret, so we’ll likely be given dual use technologies like drones, resilient communications, and perhaps core stack stuff like deterministic runloops and work dispatchers. Anything secret will be reserved for citizens of the participating countries. I guess we’ll find out.

I’ve found myself pondering what do I want in my next employment? I do genuinely actually find AI appealing, and no I’m not lying nor faking that sentiment. I think AI will not solve 95% of the things everybody is trying to use it to solve. But that other 5% … I think it could be transformative to my personal productivity, and possibly to enough others to matter overall to an economy wide picture.

However, I must be realistic: I am forty-seven years old, and almost all of AI does not make direct use of the low level skillsets I specialise in. I am therefore unlikely to be a good fit for getting into AI at my age and skillset. Also – to be blunt – I think the AI balloon is beginning to deflate in any case anyway, and then it’ll go through a period of being out of fashion until it comes back into fashion again. Such is the tech economy.

Having pondered on this for the past month … why on earth hasn’t IoT become an expanding bubble yet? I, like many others, feel that IoT ought to be an exponential growth opportunity like AI has just been at some point. But we never seem to reach that point. Rather, IoT seems to just constantly keep incrementally creeping forwards and inwards subtly and never experiencing a ‘break out’ moment as it infests ever more parts of the global economy and human existence in general. Maybe those are its dynamics, who knows?

I guess what I’m really asking here is ‘where will the tech industry go next?’ i.e. what inflating balloon do I need to hitch myself onto before it rises? I have been relatively successful and/or fortunate in predicting/happening to be right place right time so far. I need to repeat that magic. I will admit that at the time of writing, apart from European weapons systems, I can’t think of anything likely to be the ‘next big thing’. Here’s hoping some clarity arrives on me in the next few months.

What’s next?

One of my long time unfinished projects has been a dedicated AI inferencing security camera PC. I combined a fifteen year old case with a ten year old PC guts and a ten year old second hand enterprise nVidia AI inferencing card plus a 28 Tb recertified hard drive to build a top of line security camera monitoring and recording box. This box has been built, debugged, and tested. I just need to write it up. There will surely be a post about that here soon.

Believe it or not sixteen months after I shook hands with my replacement builder a first draft of construction detail has turned up. Yes of course it would appear now when I’ve lost my job. In any case, there is now a reasonable expectation that a final quote could be issued some time in the next six months. I almost certainly cannot take a decision on build vs no build until the Autumn, but if we could get that final quote before then … that would be great as there would be actual forward progress at very long last. There will surely be a post here about the first draft of construction detail etc at some point.

The differential pressure sensors I mentioned some posts ago have turned up and those will need testing. I have also acquired an anemometer so I can finally say what air flow is passing for given readings of those barometric sensors. That will likely also result in a post here.

Finally, the next post will be all about cargo e-bikes, as I already have written that post due to buying and testing a cargo e-bike. All will be explained in that post!

#unemployment

Saturday 17 May 2025: 22:36. It’s been a month since my last post about my house build and there is big news. No, not of a ‘yay they’re going to start building my house next week variety’, but rather of a ‘I probably will have to hit pause on building this house’ variety unfortunately.

At the end of last month I was given notice of early contract termination from my current client, so I am currently working out my final month of full time with them. Without a regular income, no mortgage provider will lend. And without a mortgage, I cannot complete this house build. Indeed, as self employed people get no unemployment welfare benefits in Ireland, we’ll be living entirely off savings aka house building fund. So it would be imprudent to begin building a house in any case until more money is found.

As much as this sucks, this is not a terrible time for this to have happened. Had we been just before first fix drawdown, we would have been in deep doo doo because we’d be out of cash to proceed the build AND out of cash to stay alive, and the banks here are known to not issue further drawdowns until you restore regular income. That would be the single worst time possible to lose regular income, and losing it now is very far from the worst time possible.

All that said, it is yet another delay one actually caused by me this time and I’m mindful that planning permission will expire in 2027. We need to get everything to wall plate level by then, or we are truly hosed.

My field of work has been in recession for about eighteen months now. It has been a constant story of headcount reductions throughout my industry with very little hiring especially for fully remote workers like me. I do not expect to gain new employment at an income level sufficient to get the mortgage I need (Ireland caps maximum mortgage to 3.5x annual income) in the next twelve months given the state of things. So there will be a period of unemployment – normally a welcome thing for me, especially given I have been working for eight years straight without a break which is unusual. But in the context of building a house, it isn’t helpful.

Given the near term unlikeliness of finding new fully remote work sufficiently well paid, I have come to an arrangement with my current client that I should remain available to them at short notice until Autumn which means I am not available for new full time employment until then. This guarantees that I will have the summer to clear a large number of backlogs which have built up during the past eight years of working without breaks. This post will be listing that priority queue of stuff to get done now I finally have the time to get them done.

1. Lose weight

Sitting in front of a computer all day long tends to add body weight especially as you age. I had been 65 kg or so before I went to St. Andrews, and I was 72 kg on exit due to too many fine dinners (I mean that quite literally – I came into some inheritance in my final year there, and much of it went on some very excellent restaurants with intelligent and interesting female companions, a very worthwhile use of that money). Working for BlackBerry in Canada took me to my heaviest ever, 86 kg and I found it hard to shift on return to Ireland in 2013. That led to blood pressure related health issues, so I tried all the usual things such as diets, more exercise etc all to no avail over the next few years.

I had to take a year off earning after the birth of my son in 2016 as my wife needed me to be home. This gave me the opportunity to properly focus on losing weight which now succeeded given I had the time to do it, going from 86 kg to 81 kg by Feb 2017, then to 78 kg by April 2017, and then to 74 kg by Sept 2017. I got it as low as 73 kg, but I never reached the 72 kg I had leaving St. Andrews. In any case, all my blood pressure related issues disappeared, and I was much better off generally at the lower weight e.g. climbing stairs no longer causes you to puff.

I did a good job of keeping the weight off after that not rising above 76 kg until covid, when it piled on again not helped by being laid up in bed by covid for four months. I have failed to shift that weight since 2022, and I’ve been between 80 and 83 kg for the last three years. Blood pressure related issues have been oscillating in the background reminding me it’s not wise to be this heavy. I need to get back towards 75 kg if I am to remain healthy long term.

Now I have the free time to properly commit to it, I intend to lose as much weight as I’ll be allowed to. It greatly helps if I can (a) sleep as much as my body needs so I’m not tired (b) not have to keep sugar levels up throughout the day so I can concentrate and (c) I have the time to get in far more hours of daily exercise instead of the quick daily 4000 steps I might grab around the park if the weather permits when I am working full time.

Given I’m a good bit older than last time, 73 kg is probably unrealistic as my metabolism will have slowed. But we’ll see how it goes. I’ve got another two weeks of paid work left, then I intend to work hard on weight loss.

2. Do stuff with the kids I couldn’t normally do

I’ll be on morning childcare this summer as usual, but normally I can’t go too far or do too much as I’d need to be back home to start work. Well, not for this summer!

They’ve got camps throughout the summer for varying weeks, so there is actually only one week where I have all three at once and for that I think I’ll take them to London during the weekdays. I’ve often said that we ought to. Now I can. So we shall.

There is only one other week where I have the elder two only so I think I’ll take them to Amsterdam that week. We can do older stuff that Julia is too young for which interests the elder two. We might also meet up with their Belgian godparents. Why not!

(The reason ‘why Amsterdam and London?’ is because those two are the only two with affordable direct flights from Cork airport. There are direct flights elsewhere, but they are not cheap. Actually, I lie – there is also a cheap direct flight to Milan, and I keep meaning to go but I’ve never been)

Those will be the standout weeks, but I expect now that I don’t have to be home by a deadline that we’ll be taking a lot more day trips and seeing how the day goes. Ireland is a tourism superpower, very few places on earth have the consistent density of tourist stuff that Ireland does. We’ve not exhausted what’s within an hour’s drive from us in my lifetime, despite our best efforts these past ten years. Ireland is very, very, very good at tourist stuff. If I can even drive ninety minutes away, it opens up a bonanza of more stuff to see and to do. I intend to make use of our very fortunate country of residence.

3. Move ISO standards committees

This June’s WG21 standards meeting will be my last. I gave it seven years of my life during which I achieved absolutely nothing. Not one of my proposals got anywhere. As I’ve mentioned elsewhere, I think that more on the C++ standards committee than on me. They have become very detached from reality recently, and it doesn’t help how much industry funding for C++ has simultaneously shrivelled either.

I’ll be taking my family with me to that June meeting, which is in Sofia, capital of Bulgaria. I’ll say goodbye to all the many excellent people I got to know during my years of service there. I’ll be moving on to the WG14 standards committee whose next meeting is in Brno, Czechia in August. There I expect to present three major feature proposal papers. Let’s hope I am far more effective at the C standards committee than I ever was at the C++ standards committee.

Now I’m free of work contracts restraining my time away from work, I intend to resume presenting and teaching at conferences. I expect to submit a talk proposal for ACCU in Bristol for April 2026. We’ll see how it goes.

4. Clear project backlogs

I only have nine weeks of summer left after all the above! But I very much intend to clear all the partially or nearly finished prototypes and projects for the house.

The highest priority of those is the 3D services layout plan. I’ve got ventilation and plumbing done. I have the wiring for one bedroom done. TBH, it’s very tedious work, I keep finding excuses to not work on it. But getting it done would save very significant time later on, so it’s highly value adding work. I just need to make myself get it done. Once it is done – and especially because it’ll be an accomplishment to have gotten it done – I don’t doubt there will be a post here about it as a show and tell with pictures.

There are several other projects I didn’t fully finish – the blind automation was not fully debugged, as an example. Yes it works, but it had quirks. Those need to be debugged and fixed. Ideally a blind should be put into long term testing too, make sure it’s truly bug free over months of testing. If unemployment continues to stretch out, I can start work on ESPHome assemblies e.g. the automation for the thermal store, or the automation for the sauna or the underfloor heating. There is plenty of work I can do now so I don’t have to do it after the frames are up.

Anything else?

There is a temptation to try more, do more, begin more now I have the freedom to do it. But I must be realistic – I will have to probably return to full time work in the Autumn. Starting anything new – as attractive as that is – doesn’t make sense as a result.

Maybe by the Autumn my builder might actually want to build me a house by then more than eighteen months since we shook hands on it? Who knows. If so, everything will hang on the quote he gives me. If I can afford to get that house watertight and still leave a sufficient cushion for us to live off in case my unemployment stretches on, I’ll probably go for it. If his quote would make that cushion too thin, I’ll just have to hit pause for now and hope sufficient new income turns up soon to reenable the mortgage application.

Things could be better. But things could be very considerably worse too. I cannot complain.

#house

Friday 25 April 2025: 20:47. Back to posts about my house build! A few weekends ago I was tidying stuff up and I found the project for testing barometric pressure sensors with the ventilation boost fan. The BMP390 pressure sensor had arrived from Aliexpress months ago, and I’d even gotten round to soldering pins onto it. I just hadn’t gotten around to completing the testing begun last October.

It was very little remaining effort, so I have now done so. To quickly recap, a proper air velocity sensor costs €55 inc VAT delivered each, so I had been wondering if I could misuse a cheaper sensor to estimate approximate air flow instead. The obvious candidate is a pressure sensor, as static pressure roughly correlates with air velocity squared. To that end, in that last post, I sellotaped a cheap BMP280 temperature + pressure + humidity sensor onto the boost fan, and took some measurements for various fan speeds.

I found that 50% and 100% forward fan speed turned into a -5 and -22 Pa pressure difference, and that 50% and 100% backwards fan speed turned into a +12 and +16 Pa pressure difference. As perhaps already obvious, the problem was the amount of noise in the readings – the BME280 is only specced for a +/- 12 Pa relative accuracy, and I found the sensor saw a 6 Pa steady state pressure reading difference just by leaving it for a few minutes – all within spec, but not useful for the air speeds I want to measure.

The BMP390 sensor is specced to have a relative accuracy of +/- 3 Pa, four times better. It costs approximately 10x more than the BMP280 sensor, but that is still only €4 inc VAT delivered. To recap the sensor spec table from that last post:

	Noise	Relative accuracy	Absolute accuracy	Cost incl delivery
BMP280	3 Pa	+/- 12 Pa	+/- 100 Pa	€0.56
BMP390	2 Pa	+/- 3 Pa	+/- 50 Pa	€4
BMP581	0.1 Pa	+/- 6 Pa	+/- 50 Pa	€53

This time I sellotaped both the BME280 and BMP390 sensors to the fan, and I once again ran the fan at 50% and 100% speeds both backwards and forwards, turning it off in between to establish how much drift there might be:

	Difference 100% backwards (percent)	Difference 50% backwards (percent)	Difference stopped to stopped (percent)	Difference 50% forwards (percent)	Difference 100% forwards (percent)
BMP280 Pressure	+38.23 Pa (+0.0384%)	+8.686 Pa (+0.0087%)	+2.469 Pa (+0.0025%)	-2.856 Pa (-0.0029%)	-18.56 Pa (-0.0186%)
BMP390 Pressure	+18.66 Pa (+0.0187%)	+4.2302 Pa (+0.0043%)	+1.036 Pa (+0.001%)	-3.082 Pa (-0.0031%)	-19.73 Pa (-0.0198%)
Sqrt pressure (static pressure ~= square of velocity)	+4.24	+2.02	0	-2.02	-4.24

Both this test and the testing last October found that the BMP280 sensor has a strong bias to air direction. The BMP390 does not. In fact, I’d say that the BMP390 is looking very good so far, only a little drift between stop-forwards-backwards-stop.

How stable is it over time? I took a reading per second for one minute, and over that time I found:

	BMP280 Humidity	BMP280 Temperature	BMP390 Temperature	BMP280 Pressure	BMP390 Pressure
Maximum difference between readings	0.29 %RH	0.036 C	0.038 C	8.057 Pa	3.833 Pa
As percent of mean	0.503%	0.183%	0.182%	0.0081%	0.0038%

From its datasheet, the BMP280 could be up to +/- 12 Pa out per reading and the BMP390 could be up to +/- 3 Pa out per reading as worst cases (we assume that the oversampling we configured removes the noise). Which is, theoretically, 24 Pa and 6 Pa max-min differences respectively. For our one minute of samples, we are 33% and 64% of worst case for the BMP280 and BMP390, so within spec.

The next obvious thing to do was to calculate a 99% confidence interval, but it turns out that (surprisingly) the variance of the values read is not Gaussian i.e. not normally distributed. They are in fact linearly distributed for the BMP390 and have a power law distribution for the BMP280:

That is an unusual distribution. It made me wonder if it’s a random walk of first order difference:

Which it is. It would appear that for any given oversampled reading from the BMP390, the reading will be a linearly distributed random +/- 2 Pa from the true value. That should be easily removed with a basic low pass filter, perhaps with a ten sample history.

I also think I misread the spec sheet – when it said +/- 2 Pa due to noise, I had assumed that the oversampling would filter it out. It would appear that they really did mean +/- 2 Pa.

Remind me why you’re not using a differential pressure sensor again?

Barometric pressure sensors are absolutely the wrong thing to use to detect air flow. What I should use (as the October post pointed out) is a hot wire mass air flow sensor, which I can’t as it is too expensive on power and won’t work in varying temperature air anyway. Failing that, I should use a differential not barometric pressure sensor.

I mentioned in the October post that a differential pressure sensor costs about €45 inc VAT. That was for a CFSensor XGZP6897D with the I2C interface and 3.3v power supply. Much to my surprise, that exact same sensor can now be had for €9.50 inc VAT delivered only six months later (tariffs? Potential slowdown?). That brings it into the realm of possibility, so I ordered some along with solderable breadboards to attach them to – I guess that makes them actually €10.44 inc VAT each delivered, more than twice the cost of the BMP390 sensor.

The XGZP6897D (and note the ’D’ at the end, the ‘A’ variant is considerably cheaper but emits an analogue signal) can be purchased with varying pressure ranges. I chose the smallest available, which measures between -500 Pa and +500 Pa, and for which its datasheet claims an accuracy of +/- 2.5% with a potential +/- 0.5% drift over months. This is +/- 12.5 Pa, which is no better than the BMP280.

I honestly won’t know until testing – I suspect fitting one BMP390 barometric pressure sensor on the exit of the fan will be the ideal bang for the buck solution. The poor absolute accuracy can be solved by stopping the ventilation completely every few hours, and figuring out the absolute pressure offset for each sensor. Thereafter relative accuracy should be quite good, we should be able to reverse the fan to reduce the airflow to within 40% of max fan speed at 12v (which I have no idea what that is, but to my hand in front of it it is a slow breeze, which I think will be ‘good enough’).

In any case, I was going to need a differential pressure sensor anyway to ensure that the MVHR outlet and inlet don’t exceed a 200 Pa difference, which the MVHR documentation says will otherwise cause excess wear and tear. As we adjust boost fans some to increase flow in some places and reduce it in others, the back pressure to the MVHR will need to be kept balanced.

I still think something good enough for cheap to detect air flow is doable here. Worst comes to worst I can just fit the expensive FS3000 sensor and measure the air velocity directly. It is only accurate to 5%, however because static pressure relates to air velocity squared, our pressure sensors will never do well with low air velocities. A proper air flow sensor should be able to get the net air flow much lower.

Finally, now I’ve bought ten of the differential pressure sensors to amortise shipping costs down, I’m finding myself thinking of several other possible use cases. One is to ensure that the house is kept at neither positive nor negative pressure between the inside and the outside. For normal timber frame (and arguably block) houses, you would want a slight negative pressure to prevent indoor humidity penetrating your building fabric when outdoor RH is less than indoor RH, and a slight positive pressure if indoor RH is less than outdoor RH. However, thanks to the superb air tightness of Passive House, one can usually run a constant slight positive pressure safely. I’ll likely run my system to be neutral with a slight bias towards positive pressure, so only very slightly off neutral. Maybe 2.5 Pa, something close to the error bar interval for the sensor.

There are other potential use cases – within a building there is a stack effect from hot air rising above cold air. For every meter of altitude, there is 11.65 Pa of pressure change at 20 C. If the pressure change is more or less than that, it is probably stack effect.

One thing this particular sensor model can’t do is measure water tank depth unfortunately – 0.1 metres of water exerts 1000 Pa of pressure amazingly enough. So these sensors would exceed their range after 5 cm of water!

#house

Thursday 24 April 2025: 21:50. Yet another not a house post this time, but in fact I originally wrote one very long post which was about three completely independent topics. So it made more sense to break them up into individual days. This one is about fibre broadband in Ireland, and the new fibre broadband just installed into my rented house.

A few months ago, our vDSL connection disappeared and when the man came out to fix it, he said the line had gone so he switched us over to another line. This new line synced at about 30 Mbps rather than the 92 Mbps we got before, and kept dropping out for twenty minutes or more at a time. This was similar to when we first had vDSL put in – I had to have the repair guy out many times until one of them finally came up with a stable vDSL solution. That had lasted since then until now.

Thankfully Ireland now has better options! You can now get fibre installed into the premises for free if you sign up to a twelve month contract. A man appeared about three weeks ago, he installed an additional cable from the pole into the current rented house, and voilà I now have a direct fibre internet connection to the world.

In my post about fibre to the premises in Ireland, it turns out I incorrectly stated that Ireland uses multi-mode fibre for the last mile. This is true in other countries, but I have since learned that Ireland went with single-mode fibre for its GPON (Gigabit capable Passive Optical Network) last mile solution throughout as we tend to have more sparsely populated urban settlements and it was easier to deploy one unified solution. Thus, in Ireland, every premises gets its own dedicated fibre pair from the ONT (Optical Network Terminal) within the premises back to the cabinet to your own dedicated port on the OLT (Optical Line Termination) card. That card is very similar to a managed switch, and it shares an upstream fibre connection between up to 128 ports i.e. premises, though it’s usually less than that (supposedly, always less than 64 in Ireland). You might then get a 2.5 Gbps uplink from the OLT to the backhaul carrier, which therefore has up to 64 premises sharing it.

The physical fibre backhaul in Ireland for historical reasons has three independent implementations: (i) the ducts of the historical telephone network, which tends to follow major roads and motorways (Open Eir) (ii) the ducts of the power supply network, which connects mobile phone towers (SIRO) (iii) where the government has spent public money on installing fibre where it is commercially not viable to otherwise do so (NBI). To make things a bit more confusing, the owners of the physical fibre are not always the operators nor maintainers of that fibre, and there has been an ongoing government push to consolidate everything under a single one stop shop – however, for now, it all remains a patchwork and there are separate OLTs for fibre from the historical telephone network (Open Eir) and for fibre from the power supply network (SIRO). So, to be specific, if I order fibre broadband from an ISP which uses Open Eir for the backhaul and when that contract ends I order fibre broadband from an ISP which uses SIRO for the backhaul, I get two ONTs with two fibre cables installed into my house. They hope to fix that by year 2030, as that’s obviously daft, but it will require new laws to be passed so there it is.

SIRO due to its physical trunking along major power cables tends to mainly appear only in urban centres, so for both my rented house and my future house being both rural the only backhaul fibre connectivity available is Eircom’s fibre network which for rural Ireland is usually the nearest ‘N-road’ which is the type which connects major towns, which eventually feed into the motorway system which ultimately ends up arriving into Dublin. In other words, if you follow the roads, you can say how the data flows which makes things a bit easier.

What happens next is a bit more complicated. The physical fibre moves the bits of data, and as far as I can tell there appear to be several Layer 3 packet switching network implementations some of which share the same duct, but may use physically separated cables or even separated fibres within the same cable. As of Q3 2024, 1.3 million premises were connected to the Eir fibre network; 609,000 to SIRO’s fibre network and 300,000 to NBI’s fibre network. There isn’t much public information about how all this hangs together, so I apologise in advance if this post also contains mistakes – I gleaned most of this off social media where network engineers say how they think things work (which isn’t always correct either I’ve noticed).

I mention all this detail because the fibre installed to the site uses the Eircom Layer 4 backhaul, whereas my rented house uses the BTIreland Layer 4 backhaul. I read online that ISPs can choose to run PPPoE or IPoE over the backhaul, and ISPs who use multiple wholesale networks tend to choose PPPoE because then they can mostly ignore which wholesale network is the backhaul, and treat the lot as if a single network. What this means, of course, is all traffic must go via Dublin if PPPoE is used, whereas it doesn’t if IPoE is used.

Let’s see this in practice:

From rented house to site:

traceroute to 217.183.227.33 (217.183.227.33), 30 hops max, 46 byte packets
 1  lo1001.bas101.cwt.btireland.net (193.95.131.6)  8.949 ms  10.137 ms  9.245 ms
 2  be137-50.rt101.cwt.btireland.net (193.95.152.1)  7.961 ms  11.220 ms  8.181 ms
 3  be103-100.core201.cwt.btireland.net (193.95.129.239)  8.915 ms  be104-100.core202.cwt.btireland.net (193.95.129.241)  9.419 ms  be103-100.core201.cwt.btireland.net (193.95.129.239)  8.204 ms
 4  be204-100.rt102.cwt.btireland.net (193.95.129.244)  9.345 ms  be203-100.rt102.cwt.btireland.net (193.95.129.242)  12.443 ms  9.375 ms
 5  lag-40.br1.6cr.border.eircom.net (185.6.36.82)  8.567 ms  11.518 ms  8.825 ms
 6  eth-trunk21.hcore1.prp.core.eircom.net (86.43.12.214)  14.236 ms  11.549 ms  20.680 ms
 7  eth-trunk11.hcore1.lmk.core.eircom.net (159.134.123.10)  18.473 ms  17.230 ms  16.650 ms
 8  lag-1-agg3-mlw-hcore1-lmk.agg3.mlw.lmk-mlw.eircom.net (86.43.253.149)  15.396 ms  15.442 ms  18.941 ms
 9  217-183-227-33-dynamic.agg3.mlw.lmk-mlw.eircom.net (217.183.227.33)  17.578 ms  18.452 ms  20.854 ms

From site to rented house:

traceroute to 194.125.122.54 (194.125.122.54), 30 hops max, 46 byte packets
 1  217-183-226-1-dynamic.agg3.mlw.lmk-mlw.eircom.net (217.183.226.1)  2.635 ms  3.926 ms  3.776 ms
 2  eth-trunk122.hcore1.mlw.core.eircom.net (86.43.253.150)  8.553 ms  3.782 ms  3.742 ms
 3  eth-trunk15.hcore1.prp.core.eircom.net (86.43.254.143)  19.218 ms  10.235 ms  10.804 ms
 4  *  *  *
 5  inex1.btireland.net (185.6.36.166)  9.500 ms  8.894 ms  8.884 ms
 6  be103-100.core201.bmt.btireland.net (193.95.129.161)  5.781 ms  9.060 ms  8.759 ms
 7  be304-100.core202.cwt.btireland.net (193.95.129.83)  8.706 ms  be304-100.core201.cwt.btireland.net (193.95.129.85)  9.394 ms  be404-100.core202.cwt.btireland.net (193.95.129.87)  8.888 ms
 8  be104-100.rt101.cwt.btireland.net (193.95.129.240)  10.145 ms  10.076 ms  10.875 ms
 9  po2.bas101.cwt.btireland.net (193.95.152.7)  10.364 ms  9.766 ms  9.866 ms
10  194.125.122.54 (194.125.122.54)  20.493 ms  17.644 ms  17.767 ms

(In case you think I am leaking my IP addresses, by the time you read this they will have been rotated – both are allocated by the ISP via dynamic DHCP and they change every few hours)

Geolocation works for these IPs! For site to rented house:

1. Mallow (Eircom)
2. Limerick (Eircom)
3. Citywest, Dublin (Eircom) – note I think this wrong, see below
4. Dublin Internet Neutral Exchange Association (INEX), which is where a lot of Ireland’s internet traffic gets peered.
5. Saggart, Dublin (BTIreland/Esatnet)
6. Swords, Dublin (BTIreland/Esatnet)
7. (Remaining IPs all geolocate to Dublin thanks to the PPPoE until …)
8. Cork (BTIreland/Esatnet)

I think the geolocation database a bit off though – in the traceroute, prp.core.eircom.net looks like Priory Park Telephone Exchange near Mount Merrion in Dublin, not Citywest; lmk.core.eircom.net looks like Roches Street Telephone Exchange in Limerick; mlw.core.eircom.net looks like some exchange in Mallow. cwt.btireland.net does look like Citywest instead.

This is the problem with my rented house’s ISP using PPPoE. If my rented house were also on Eir for its broadband – and because Eir only use their own backhaul exclusively – traffic between the two sites wouldn’t leave the Mallow switch and round trip ping times would be under three milliseconds. By forcing everything via Dublin, we get eighteen milliseconds instead. In case you are wondering if Eir’s Limerick node will route traffic destined for North America directly without it going via Dublin, unfortunately not – I assume it must go to Dublin to get routed onto whoever provides transatlantic cabling.

You’re probably wondering ‘get to what we care about – how fast is it?’:

The download speed is gated by the new G.hn powerline adapters I installed, which do at least go 3x faster than the Homeplug AV2 ones they replaced, and my testing of those did show they capped out on my rented house’s AC wiring at pretty much exactly that speed. So I’ll never see the full 500 Mbps the connection is supposedly capable of, but 345 Mbps is perfectly respectable especially when we got 92 Mbps under the former vDSL.

BUT there is more to a broadband connection than how it performs when the wind is blowing right. We began to notice slowdowns and disconnections, so I put a ping trace on both the rented home fibre broadband and onto the site’s fibre broadband connection. It pings the outermost node in BTIreland’s or Eir’s network every thirty seconds (i.e. their Dublin termination node), and these are a week of measurements up to last Monday:

So, firstly, yeah the BTIreland backhaul clearly has congestion problems in the evenings, there is a clear repeating spike in latencies in the second half of each day. There are also stability problems where the connection will randomly just hang and it takes a minute or two to renegotiate the PPPoE connection. There doesn’t seem to be a pattern to this – Wednesday on the graph was particularly bad, yet Sunday-Monday has been solid.

I showed these graphs to the ISP’s customer support and they said that because I wasn’t using their router, there would be zero support. It’s what you get for €35 per month I suppose. Eir costs €74 per month so it’s double the cost (albeit for a 1 Gbps connection instead of 500 Mbps).

Eir’s backhaul, in comparison, seems to never experience congestion at all. The one (mild) ping spike on Saturday is because my children turned on the TV at the site, which then synchronised the media library with home which meant the connection ran flat out for a few minutes as lots of data got moved. Even then, ping times went from less than ten milliseconds to less than fourteen milliseconds. Not bad, Eir.

In case you’re wondering if BTIreland’s connection is being presented unfairly here, I can tell you we don’t stream anything over the internet. Yes it’s a more actively used connection than the site connection, but when we max it out is usually in the early hours of the morning where no ping spikes appear. This makes sense – what is throttling the maximum download speed is the powerlink connection, not the fibre connection, and I believe G.hn has a separate priority queue for ICMP.

I tried my best to figure out how the PPPoE implementation is routed to BTIreland in Citywest. I didn’t find out a conclusive answer, so I’m going to assume it is routed over Eir’s fibre and that the traffic congestion issues are within its Citywest data centre. The connection reliability issues may be the Powerline network, they may be the fibre between here and the pole, the cabinet or anything else. Or it could also be a Citywest capacity issue. I can’t really tell without a lot more work, and in the end it is ‘good enough’.

One thing I did learn is that almost all ISPs apart from Eir use PPPoE over fibre in Ireland. This makes sense for them, Ireland is a small network so a single IPv4 range could traverse all wholesale providers as a single network. But PPPoE isn’t without overhead, both in terms of ‘routing stupidity’ but also in CPU and network overhead. IPoE is definitely preferable, and Eir is currently the only ISP in Ireland guaranteeing that.

In fairness to Eir fibre broadband, you do get a nearly static public IPv6 /56 allocation – I say nearly static because I’ve never seen it change yet in a year, but because it’s handed out by DHCPv6 and Eir gave me no guarantees, it could change. The IPv4 address does change frequently though, leases are only one hour long and they appear to change every few days. OpenWRT makes setting up dynamic DNS easy, so the Wireguard VPN between the sites is never out for more than a few minutes.

My ISP using BT Ireland does not support IPv6 at all incidentally. From my experience to date, I think I’ll be sticking with Eir when we move into the new house. This is despite that, from my limited testing, international peering appears to better for BTIreland than for Eir e.g. I get 228 Mbps from New York to rented house whereas just 159 Mbps from the same New York provider to the site. Similarly, my colocated servers in the Czech Republic actually have no peering at all because it is Vodafone/BT networks all the way between home and the colocation.

Still, reliability and consistency trumps peak performance, and on that Eir is well ahead of BTIreland.

#broadband #internet