Wednesday 14th August 2013: 3.59am. VS2010 now compiles proposed Boost.AFIO. It has even been known to, sometimes, pass it its unit tests ... still some work to go there, though there is sadly a showstopping segfault in the VS2010 runtime regarding null exception_ptr returns from std::current_exception(), so we may have look into patching in the greatly fixed VS2010 runtime from VS2012. My congrats to Paul for doing such great work in C preprocessor programming! ...
Saturday 24th August 2013: 5.18am. Boost.AFIO finally goes all green on the CI ... you have no idea how many dozens and dozens of hours of debugging race conditions in other people's code (and then figuring out workarounds in AFIO code) it has taken to reach this point.I am actually quite emotional. Such a huge amount of effort. So many, many bugs and weirdnesses in the older STLs and Boost. ...
Thursday 29th August 2013: 1.15am. Link shared: https://github.com/ned14/FastDirectoryEnumerator So last few days I've been doing some prototyping and experimentation with how to enumerate directories far, far quicker than one usually can as part of implementing the #1 requested feature for proposed Boost.AFIO before GSoC ends, Directory Monitoring Support (http://boostafio.uservoice.com/forums/218980-boost-afio-feature-request/filters/top), and I have discovered many interesting things worth sharing.We all know the common advice to never put too many files into a single directory because " ...
Saturday 7th September 2013: 12.14am. Assuming the CI goes all green, fast, asynchronous directory enumeration for Boost.AFIO is done! I'm feeling pretty wrecked, it's been one of those "burn the candle at both ends while you have the open window of time" sorts of things.The process of finding a new job in North America has taken a turn for the worse: the initial very positive recruitment feedback from the tech multinationals - and mention, in at least one instance, of me being the best candidate seen in some years - has been stopped in its tracks once they realised that there is currently a nine month backlog in US green card processing, and a three month backlog in Canadian Labour Market Opinion processing. ...
Thursday 19th September 2013: 5.43pm. So back to the GSoC grindstone now I have a few days before more interviews. Current problem, the one which kept me up last night, is that the newly released VS2013 provides an unusual mix of variadic templates, a brittle decltype() and a std::result_of<> that isn't SFINAE friendly. The result is that code fails when trying to use AFIO's variadic template overloads because VS2013 tries instantiating std::result_of<> for template types it doesn't like, and instead of disabling the overload as an overload resolution possibility like on any other compiler, VS2013 barfs a compilation error. ...
Tuesday 24th September 2013: 1.14am. So Google Summer of Code 2013 ended today! Boost.AFIO is more or less peer review ready - some additional unit tests to be added mainly to get the code coverage back up, otherwise it's done. Paul will still need some time to finish his directory monitoring feature, but that is fairly orthogonal to the existing code base and its lack of presence doesn't disturb anything. ...
Sunday 29th September 2013: 7.03pm. Bleh I'm feeling rough today. Last night whilst watching TV with Megan I had this crazy idea of how to make proposed Boost.AFIO - yes I know it's supposed to be finished as of last Monday - lock free, even wait free apart from the pesky std::unordered_map<> which would need locks around it. And it sufficiently excited me that I just had to get to work on it right now i. ...
Monday 30th September 2013: 9.10am. Link shared: https://github.com/BoostGSoC/boost.afio/blob/increase_parallelism2/boost/afio/detail/MemoryTransactions.hpp Added a further 15% to AFIO's performance :) I noticed that the standard spinlock was being stupid and was looping compare-exchange which of course sends a ton load of cache line invalidations to all other CPUs. Adding a simple read check to my custom spin lock implementation before the compare-exchange eliminates those cache line invalidations, and voila up goes performance again. My new custom spin lock class is very nice, you compose its behaviour out of templated policy types and it is written using C++ 11 atomics. ...
Tuesday 8th October 2013: 2.37pm. Location: Waterloo, Ontario. Big day today - I should be able to send AFIO into the Boost peer review queue, and hence drawing a line under what has probably been my main excuse against getting a job. It's been a long, long summer indeed! Next week I go to the Googleplex in California for the Google summer of code mentors summit - never been to California before, so that should be interesting. ...
Wednesday 9th October 2013: 4.52am. Looks like AFIO won't enter peer review tonight ... the GCC 4.6 Linux build hangs some of the time on the CI, but I'm dammed if I can replicate the problem on my workstation here. Other than specifically GCC 4.6 on Linux, it's all green across the board, which is great, except for that one remaining obstacle. Grrr ...#boostafio #boostcpp ...
Thursday 10th October 2013: 12.51am. Boost.AFIO has just been submitted for Boost peer review. After so many months of effort, I'd like to thank Megan for putting up with all the hours I spent after work pattering away on it; I'd like to thank Paul for doing such a great job in porting the code to Boost for Google Summer of Code; I'd like to thank Google for sponsoring Paul, and the Boost C++ Libraries for voting for AFIO as a GSoC sponsorship candidate; and finally, I'd like to thank my former colleagues at BlackBerry who pulled a lot of favours internally to get BlackBerry Legal to eventually sign off on me working on this in my free time (first ever in BlackBerry history I believe that permission was granted for an employee to contribute to a non-work related open source library). ...
Wednesday 27th November 2013: 3.52am. Link shared: https://github.com/BoostGSoC/boost.afio/commits/content_hashing_merge Was very pleased today with my fourth attempt at an asynchronous batch hash engine for Boost.AFIO which can schedule digest hashing of arbitrary length texts to all SIMD streams on all CPU cores (with a preference for filling SIMD streams before CPU cores). Getting the scheduler design right was quite tricky, because as individual hash operations end, the tail operations from the end of the scheduled batch need moving downwards to fill holes such that individual SIMD streams in CPUs aren't doing useless work, and this must be done while each CPU core runs entirely independently from the others (a CPU core only breaks processing if one stream runs out of data) so getting the locking semantics correct was non-trivial. ...
Saturday 11th January 2014: 5.37pm. After another day spent optimising proposed Boost.AFIO's build infrastructure and patching in precompiled headers support, I now have total rebuild time including all unit tests and code examples on my Intel Atom 220 down to 5 minutes 8 seconds. That's down from somewhere north of 40 minutes originally, which makes for an eightfold improvement in build times.It's quite amazing how lazy modern CPUs make us actually. ...
Monday 27th January 2014: 3.48am. Link shared: https://ci.nedprod.com/job/Boost.AFIO%20Valgrind%20Linux%20GCC%204.8/143/valgrindResult/pid=31061,0x0/ It's half three in the morning and I spent much of today replacing Boost.AFIO's use of std::packaged_task<> with a custom implementation which is much faster and simplifies the internals, plus removes a chicken-and-egg block on the next feature I intend to add. Unfortunately, there is something wrong with my new custom implementation - basically, on Linux only and on the Jenkins CI only it shows a timing race condition which definitely wasn't there in the old code. ...
Friday 31st January 2014: 2.21am. Link shared: https://github.com/BoostGSoC/boost.afio/tree/completion_handler_refactor2 About two thirds way through my "refactor Boost.AFIO before the baby arrives" sprint - I've half reworked all the completion handlers such that it is all green on the CI for Linux and BSD, yet it hangs on Windows annoyingly. I'm hoping my second half of the rework and the fact I can delete whole sections of boilerplate completely given the new improved structure will fix the problems on Windows. ...
Monday 3rd February 2014: 5.47am. Link shared: http://stackoverflow.com/questions/21520308/is-it-intended-by-the-c-standards-committee-that-in-c11-unordered-map-destro After three days of work, finally nailed the last bug in my third major refactoring of Boost.AFIO's core dispatch engine (which has had its line count reduced by fifty lines, so the implementation is now quite considerably shorter and simpler than before), so now the baby is welcome to turn up whenever it likes! :) This last bug was a real corker only affecting the very most recent C++11 compilers only, not helped by me being completely unable to replicate it here at home on VS2013 which doesn't exhibit the problem (I had my poor netbook soak testing the unit test suite, repeatedly running it for hours and nary a crash nor failure in sight! ...
Wednesday 5th February 2014: 2.51am. The third refactoring of Boost.AFIO's core engine is complete - witness that beautiful sea of green on the continuous integration server! Anyway, that's version 1.2 done for now, onto having a baby next!https://github.com/BoostGSoC/boost.afio#boostafio ...
Friday 28th March 2014: 8.36pm. Link shared: https://travis-ci.org/BoostGSoC13/boost.afio/jobs/21789340 Finally got round to adding a clang 3.4 ThreadSanitize and UndefinedBehaviorSanitize pass per commit to proposed Boost.AFIO. The reason I put it off for so long was the labour of writing the suppressions file to filter out all the false positives generated by the Boost libraries - I really wish the maintainers of those Boost libraries would get round to fixing up their code with _attribute_((no_sanitize_thread)) where needed :( Anyway, tis done now, and if I should ever accidentally introduce racy code in the future, Travis CI will shout at me :)#boostcpp #boostafio #clang #llvm #racecondition ...
Sunday 20th July 2014: 1.20am. Shortly forthcoming Boost v1.56 is the first source code modularised Boost to be released, and the breakages it has imposed on proposed Boost.AFIO meant I had to throw out the old CI and start from scratch. Below is the new CI test matrix dashboard for AFIO, so far I only have build working, testing is still some way off. Even just to reach building everything including docs and PDFs correctly has taken several weeks of after work time, there was an enormous amount of breakage to work around - quite a lot of Boost is broken in subtle ways too. ...
Sunday 3rd August 2014: 1.54am. It's taken four weekends of effort to get that (almost) sea of green for the newly rebuilt CI for the newly modularised proposed Boost.AFIO as the modularisation broke the old CI config so badly I decided I might as well begin afresh with a much more orchestrated and scripted automation. I've learned an enormous amount about modularised Boost, Jenkins and how clang 3.4 simply does not produce actually working C++ executables on ARMv7 - yes, 3. ...
Monday 16th February 2015: 12.01am. Just cleared two weeks of email backlog. About the same time that I have been burning the midnight oil staying up after Megan and Clara go to bed trying to get the next release of proposed Boost.AFIO out the door ... and it is still not there, nor will it be for many more nights :(. I have bought myself a raft of alcohol which is to be my reward after I finish and ship the release - until then, I get to look at it and not touch a drop, maybe it might aid motivation. ...
Tuesday 3rd March 2015: 1.30pm. Woke up after particularly little sleep last night to find a sea of green on the AFIO CI dashboard as you can see below (yes, the number of test targets is very considerably higher now, plus there is now official support for Apple OS X/iOS and Android). Which was a surprise actually. Megan returned to work four weeks ago now, and I was supposed to have shipped AFIO v1. ...
Monday 16th March 2015: 6.57am. Six weeks have now passed with me staying up till 5am each night on top of the day job working on proposed Boost AFIO to finish it to a release quality for the v1.3 release. Probably 200 extra hours invested or so now on what was supposed to be almost ready to go, absolutely crazy. I had originally thought maybe a week or two maximum so get it out the door, I was in fact very wrong. ...
Friday 19th June 2015: 11.31pm. Link shared: http://boost.2283326.n4.nabble.com/next-gen-futures-Lightweight-monad-ready-for-inspection-td4677322.html My lightweight very simple single purpose monad has been sent to boost-dev for community feedback http://boost.2283326.n4.nabble.com/next-gen-futures-Lightweight-monad-ready-for-inspection-td4677322.html.This monad can hold a fixed variant list of empty, a type R, a lightweight error_type or a heavier exception_type at a space cost of max(20, sizeof(R)+4). Features: * Very lightweight on build times and run times up to the point of zero execution cost and just a four byte space overhead. ...
Friday 19th June 2015: 11.31pm. Link shared: http://boost.2283326.n4.nabble.com/next-gen-futures-Lightweight-monad-ready-for-inspection-td4677322.html My lightweight very simple single purpose monad has been sent to boost-dev for community feedback http://boost.2283326.n4.nabble.com/next-gen-futures-Lightweight-monad-ready-for-inspection-td4677322.html.This monad can hold a fixed variant list of empty, a type R, a lightweight error_type or a heavier exception_type at a space cost of max(20, sizeof(R)+4). Features: * Very lightweight on build times and run times up to the point of zero execution cost and just a four byte space overhead. ...
Sunday 28th June 2015: 3.10am. Link shared: https://ci.nedprod.com/view/Boost%20Thread-Expected-Permit/job/Boost.Spinlock%20Test%20Linux%20GCC%204.8/doxygen/classboost_1_1spinlock_1_1lightweight__futures_1_1basic__future.html Some preliminary results from my lightweight non-allocating promise-futures which are heading towards Boost later this month as compared to STL promise-futures:All values are CPU cycles on a i7-3770K @ 3.9Ghz.## clang 3.7:### libstdc++ 4.9 future promise:Simple loop: 761Producer Consumer: 838 Creation and setting: 674 Getting from future: 160 Destruction of future: 4### lightweight future promise:Simple loop: 260Producer Consumer: 378 (2.22x faster) Creation and setting: 281 Getting from future: 51 Destruction of future: 46## GCC 5. ...
Tuesday 30th June 2015: 2.47pm. Link shared: https://github.com/ned14/boost.spinlock/blob/master/Readme.md I have the first set of benchmarks for lightweight non-allocating monadic futures ready which are 98% compliant with the forthcoming ISO Concurrency TS for C++1z (essentially they are missing std::allocator support for obvious reasons). They aren't final, but they are representative of what to expect as compared to the STL future-promise. Note these futures are configured with a superset of the facilities of STL future-promise, so you have all your monadic programming goodness in there as a lightweight future inherits from basic_monad plus you can transport error_code as well as exception_ptr:clang 3. ...
Wednesday 8th July 2015: 12.42pm. As lightweight C++ futures have matured it became painfully obvious that a ground up refactor of its design was required. The code base until very recently employed lots of "safe" undefined behaviour to save me writing code to deal with future to shared_future conversion etc, and of course making use of undefined behaviour always produces faster code, so specifically speaking what I was doing was to use reinterpret_cast to have the compiler not concern itself with whether the future the promise was talking to was really a future or a shared_future state. ...
Monday 20th July 2015: 2.04am. Location: Dromahane. It's 3am the night before my C++ library proposed Boost.AFIO goes before peer review. This is the culmination of about 30 months of effort, with pretty consistent 60-70 hour weeks since February, and about 100 hours this week. Megan has been amazing to get me this far. Anyway, the final code base to be presented is finally done and right now the CI is testing it, I'm just going to go eat something and then tackle the documentation to be presented for review, hopefully will reach bed before 6am as I'll be minding Clara from 9am onwards all day tomorrow. ...
Friday 14th August 2015: 11.48am. I just crashed my dev workstation hard drive firmware when running the second tutorial program for proposed Boost.AFIO - as in, drive dropped out and was no longer recognised by the system until the next power off. This is rather worrying - SMART says everything is fine, though it's an old drive at 16,800 hours powered on. I think I might run a backup anyway tonight of all 400Gb, better safe than sorry. ...
Friday 21st August 2015: 5.40pm. Just sent proposed Boost.AFIO off for its final CI pass before it is submitted for Boost peer review starting later today and lasting until 31st August. The last few days have been stressful - the new workshop tutorial for AFIO is a step-by-step workshop on implementing an asynchronous key-value store, and that meant running many benchmarks which heavily stressed my mechanical hard drive, which began to hang itself until power cycle. ...
Friday 21st August 2015: 9.13pm. Location: Dromahane. And proposed Boost.AFIO is submitted for peer review (the one failing test is a test timeout and can be disregarded)!#boostcpp#cplusplus#boostafio ...
Sunday 23rd August 2015: 9.19pm. Yay just got a direct-from-mmap dense hash map implementation working, so this hash table works straight from a memory mapped file which means it can scale out to really huge hash tables - like billions of entries. On modification, it rehashes the entries into a new hash table which can also go straight into a mmap, though I suspect the linear complexity would make modifying billion entry hash tables rather slow. ...
Wednesday 14th October 2015: 10.52am. CppCon videos are up, and here is mine on Racing the File System which is a beginner's level workshop on race free filing system techniques leading up to proposed Boost.AFIO, an asynchronous file and race free filesystem library for C++. As much as it's "beginner's level", it's more really "from first principles", so it gets into more interesting stuff by the end including the transactional key-value store I'd like to be standardised into the C++ runtime based on AFIO. ...
Monday 8th February 2016: 12.30am. Link shared: https://github.com/BoostGSoC13/boost.afio/tree/master/fs_probe I reached a major milestone in the post peer review Boost.AFIO v2 rewrite today which has consumed most of my free time since November: it now has working Windows and POSIX AIO async file i/o backends with file open, close, clone, scatter-gather read/writes and truncate implemented, so a long, long way still from AFIO v1's comprehensive facilities but still an achievement. The POSIX AIO backend is 100% pure, and so therefore has awful performance because POSIX AIO has a terrible design, but it does work on FreeBSD and Linux without surprises and the storage profile probing yielded many interesting answers to long standing questions about concurrent file i/o atomicity which are now answering the many questions about this on Stackoverflow. ...
Saturday 9th April 2016: 7.33am. Link shared: http://accu.org/index.php/conferences/accu_conference_2016/accu2016_sessions#Distributed_Mutual_Exclusion_using_Proposed_Boost.AFIO_(asynchronous_filesystem_and_file_io) The ACCU conference is only ten days away! Proposed Boost.AFIO v2, which I'll be presenting for the first time at ACCU, has the functionality I'm presenting written as of yesterday but it most definitely is not working - as the final part I added all required itself to work, I spent two weeks writing code, only checking to see if it compiles, not if it works until now so I have a raft of debugging to do this weekend. ...
Friday 15th April 2016: 10.47pm. Employment is on pause from now until I present proposed Boost.AFIO v2 for the first time at the ACCU conference next week. I'm up in Bristol 2 room, Thursday at 11am! The distributed mutual exclusion file system algorithm I'm presenting is completed, the slides have proved to be a bit of a bear though, can't get the structure to where I'm happy with it.#boostafio #boostcpp #c++ #accu ...
Monday 18th April 2016: 10.08pm. The final slide in my proposed Boost.AFIO v2 talk at the ACCU conference on Thursday showing the colour coded empirical benchmarks for three methods of mutual exclusion on the file system across the file systems NTFS, ReFS, FAT32 and exFAT. The top three are for contended locks, the bottom three for uncontended locks.#accu2015 @ACCUConf #boostafio #boostcpp ...
Thursday 21st April 2016: 2.10pm. Link shared: https://github.com/ned14/boost.afio/tree/master/doc/presentations Slides for my ACCU talk on proposed Boost.AFIO are now online at https://github.com/ned14/boost.afio/tree/master/doc/presentations#ACCUConf #accu2016 #boostcpp #boostafio
...
Thursday 26th May 2016: 8.20am. Link shared: http://melpon.org/wandbox/permlink/4rJCUypvBbGGFTqx I'm not a natural C++ metaprogrammer, so it took me two mornings before work to come up with this of which I am quite proud:```#include <stdio.h>#include <string>template <class... Args> struct Foo {};template <class T, class... Args> struct Foo<T, Args...> { T v; Foo<Args...> rest;};template <class T> struct Foo<T> { T v; };namespace detail {template <size_t N, class T, class... Args> struct getFoo { constexpr auto operator()(const Foo<T, Args. ...
Monday 27th June 2016: 10.58am. My CppCon talk "Memory mapped distributed mutual exclusion using Proposed Boost.AFIO (asynchronous filesystem and file i/o)" has been accepted for presentation at CppCon 2016 in Seattle this September. My thanks to the Programme Committee for taking a bit of a risk on this topic, because as you'll see from the proposed abstract it is not certain if the topic is even feasible (if it isn't, I'll present the results of my failure :) )" ...
Wednesday 7th September 2016: 7.27am. Link shared: https://cppcon2016.sched.org/event/7nKi/better-mutual-exclusion-on-the-filesystem-using-boostafio-asynchronous-file-and-filesystem Ten days exactly to go before I depart for CppCon 2016 in Seattle! I am very glad to report that the code I am presenting (link to talk is below) finally passed the last of its unit tests yesterday, so it is finally debugged and I have some performance benchmarks. As expected, this new filesystem locking algorithm is a full order of magnitude (10x-20x) faster than anything else in AFIO v2 that was presented at ACCU 2016 last April. ...
Tuesday 13th September 2016: 11.10am. Link shared: https://cppcon2016.sched.org/event/a36061492fca1e20d6521b6fbbc56e37 Finished writing the talk description of my CppCon workshop next week:"This is the third and likely final part of a "from first principles" series of beginner's workshops based on developing the v2 post-peer-review rewrite of proposed Boost.AFIO, a C++ library wrapping the advanced features of the filesystem intended for eventual ISO C++ standardisation. If you're the kind of library developer who likes building unusual low level concurrent algorithms using the very latest C++ 14-17 (proposed) features and testing them for time and space complexities, this is definitely your kind of talk. ...
Wednesday 4th January 2017: 8.48pm. Link shared: https://github.com/ned14/boost-lite/blob/master/cmake/BoostLiteSetupProject.cmake#L9 Nailed this cunning piece of cmake hackery today, so I thought I would share it. It solves the problem of cmake annoyingly not supporting out of the box per-target setting of C++ exceptions on/off, C++ RTTI on/off and whether to use the static or dynamic C++ runtime. It also, very usefully, stops the warning MSVC makes about overriding the C++ exceptions enable that cmake enforces in the core compile flags on MSVC, this is amazingly useful when you turn on error on warnings and your C++ has exceptions disabled. ...
Saturday 8th April 2017: 11.49pm. Link shared: http://my.cdash.org/index.php?project=Boost.AFIO Proposed Boost.AFIO v2 is resurrected, and now passes all its unit tests for the first time since Oct 17th. All the work done to proposed Boost.Outcome to get it ready for peer review (which will happen mid-May!) had caused AFIO to suffer hefty code rot given how dependent it is on Outcome, but none of the derotting fixes were hard, just time consuming. ...
Tuesday 2nd May 2017: 2.47pm. My latest conference video, this one being on the lightweight monadic transport expected<T, E> proposed for C++. This is my first conventional "knowledge transfer" talk where I simply pour knowledge out of my brain into the jug that is the audience as is traditional pedagogy, up until now I had argued a case or had done workshops, knowing that conference organisers always feel a lack of the latter and therefore tend to accept workshops quicker. ...
Tuesday 12th September 2017: 1.16am. Been making great use of my time without employment working on AFIO, my proposed low latency file i/o library for standardisation. I wrote a toy transactional key-value store with AFIO permitting up to 48 concurrent writers, and here is how it performs for 128 bit keys and 1Kb values with 1 thread:1Kb values Windows with NTFS, no integrity, no durability, mmaps: Inserting 1M key-value pairs . ...
Contact the webmaster: Niall Douglas @ webmaster2<at symbol>nedprod.com
(Last updated: 2017-09-12 01:16:16 +0000 UTC)