Niall’s virtual diary archives – Tuesday 12th September 2017

Tuesday 12th September 2017: 1.16am. Been making great use of my time without employment working on AFIO, my proposed low latency file i/o library for standardisation. I wrote a toy transactional key-value store with AFIO permitting up to 48 concurrent writers, and here is how it performs for 128 bit keys and 1Kb values with 1 thread:

1Kb values Windows with NTFS, no integrity, no durability, mmaps:
Inserting 1M key-value pairs ...
Inserted at 518403 items per sec
Retrieving 1M key-value pairs ...
Fetched at 2192982 items per sec

1Kb values Linux with ext4, no integrity, no durability, mmaps:
Inserting 1M key-value pairs ...
Inserted at 695894 items per sec
Retrieving 1M key-value pairs ...
Fetched at 5988023 items per sec

To put those figures in context, LMDB which is considered one of the fastest key-value stores around achieves about 1.4 million fetches/sec with four threads. With integrity checking on, we nearly match that, but with a single thread rather than four. This is because we use a hash table, and LMDB uses a B-tree, so we'll perform really well up until there is a hash collision at which point my toy key-value store gives up entirely as well, it's toy. Another comparator is https://github.com/simonhf/sharedhashfile where he's seeing 10 million fetches/sec with eight threads on Linux. I'd say I'd match him, and unlike him, this is a persistent store with copy-on-write. And finally there is Vinnie's NuDB https://github.com/vinniefalco/NuDB where he achieves about 330k fetches or inserts/sec per thread. We are far ahead of that, but then this is a much simpler design oddly enough. We do much less than NuDB, so we are much faster.

So not bad for a toy first attempt, and a nice proving ground for AFIO's amazing facilities for writing seriously high performance filesystem code. But I expect to not do much more work on it for now, my next task is to prepare my WG21 paper P0762R0 for submission this time next month. If after that's done I'm still out of contract, I may return to the toy key-value store, see how it does under concurrent modification etc.

#boostafio #cpp