-
-
Notifications
You must be signed in to change notification settings - Fork 267
Plugin Pattern Discussion
I've built a bunch of modules for extending levelup, supporting job queues, map-reduce,
live-streams (think tail -f), and scuttlebutt.
I've taken care make things as flexible as possible, and to not abuse levelup. However, sometimes I have gone slightly further than a polite module user might.
Yes, I am talking about monkey-patching.
This was entered with the spirit of experimentation, being much faster than making a pull request, And I did not want to include my experimentations in the main project, incase they got stuck there, but turned out not to be such a good idea.
Each module is installed by passing a database instance to it.
var hooks = require('level-hooks')
hooks(db)The plugin may add a property to db with the same name as the plugin.
(that is how the user may access the plugin's functionality)
The plugin must check whether it is already added, and not add itself twice.
The plugin may add any other plugins it depends on - (which will check they are not there already)
level-hooks is a little bit more naughty than that, as I will describe shortly.
How all my plugins fit together.
All the features are based on 3 important modules
hooks allows you to intercept put, del, and batch.
with pre you get the transaction before it goes to the database,
and you have the chance to change it - say adding something to the batch.
The transaction is always converted into an array, as accepted by batch
db.hooks.pre(function (batch) {
batch.push({type: 'put', key: 'log:'+Date.now(), value: Date.now})
return batch
})
db.hooks.post(function (item) {
console.log(item) //{key: key, value: value}
})range-bucket is not a levelup plugin, but is just a module on it's own.
It's used to generate keys that will have certain sorting properties.
levelup.readStream({start: s, end: e}) streams all keys between s,
and e inclusive.
Keys are sorted lexiographically. leveldb does support a custom sort, but it must be configured into the database at compile time, and written in C.
For all intents an purposes - keys in levelup are lexicographically sorted, and there is no way to change that.
However, it's very useful to be able to partition some set of documents into a particular range.
Basically, "lexicographically sorted" means sorted by the initial-bytes in the key. two keys are compared from the start, until one has a byte that is different to the other's byte at that position.
So, there are two particullarity important byte values here - null '\x00', and "LATIN SMALL LETTER Y WITH DIAERESIS" '\xff' http://www.utf8-chartable.de/
If a key looks like this: <0xFF 'a' 'b' 'c'> then it will always sort AFTER anything else, including '\xFF', so if you never insert a document under key: FF you can always retrive all the regular documents with db.readStream({start: '', end: '\xFF'}).
All my plugins that insert data use range-bucket, prefixing anything they insert with '\xFF$PREFIX where $PREFIX is a customizable prefix. some modules use more than one prefix.
range-bucket also has a way to handle arrays as keys, but that is only used by level-map, and level-reduce
I think now, that since all the modules use range-bucket, it should be handled differently - maybe it should be built into the way that plugins work - but more on that later.
The basic feature set provided by leveldb is great, but to build applications we need more than just get/set
if you have a map-reduce, or something, you want that to reliably update when I insert new data, without having to manually trigger map-reduce when I do a put.
Say every time we insert a doc with a certain prefix into the db we want to generate transformations of that doc, and save them elsewhere in the db (for easy retrieval, as part of a range for a particular query)