jeudi 1 décembre 2016

Parallelize rocksdb iterator

My iterator code:

Iterator* iterator = _db->NewIterator(ReadOptions());
for (iterator->SeekToFirst(); iterator->Valid(); iterator->Next()) {
    Slice const& key = iterator->key();
    Slice const& value = iterator->value();
    bool continue = callback(key, value);
    if (!continue) {
        break;
    }
}

The order of iterating doesn't matter, but breaking the loop is important so we won't iterate over unneeded elements. I use the fact that the elements are sorted by a meaningful order to know when to break. callback can potentially take a long time.

So the pseudo code of what I would like to do is

parallel-iterate(iterator)
   if(!callback(key,value)
     stop-parallel

I tried to apply concurrency::parallel_for_each, but seems like it won't fit rocksdb api.

How would you suggest to implement concurrent iteration?

Aucun commentaire:

Enregistrer un commentaire