At Connectifier, we work with large scale data. Given that, and the difficulty of working with large data sets, we’re always keeping an eye on the latest data technologies for ways to improve our performance at scale. With MongoDB 3.0 soon to be released, the datastore will allow you to utilize document level locking and compression, which are huge wins in improving performance as scale compared to previous versions of MongoDB.
MongoDB 3.0 will implement these new features by supporting additional storage engines. WiredTiger is the new storage engine being offered in MongoDB 3.0, but there are also some experimental storage engines available when you compile a binary from source. One of these is a RocksDB engine that was built by folks from Facebook. RocksDB builds on the LevelDB work of Jeff Dean and Sanjay Ghemawat from Google who have been heavily involved in building that company’s infrastructure since the early days. Nearly a third of Connectifier employees have spent some time at Google, so we’re well aware of this work.
I took a bit of time to benchmark the experimental RocksDB storage engine because I wanted to test the performance of a patch to massively improve performance by eliminating the need for a disk read on the update of a record.
You first need to build RocksDB. There are some great instructions on the compose.io blog.
$ git clone https://github.com/facebook/rocksdb.git $ cd rocksdb $ make static_lib $ cp librocksdb.a /usr/local/lib $ cp -r include/* /usr/local/include
Next you need to build MongoDB with the RocksDB engine enabled.
$ git clone https://github.com/mongodb/mongo.git $ cd mongo $ sudo apt-get install libbz2-dev $ scons --rocksdb mongod
Be sure to raise the
ulimit -n in
/etc/security/limits.conf and follow other productionization steps to ensure you’re running at top capacity:
* hard nofile 64000 * soft nofile 64000
Alright, time to start Mongo w/ RocksDB:
./mongod --dbpath ~/rocks --storageEngine=rocksExperiment
And kick off a benchmark run:
$ wget https://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0/mongo-java-driver-2.13.0.jar $ export CLASSPATH=$PWD/mongo-java-driver-2.13.0.jar $ ./run.simple.bash myconfig.bash
And here are the results. Numbers are average sysbench transactions per second (higher is better):
|5 indexed & 5 un-indexed
updates per transaction
|25 indexed updates