Perl Benchmark Cache with Expires and Max Size

I’m dealing with Cache at work, and I need a high performance in memory cache system. I’m actually using CHI with Memory Backend, and I have notice that the result setting is really slow comparing to the over.

So I have decide to benchmark all the module I can found on metacpan about cache and try to find the best one.

The best for me is the one that support expires and max size, if possible not both at the same time, with high speed on write and read.

The table below will report the Read/Write speed per second, and the memory consumption. It will also report the number of element found in cache. It is supposed to be near the max_size setting.

I will set the cache to 500k and write 600k of data.

First of all, here a test program with CHI Memory. The other test will be done the same way, adapting a little the syntax of set/get.

And now the result :

Engine Write Speed Read Speed Cache hit Memory usage in KB
Pure Hash 564 181 1 523 941 600 000 203 804
CHI Memory 12 977 171 939 7 100 86 436
CHI RawMemory 21 757 118 395 499 999 490 356
CHI FastMmap (25m file) 23 420 48 960 291 303 46 580
Tie Cache LRU Expires 141 878 169 350 500 000 420 280
Tie Cache FastMmap (25m file) 47 068 92 067 298 020 46 340
Cache LRU 219 534 410 147 500 000 424 108
Cache BDB 39 014 58 737 600 000 588
Cache FastMmap (50m) 54 593 91 893 600 000 65 968
Cache Ref CART 59 186 225 209 500 000 347 360
Cache Ref CAR 55 622 221 103 500 000 308 356
Cache Ref FIFO 147 548 398 192 500 000 211 016
Cache Ref LRU 85 039 108 123 500 000 330 576
Redis TCP / String val 7 275 7 418 600 000 224
Redis TCP / Sereal Encode Decode 6 235 6 771 600 000 224
Redis Socket / String val 20 528 20 205 600 000 196
Redis Socket / Sereal Encode Decode 18 318 19 913 600 000 260
Cache Memcached Fast / Sereal 29 174 39 490 441 472 216
Riak Light (pbc) 734 923 600 000 312
Net Riak (pbc) 396 519 600 000 252
MongoDB 2 701 2 069 600 000 5 036

Here the graph of the different results :

Write / Read

Memory Usage

Read : With Share DB File (1, 2, 4 process)

Write : With Share DB File (1, 2, 4 process)

Conclusion :

About CHI :

The CHI Memory seems bad on keeping cache. I have set 500k max_size and it hold only 13k !

The CHI RawMemory is bad about memory consumption. It take twice the memory need to hold data.

The CHI FastMmap is great for memory usage: low memory consumption share between multiple process. With 4 process and heavy load I have seen errors appear.

CHI is usually poor on writing and have no excellent performance on reading.

About TIE Cache :

Tie Cache LRU has a nice performance but consume memory almost like CHI.

The FastMmap implementation is faster than the CHI one.

The usage, it is like using a simple HASH.

About Cache :

Cache LRU has the highest performance of all Cache packages I have tested. The writing and reading is very very high.

The memory consumption is still very high and cannot be used for any kind of usage. It may be nice for keeping only a maximum amount of data for a compute job.

Cache BDB use a Berkeley Database to save your data. It cost absolutely no memory ! It could be great for long running program with a high amount of data to keep in a local cache. One very big issue is that the database can be corrupted very easily. I have run only 3 processes at the same time and the database is corrupted and impossible to make it work again except by deleting it. Each process need a different database, the only thing is that it will consume disk space. But usually we have a lot more disk space than memory. I will really considerate that one to keep data on long running web application. It is very nice to save memory and keep a lot of data in a local cache.

And Cache FastMmap is the fastest local file cache I have tested and work perfectly with many process. This is my best choice !

About Cache Ref :

Cache Ref has a very nice implementation of memory cache. It cost a little more memory than a simple hash and has a very nice performance.

The Cache Ref FIFO is very simple, use just a little more than the simple hash, and great to keep the last X values. Very great performance, and very nice for memory.

About Redis :

Redis in order to read / write locally, is better in pure socket. The read speed is almost the same as the right speed. The good point is that it doesn’t cost any memory to your process, and all of them can share the memory used by Redis.

It is still more than twice better to use a FastMmap file instead of having a Redis instance to do the same job.

About Memcached:

Memcached is a really good server for caching purpose, and they is XS module to read/write on it, which lead you to a better speed than Redis. But Redis has a lot more feature. The FastMmap is still twice better, and easier to setup.

About Riak:

Riak, by default (I haven’t test with a lot of nodes), is really slow comparing to the other one, and I’m pretty certain that is not the best choice for local caching.

Note that Riak::Light is almost twice faster than Net::Riak.

About MongoDB:

MongoDB give better result on local instance, I’m not pretty sure I have use the unixsocket, the driver MongoDB is not very explicit on this. But the speed is really low comparing to the other caches.

Well, I hope you like the benchmark. Feel free to share. If you have other great module and experience, please share, I would love to add it to the bench.

You can find the tests scripts on github :

Enjoy !


Short URL:
  • dams
    • Celogeek

      I haven’t seen the same result at all on my test.

      • dams
        • Celogeek

          copy that.

          The issue is with a key like string and hash as a result, the speed is not the same.

          CHI is an unified drivers above the other cache system, so it is obviously slower because it map and do stuff to keep all the interface exactly the same.

  • wintrmute

    I’d be interested in seeing your results when using memcached, riak, redis, etc.

    • Celogeek

      Oh interesting.

      I can do it for memcache, redis pure socket and redis tcp (local cache only),
      about riak .. not sure it is ok for a local tiny cache

      I will try to do it today or tomorrow on the same machine as the other test

      • wintrmute

        Regarding Riak.. Yeah, it’s probably overkill. You can configure it to run with just one node rather than the default of three if you want to, though.
        It does perform quite nicely though. Just watch out for which Perl library you use; we found that the common ones ~12 months ago were very slow and so wrote our own, but hopefully the open-source ones on CPAN have caught up.

        • Celogeek

          We have made at work the module Riak::Light, it should be very fast and light, I will try to make a bench with it to compare to redis

        • Celogeek

          I have add the Redis test !

          Write one by one has a limit. The socket is greatly faster that the tcp version.

          In tcp, adding 10k keys in a set per command, with 5 threads lead me to a write of 1,7M of keys / s !
          But for a cache usage, you cannot do that.