Understanding hash-max-zipmap-entries and "hash of hashes" optimization
When you set key in Redis, it creates some data structure, which involves pointer and some extra data.
Hashes are made in same way - data structure with ponters for each hash field / value.
However, in recent Redis versions (Redis 2.2), there is optimized way of storing hashes - it may be compared as PHP serialized string or like JSON string:
{"a":1,"b":2}
Note this is only example for better understanding, it does not represent internal way Redis uses.
When using this method, Redis may do huge savings of memory, however when access hash elements, it will need to iterate all hash field, e.g. Redis trades CPU for memory.
To control this behavour, Redis config have two parameters:
- hash_max_zipmap_entries - if hash fields are less this value the optimized method is used.
- hash_max_zipmap_value - if hash values sizes are less this value the optimized method is used.
There are similar config parameters not only for hashes, but also for sets and lists:
# conf parameter, default value
hash-max-zipmap-entries 64
hash-max-zipmap-value 512
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
You can use those in redis.conf.
In Redis 2.4 there will be such setting for sorted lists as well.
Example use:
We made simple PHP code that utilize this optimization, along with some results.
The idea behind the code belong to Redis author - Salvatore Sanfilippo.
Instead of setting key=value, we calculate md5/sha1 from the key, get first 4 characters, and use it as hash name. In this hash we set field with original key name.
If we set 1M keys, there will be 65536 hashes, each with ~16 keys (1M / 65536). This is at least on theory.
Note if you decide to do so in production environments, you will not be able to do expiration and most of key functions must be performed in different way.
PHP code:
$r = new Redis();
$r->connect("127.0.0.1", "6379");
// Calculates the hash name...
function hkey($key){
$maxlen = 4;
$prefix = "_";
return $prefix . substr(md5($key), 0, $maxlen);
}
// set key...
function hset($r, $key, $val){
$key1 = hkey($key);
return $r->hset($key1, $key, $val);
}
// get key...
function hget($r, $key, $val){
$key1 = hkey($key);
return $r->hget($key1, $key);
// ======================================
// Non optimized version
function hset1($r, $key, $val){
return $r->set($key, $val);
}
for($i=0; $i < 1000000; $i++)
hset($r, "key:" . $i, $i);}
The results:
Non optimized version first:
used_memory_human:57.92M
db0:keys=1000000,expires=0
Optimized version:
used_memory_human:22.40M
db0:keys=65536,expires=0
Here you are (default) config parameters we used for this test:
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
Here you are a random element:
redis 127.0.0.1:6379> randomkey
"_d488"
redis 127.0.0.1:6379> hgetall "_d488"
1) "key:50348"
2) "50348"
3) "key:101947"
4) "101947"
5) "key:128127"
6) "128127"
7) "key:260902"
8) "260902"
9) "key:292977"
10) "292977"
11) "key:338186"
12) "338186"
13) "key:359584"
14) "359584"
15) "key:441871"
16) "441871"
17) "key:619595"
18) "619595"
19) "key:630408"
20) "630408"
21) "key:876941"
22) "876941"
23) "key:918003"
24) "918003"
25) "key:931704"
26) "931704"
Links:
blog comments powered by Disqus
Articles library
Redis as session handler in PHP
How to use Redis for session handler in PHP in a way similar to msession or PostgreSQL sessions
Date: 2011-07
Tags: php system administration
Lazy starter script on Linux such Redhat CentOS or Fedora
How to quickly made Redis to start after Linux boot.
Date: 2011-07
Tags: system administration
Redis vs Memcached comparison
Here is basic comparison of Redis Memcached and Memcachedb.
Date: 2011-07
Tags: memcached
Why RDBMS and SQL are difficult...
...while sometimes NoSQL solution may be easier and way faster? This arcle deals with incremental counters and how they needs to be implemented properly in RDBMS-es such MySQL
Date: 2011-07
Updated: 2011-09-30
Tags: mysql programming anti-pattern
Seamless migration from one Redis server to another
How to use replication to migrate Redis server without service interruption
Date: 2011-07
Tags: system administration
Redis swap issue while save
Explanation why your server load average may spike and server to swap memory when running redis
Date: 2011-08-21
Tags: system administration
Amazon EC2 and Amazon ElastiCache service
Memcached at Amazon first impressions
Date: 2011-08-23
Tags: memcached cloud system administration
Understanding hash-max-zipmap-entries and 'hash of hashes' optimization
Explanation of config parameters such hash-max-zipmap-entries and how we can make huge memory savings in new Redis 2.2
Date: 2011-08-28
Tags: optimization php programming system administration
NoSQL database design example
This article explains how we made our article section
Date: 2011-09-02
Tags: mysql database design programming
Redis save and backup script
Simple backup script for Redis
Date: 2011-09-28
Tags: optimization system administration
Postgres 9.1 foreign data wrapper interface
Postgres 9.1 gives you ability to access data from different data sources including Redis
Date: 2011-10-20
Tags: memcached optimization mysql programming pgsql
Redis high traffic connection issue
Explanation on 'Cannot assign requested address' connection error
Date: 2011-11-12
Tags: optimization scaling system administration
Redis too many open files error on high traffic sites
Explanation on 'Accepting client connection: accept: Too many open files'
Date: 2011-12-21
Tags: memcached security system administration
PHP Redis bug with PHP 5.4
Description of bug PHP Redis bug
Date: 2012-07-02
Tags: php system administration
Case Study: Using Redis intersect on very large sets
Intersection of 2 x 120M MySQL records
Date: 2012-08-29
Tags: optimization cloud php mysql programming
Getting started with Python and Redis on CentOS 6.X
How to install and test Redis client for Python on CentOS
Date: 2013-03-21
Tags: python system administration
GeoIP in Redis
Put Maxmind's GeoIP in Redis for best berformance for cient apps written in PHP or Python
Date: 2013-03-24
Tags: optimization php scaling mysql programming python
Redis database file examination
Examination of Redis rdb file in order to find huge memory consumption
Date: 2013-08-23
Tags: optimization system administration