Resenly we have problem with one Redis instance than raise from about 5 GB to 22 GB. Then we temporary increased the size to 24 GB, but it quickly fill this space too.

Nobody really knew whats *that* big inside, so we decided to investigate.

We got excelent Redis-rdb-Tools from here:
redis-rdb-tools @ GitHub.

Then we got saved file "dump.rdb" copy it to different server and generated the CSV with all keys:

rdb -c memory dump.rdb1 > memory.csv

Because the database had 1,954,566 keys, we used a small Python program written by us available here:
aggregate.py @ GitHub.

The program does not need any installation, just copy and run against the CSV file.

Result was following:

# ./aggregate.py memory.csv
     1954000 records processed 

Distribution by Database
-------------------------------------------------------------------------------------------
           # :        count : count% :   size bytes :  size% :      avg size :     max size
-------------------------------------------------------------------------------------------
           0 :      1954565 : 100.00 :  26274261265 : 100.00 :         13442 :   9376828523
-------------------------------------------------------------------------------------------
       total :      1954565 : 100.00 :  26274261265 : 100.00 :         13442 :   9376828523


Distribution by key type
-------------------------------------------------------------------------------------------
           # :        count : count% :   size bytes :  size% :      avg size :     max size
-------------------------------------------------------------------------------------------
         set :           29 :   0.00 :  22569307097 :  85.90 :     778251968 :   9376828523
   sortedset :          199 :   0.01 :   3401076282 :  12.94 :      17090835 :   3400599394
      string :      1954337 :  99.99 :    303877886 :   1.16 :           155 :          260
-------------------------------------------------------------------------------------------
       total :      1954565 : 100.00 :  26274261265 : 100.00 :         13442 :   9376828523


Distribution by sizetype
-------------------------------------------------------------------------------------------
           # :        count : count% :   size bytes :  size% :      avg size :     max size
-------------------------------------------------------------------------------------------
          64 :         1227 :   0.06 :       132906 :   0.00 :           108 :          119
         128 :      1953297 :  99.94 :    303774854 :   1.16 :           155 :          255
         256 :            4 :   0.00 :         1031 :   0.00 :           257 :          260
         512 :            1 :   0.00 :          636 :   0.00 :           636 :          636
        1024 :            2 :   0.00 :         2746 :   0.00 :          1373 :         1549
       16384 :            2 :   0.00 :        63454 :   0.00 :         31727 :        31841
       32768 :            1 :   0.00 :        41516 :   0.00 :         41516 :        41516
       65536 :            1 :   0.00 :        85151 :   0.00 :         85151 :        85151
      131072 :            2 :   0.00 :       294151 :   0.00 :        147075 :       147878
     2097152 :            1 :   0.00 :      2669695 :   0.01 :       2669695 :      2669695
     4194304 :            2 :   0.00 :     14824024 :   0.06 :       7412012 :      7474900
     8388608 :            9 :   0.00 :    101555171 :   0.39 :      11283907 :     13519039
    16777216 :            3 :   0.00 :     68105755 :   0.26 :      22701918 :     26458624
    67108864 :            2 :   0.00 :    157171501 :   0.60 :      78585750 :     78589018
   268435456 :            1 :   0.00 :    491332940 :   1.87 :     491332940 :    491332940
   536870912 :            4 :   0.00 :   3406859501 :  12.97 :     851714875 :    981209934
  1073741824 :            3 :   0.00 :   3849215316 :  14.65 :    1283071772 :   1286023059
  2147483648 :            1 :   0.00 :   3400599394 :  12.94 :    3400599394 :   3400599394
  4294967296 :            1 :   0.00 :   5100703000 :  19.41 :    5100703000 :   5100703000
  8589934592 :            1 :   0.00 :   9376828523 :  35.69 :    9376828523 :   9376828523
-------------------------------------------------------------------------------------------
       total :      1954565 : 100.00 :  26274261265 : 100.00 :         13442 :   9376828523

As you can see 29 Redis sets took 85.90% of total space!!!
Also just 3 keys - probably same sets, took 12.94 + 19.41 + 35.69 = 68.04 % of total space.

To find those we did following grep commands:

# grep 3400599394 memory.csv
0,sortedset,"COUNTERS",3400599394,skiplist,18742508,420669

# grep 5100703000 memory.csv
0,set,"USER_LINKS",5100703000,hashtable,56295349,489

# grep 9376828523 memory.csv
0,set,"WORDS2",9376828523,hashtable,104285128,67

# grep ",set," memory.csv
0,set,"WORDS_X",2669695,hashtable,32188,39267
0,set,"pd:all",155,intset,17,8
0,set,"pd:20130811",8570259,hashtable,94568,67
0,set,"pd:20130812",7474900,hashtable,80483,66
0,set,"pd:20130813",13458297,hashtable,140195,67
0,set,"pd:20130814",11186286,hashtable,129373,65
0,set,"pd:20130815",11175420,hashtable,129631,65
0,set,"pd:20130816",10915606,hashtable,125973,67
...

In this way we can find exact keys, such "COUNTERS", "USER_LINKS", "WORDS2" etc. So we can check what are they for and eventually optimize the scripts or delete them.

In our particular case we deleted some set that were not used anymore and reduced the database size to 1.5 GB :-) !!!


blog comments powered by Disqus





Articles library

Redis as session handler in PHP

How to use Redis for session handler in PHP in a way similar to msession or PostgreSQL sessions
Date: 2011-07

Tags: system administration php


Lazy starter script on Linux such Redhat CentOS or Fedora

How to quickly made Redis to start after Linux boot.
Date: 2011-07

Tags: system administration


Redis vs Memcached comparison

Here is basic comparison of Redis Memcached and Memcachedb.
Date: 2011-07

Tags: memcached


Why RDBMS and SQL are difficult...

...while sometimes NoSQL solution may be easier and way faster? This arcle deals with incremental counters and how they needs to be implemented properly in RDBMS-es such MySQL
Date: 2011-07
Updated: 2011-09-30

Tags: anti-pattern programming mysql


Seamless migration from one Redis server to another

How to use replication to migrate Redis server without service interruption
Date: 2011-07

Tags: system administration


Redis swap issue while save

Explanation why your server load average may spike and server to swap memory when running redis
Date: 2011-08-21

Tags: system administration


Amazon EC2 and Amazon ElastiCache service

Memcached at Amazon first impressions
Date: 2011-08-23

Tags: system administration memcached cloud


Understanding hash-max-zipmap-entries and 'hash of hashes' optimization

Explanation of config parameters such hash-max-zipmap-entries and how we can make huge memory savings in new Redis 2.2
Date: 2011-08-28

Tags: optimization system administration programming php


NoSQL database design example

This article explains how we made our article section
Date: 2011-09-02

Tags: database design programming mysql


Redis save and backup script

Simple backup script for Redis
Date: 2011-09-28

Tags: system administration optimization


Postgres 9.1 foreign data wrapper interface

Postgres 9.1 gives you ability to access data from different data sources including Redis
Date: 2011-10-20

Tags: programming mysql pgsql memcached optimization


Redis high traffic connection issue

Explanation on 'Cannot assign requested address' connection error
Date: 2011-11-12

Tags: optimization system administration scaling


Redis too many open files error on high traffic sites

Explanation on 'Accepting client connection: accept: Too many open files'
Date: 2011-12-21

Tags: system administration memcached security


PHP Redis bug with PHP 5.4

Description of bug PHP Redis bug
Date: 2012-07-02

Tags: system administration php


Case Study: Using Redis intersect on very large sets

Intersection of 2 x 120M MySQL records
Date: 2012-08-29

Tags: programming mysql optimization cloud php


Getting started with Python and Redis on CentOS 6.X

How to install and test Redis client for Python on CentOS
Date: 2013-03-21

Tags: system administration python


GeoIP in Redis

Put Maxmind's GeoIP in Redis for best berformance for cient apps written in PHP or Python
Date: 2013-03-24

Tags: programming mysql optimization php scaling python


Redis database file examination

Examination of Redis rdb file in order to find huge memory consumption
Date: 2013-08-23

Tags: optimization system administration


© Jul.2011 - 2017, E-Nick.org