Case Study: Using Redis intersect on very large sets

Week or two ago, we needed to calculate difference between two large MySQL tables.

The client had csv dumb in text file of both tables. Because of that we decided to import those dumps in two Redis sets and to see what will happen.

require_once "config.php";

$file = "dump_keys.001.txt.gz";

echo "Importing $file...\n";

$i = 0;

if ($F = gzopen($file, "r")){
        while( ( $line = fgets($F, 1024)) !== false){
                $line = trim(strtolower($line));

                $r->sadd("DUMP1", $line);

                printf("%10d %s\n", $i, $line);
                $i++;
        }

        fclose($F);
}

Both dumps were around 120,000,000 items (keys of MySQL records) and were imported for under 30 min each.

Redis used memory were 21.87 GB.

Finally we did:

redis 127.0.0.1:6379> SDIFFSTORE DUMP DUMP1 DUMP2
(integer) 2655487
(264.29s)

As seen, the intersect took less 300 seconds (less 5 min).

Finally we did script that SPOP() from result set and do some MySQL processing. Whole "difference calculation" were done for less 2 hours.

We did not use cloud, because we have lots of cheap dedicated servers, but example is perfect for demonstrating power of cloud computing.

In case one use Amazon AWS, "Double Extra Large" is the way to go. Since it it one-time job, suppose it will take 3 hours x 0.90 USD.
This makes price less 3 USD to get the job done.

blog comments powered by Disqus

Case Study: Using Redis intersect on very large sets

Articles library

Redis as session handler in PHP

Lazy starter script on Linux such Redhat CentOS or Fedora

Redis vs Memcached comparison

Why RDBMS and SQL are difficult...

Seamless migration from one Redis server to another

Redis swap issue while save

Amazon EC2 and Amazon ElastiCache service

Understanding hash-max-zipmap-entries and 'hash of hashes' optimization

NoSQL database design example

Redis save and backup script

Postgres 9.1 foreign data wrapper interface

Redis high traffic connection issue

Redis too many open files error on high traffic sites

PHP Redis bug with PHP 5.4

Case Study: Using Redis intersect on very large sets

Getting started with Python and Redis on CentOS 6.X

GeoIP in Redis