Using Redis as "data buffer"

Sometimes you need to store some statistical or log information in RDBMS such MySQL or NoSQL like Cassandra. However if you do it directly, the application or website will slow down much.

You can avoid it using several techniques. Here are some advanced examples using PHP with Apache Web Server:

Case 1

You can process the data store after finishing all other "stuff":

// PHP program code
//...
flush(); // this flush output buffer and the visitor see complete webpage

// 2 minutes timelimit
set_time_limit(60 * 2); 

// store the data
store_data($data);

The code works and the user do not see interruptions or delays.
However the drawback here is that the web server can not reuse the connection, because the connection stay open. This means more HTTP clients will stay connected and probably in some case, the webserver will stop responding.

Case 2

If you are running on modern UNIX system like Linux of FreeBSD, you can use fork().

fork() is difficult to debug, but once it works there will not be any problem.

Note the code here is incomplete, and probably will not work.

// PHP program code
//...

$pid = pcntl_fork();	// fork

//if ($pid < 0){ ... }	// error will be processed as child.

if ($pid){		// parent
	exit;
}			// child

// set session leader
$sid = posix_setsid();
        
//if ($sid < 0)	// we do not check for error

// 2 minutes timelimit
set_time_limit(60 * 2);

//because we are in different process, 
//we will need to reopen the database connection again.
reopen_data_connection();

// store the data
store_data($data);

The benefit here is we free the Apache webserver connection, so webserver can serve other clients.

The drawback is we use more memory and more processing power.

Case 3

With previous solutions here is one big drawback we did not mentioned.

Suppose the website have 500, 1000 or even 1500 simultaneous connections. In this case same number of connections with inserts will be send to our database server. No matter if this is MySQL, Oracle or Cassandra, the database will dramatically slow down or even stop responding.

Here is Redis solution.

Instead of inserting the data directly, you can first store it in Redis. Then second process can pick it up from Redis and insert it into the database, one "data portion" at a time.

Drawback here is you will need to have enought memory for Redis.

// PHP program code
//...
// assuming $r is already connected Redis() instance.
$r->sadd("data_pipe", serialize($data));

Then pick up process may look like this:

// Open connection to the database
open_data_connection();

$r = new Redis();
$r->connect("127.0.0.1", "6379");

while(true){
	while($xdata = $r->spop("data_pipe")){
		$data = unserialize($xdata);
	
		// store the data
		store_data($data);
	}

	// No more data
	sleep(30);
}

If database can not keep up, e.g. insert data as fast as new data come in, you can always start the pick up process twice or more, or to do some kind of sharding, by importing the data in 2-3 databases.


Redis connection note

The example uses generic connection to the Redis server. To make the example work with our service you will need to use code similar to this one:

// change these parameter according to the information in your instance list
$host = '50.30.35.9';
$port = 1234;
$password = 'somehashcode';
$db = 0;

$r = new Redis();
$r->connect($host, $port);
$r->auth($password);
$r->select($db);

blog comments powered by Disqus





Code library

Note:
Because we want this page to be useful for memcached users, we tagged with memcached all examples that may be "recreated" for memcached server.


MySQL cache in PHP

How to use Redis for cache MySQL queries in a way similar to memcached
Date: 2011-07

Tags: memcached php cache mysql


How to prevent website to be web-scraped

How to use Redis to prevent web-scraping parsing and web-spam
Date: 2011-07

Tags: webstats memcached php security anti-spam


'Rotating' news

How to do 'Rotating' news list
Date: 2011-07

Tags: php


Show random element

How pool random element from news list
Date: 2011-07

Tags: php


Accounting / Vote example

How Redis can help us with Accounting / Vote / Like / Recommend / +1 clicks
Date: 2011-07

Tags: php mysql


Scoreboard example

How to implement fast hi-score table
Date: 2011-07

Tags: php


Scaling example

How to scale Redis on several servers using sharding
Date: 2011-07

Tags: cloud memcached php sharding scaling


Distributed work pool / pipeline example

How to implement distributed work pool and to scale some work across many computers
Date: 2011-07

Tags: cloud php scaling


Simple realtime web counter

How to collect and store information for page visitors
Date: 2011-07

Tags: php webstats


Emulation of expiration of the set members

How to make set members to expire
Date: 2011-07

Tags: php


Lock example

How Redis can help us with user level locks
Date: 2011-08-23

Tags: memcached php lock


Message queue example

Creating distributed message queue
Date: 2011-09-17

Tags: cloud php queue scaling


Index example

Building reverse index using sets
Date: 2011-11-01

Tags: php


Using Redis for MySQL autoincrement

Using Redis to speed up MySQL inserts
Date: 2011-01-13

Tags: cloud php scaling mysql


Using Redis as "data buffer"

Using Redis for caching MySQL or Cassandra data inserts
Date: 2012-08-04

Tags: webstats mysql cloud cassandra php sharding scaling


Calculating how many visitors are on the same webpage using rolling average

How to produce 'There are 123 visitors on this page' sign
Date: 2011-07

Tags: memcached php webstats


© Jul.2011 - 2017, E-Nick.org