Distributed queue example

There were lots of discussing lately about implementing distributed message queues using MySQL.

This example show how we can implement distributed queue using Redis and PHP.
It is very similar to the Distributed work pool / pipeline example, but instead of random work / messages processing, it use FIFO (first in - first out) functionality.

First file "master.php" that "push" messages.

For each message we do following:

Second file called "slave.php":

Because Redis is single threaded and atomic, we are insured that when some slave gets the message, no one slave will get same message.

Note that you can have many slaves and there is no requirement those slaves to be on same machine, so you can scale on as many machines as you want.

master.php

$r = new Redis();
$r->connect("127.0.0.1", "6379");


// Get new ID...
$message_id = $r->incr("all:message");

// Prepare the message, we use hash/array, but you can do it different way...
$message = array(
	"id"		=> $message_id		,
	"message_data"	=> rand()
);

// Set the hash...
$r->hmset("message:$message_id", $message);

// push the id into the queue...
$r->lpush("queue:message", $message_id);

slave.php

$r = new Redis();
$r->connect("192.168.0.1", "6379");

while(true){
	// pop the message ID from the queue, or block until one is available...
	$message_id = $r->blpop("queue:message");
	
	
	// Get the message itself...
	$message = $r->hgetall("message:$message_id");
	
	if (count($message) == 0)
		continue;
		
	// If you need, delete the key...
	$r->del("message:$message_id");
	
	// Process the message...
	process($message);
}

Redis connection note

The example uses generic connection to the Redis server. To make the example work with our service you will need to use code similar to this one:

// change these parameter according to the information in your instance list
$host = '50.30.35.9';
$port = 1234;
$password = 'somehashcode';
$db = 0;

$r = new Redis();
$r->connect($host, $port);
$r->auth($password);
$r->select($db);

blog comments powered by Disqus





Code library

Note:
Because we want this page to be useful for memcached users, we tagged with memcached all examples that may be "recreated" for memcached server.


MySQL cache in PHP

How to use Redis for cache MySQL queries in a way similar to memcached
Date: 2011-07

Tags: memcached cache php mysql


How to prevent website to be web-scraped

How to use Redis to prevent web-scraping parsing and web-spam
Date: 2011-07

Tags: webstats memcached security php anti-spam


'Rotating' news

How to do 'Rotating' news list
Date: 2011-07

Tags: php


Show random element

How pool random element from news list
Date: 2011-07

Tags: php


Accounting / Vote example

How Redis can help us with Accounting / Vote / Like / Recommend / +1 clicks
Date: 2011-07

Tags: php mysql


Scoreboard example

How to implement fast hi-score table
Date: 2011-07

Tags: php


Scaling example

How to scale Redis on several servers using sharding
Date: 2011-07

Tags: memcached cloud sharding php scaling


Distributed work pool / pipeline example

How to implement distributed work pool and to scale some work across many computers
Date: 2011-07

Tags: cloud php scaling


Simple realtime web counter

How to collect and store information for page visitors
Date: 2011-07

Tags: php webstats


Emulation of expiration of the set members

How to make set members to expire
Date: 2011-07

Tags: php


Lock example

How Redis can help us with user level locks
Date: 2011-08-23

Tags: memcached php lock


Message queue example

Creating distributed message queue
Date: 2011-09-17

Tags: cloud php scaling queue


Index example

Building reverse index using sets
Date: 2011-11-01

Tags: php


Using Redis for MySQL autoincrement

Using Redis to speed up MySQL inserts
Date: 2011-01-13

Tags: cloud php scaling mysql


Using Redis as "data buffer"

Using Redis for caching MySQL or Cassandra data inserts
Date: 2012-08-04

Tags: mysql webstats cloud sharding php cassandra scaling


Calculating how many visitors are on the same webpage using rolling average

How to produce 'There are 123 visitors on this page' sign
Date: 2011-07

Tags: memcached php webstats


© Jul.2011 - 2014, E-Nick.org