Distributed work pool / pipeline example
This example show how we can implement distributed work pool.
First file "master.php" distribute the work.
For each work it create a hash with work parameters (hash fields"a" and "b") and put the work id into a set called "pool:work".
Second file called "slave.php":
- pop random work,
- check the work parameters - e.g. hash fields,
- do the work (e.g. calculate a + b),
- store result into "result" hash field,
- put it back into different set "pool:finishedwork"
After that the slave get new task.
Because Redis is single threaded and atomic, we are insured that when some slave gets the work, no one slave will get same work.
Note that you can have many slaves and there is no requirement those slaves to be on same machine, so you can scale on as many machines as you want.
Also note that slaves may work in a background as a server and it will stay and wait for work:
[~]$ php slave.php > /dev/null 2> /dev/null &
If job is big you can easyly hire thousend machines in Amazon EC2 and you can easyly do 1000 hours jobs in minutes
master.php
$r = new Redis();
$r->connect("127.0.0.1", "6379");
for($i = 0; $i < 1000; $i++){
$x = $r->incr("id:work");
$r->hmset(
"work:$x", array(
"id" => $i ,
"a" => rand() ,
"b" => rand()
)
);
$r->sadd("pool:work", $x);
}
echo "Work distributed\n";
slave.php
$r = new Redis();
$r->connect("192.168.0.1", "6379");
while(true){
// Get random work id...
$x = $r->spop("pool:work");
if ($x === false){
echo "No more work to do\n";
sleep(10);
continue;
}
// Get work description...
$data = $r->hgetall("work:$x");
if (!count($data))
continue;
// Do the work...
$total = $data["a"] + $data["b"];
// Store back...
$r->hset("work:$x", "total", $total);
// And store id in finished pool...
$r->sadd("pool:finishedwork", $x);
echo "Work #$x is finished...\n";
// sleep(2);
}
Screen tesing
[~]$ php master.php
Work distributed
[~]$
[~]$
[~]$ redis-cli
redis 127.0.0.1> scard pool:work
(integer) 1000
redis 127.0.0.1> scard pool:finishedwork
(integer) 0
redis 127.0.0.1> quit
[~]$
[~]$
[~]$ php slave.php
...
Work #238 is finished...
Work #291 is finished...
Work #453 is finished...
Work #858 is finished...
Work #777 is finished...
No more work to do
[~]$
[~]$
[~]$ redis-cli
redis 127.0.0.1> scard pool:work
(integer) 0
redis 127.0.0.1> scard pool:finishedwork
(integer) 1000
redis 127.0.0.1> quit
[~]$
Redis connection note
The example uses generic connection to the Redis server. To make the example work with our service you will need to use code similar to this one:
// change these parameter according to the information in your instance list
$host = '85.25.11.9';
$port = 1234;
$password = 'somehashcode';
$db = 0;
$r = new Redis();
$r->connect($host, $port);
$r->auth($password);
$r->select($db);
blog comments powered by Disqus
Code library
Note:
Because we want this page to be useful for memcached users,
we tagged with memcached all examples that may be "recreated" for memcached server.
MySQL cache in PHP
How to use Redis for cache MySQL queries in a way similar to memcached
Date: 2011-07
Tags: cache memcached mysql php
How to prevent website to be web-scraped
How to use Redis to prevent web-scraping parsing and web-spam
Date: 2011-07
Tags: memcached webstats security php anti-spam
'Rotating' news
How to do 'Rotating' news list
Date: 2011-07
Tags: php
Show random element
How pool random element from news list
Date: 2011-07
Tags: php
Accounting / Vote example
How Redis can help us with Accounting / Vote / Like / Recommend / +1 clicks
Date: 2011-07
Scoreboard example
How to implement fast hi-score table
Date: 2011-07
Tags: php
Scaling example
How to scale Redis on several servers using sharding
Date: 2011-07
Tags: cloud memcached sharding scaling php
Distributed work pool / pipeline example
How to implement distributed work pool and to scale some work across many computers
Date: 2011-07
Simple realtime web counter
How to collect and store information for page visitors
Date: 2011-07
Emulation of expiration of the set members
How to make set members to expire
Date: 2011-07
Tags: php
Lock example
How Redis can help us with user level locks
Date: 2011-08-23
Message queue example
Creating distributed message queue
Date: 2011-09-17
Index example
Building reverse index using sets
Date: 2011-11-01
Tags: php
Using Redis for MySQL autoincrement
Using Redis to speed up MySQL inserts
Date: 2011-01-13
Using Redis as "data buffer"
Using Redis for caching MySQL or Cassandra data inserts
Date: 2012-08-04
Tags: cloud mysql sharding webstats scaling php cassandra
Calculating how many visitors are on the same webpage using rolling average
How to produce 'There are 123 visitors on this page' sign
Date: 2011-07