Redis high traffic connection issue

Recenly we have a problem with connection to high traffic redis server.

It took us lots of time to figure out what is really happening, but here is the final breakdown:

[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204010470
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204011369
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204011834
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204012230
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204012561
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204012877
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204013196
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204013397
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
total_connections_received:204013659
[root@redis3 ~]# redis-cli -h 192.168.0.2 -p 6379 info | grep total_connections_received
Could not connect to Redis at 192.168.0.2:6379: Cannot assign requested address

After searching for this error we found that it is related to TCP TIME_WAIT state.

This TCP states is when your system is negotiating a "graceful" disconnect with the other end. TIME_WAIT means both sides have agreed to close and TCP must now wait a prescribed time before taking the connection down.

Solution on Linux is:

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

This allow the OS to quickly reuse those TIME_WAIT TCP sockets.

Solution on MacOSX is:

sysctl -w net.inet.tcp.msl=1000

Please note we never tested on Mac, but some people suggest this setting.

Articles library

Redis as session handler in PHP

How to use Redis for session handler in PHP in a way similar to msession or PostgreSQL sessions
Date: 2011-07

Tags: system administration php

Lazy starter script on Linux such Redhat CentOS or Fedora

How to quickly made Redis to start after Linux boot.
Date: 2011-07

Tags: system administration

Redis vs Memcached comparison

Here is basic comparison of Redis Memcached and Memcachedb.
Date: 2011-07

Tags: memcached

Why RDBMS and SQL are difficult...

...while sometimes NoSQL solution may be easier and way faster? This arcle deals with incremental counters and how they needs to be implemented properly in RDBMS-es such MySQL
Date: 2011-07
Updated: 2011-09-30

Tags: programming anti-pattern mysql

Seamless migration from one Redis server to another

How to use replication to migrate Redis server without service interruption
Date: 2011-07

Tags: system administration

Redis swap issue while save

Explanation why your server load average may spike and server to swap memory when running redis
Date: 2011-08-21

Tags: system administration

Amazon EC2 and Amazon ElastiCache service

Memcached at Amazon first impressions
Date: 2011-08-23

Tags: cloud memcached system administration

Understanding hash-max-zipmap-entries and 'hash of hashes' optimization

Explanation of config parameters such hash-max-zipmap-entries and how we can make huge memory savings in new Redis 2.2
Date: 2011-08-28

Tags: programming system administration optimization php

NoSQL database design example

This article explains how we made our article section
Date: 2011-09-02

Tags: programming database design mysql

Redis save and backup script

Simple backup script for Redis
Date: 2011-09-28

Tags: optimization system administration

Postgres 9.1 foreign data wrapper interface

Postgres 9.1 gives you ability to access data from different data sources including Redis
Date: 2011-10-20

Tags: programming mysql optimization memcached pgsql

Redis high traffic connection issue

Explanation on 'Cannot assign requested address' connection error
Date: 2011-11-12

Tags: system administration optimization scaling

Redis too many open files error on high traffic sites

Explanation on 'Accepting client connection: accept: Too many open files'
Date: 2011-12-21

Tags: memcached system administration security

PHP Redis bug with PHP 5.4

Description of bug PHP Redis bug
Date: 2012-07-02

Tags: system administration php

Case Study: Using Redis intersect on very large sets

Intersection of 2 x 120M MySQL records
Date: 2012-08-29

Tags: programming mysql cloud optimization php

Getting started with Python and Redis on CentOS 6.X

How to install and test Redis client for Python on CentOS
Date: 2013-03-21

Tags: system administration python

GeoIP in Redis

Put Maxmind's GeoIP in Redis for best berformance for cient apps written in PHP or Python
Date: 2013-03-24

Tags: php python programming mysql optimization scaling