redis · diff
v2.0.1 to v2.0.1
343 added, 256 removed. Audit A to A.
---
name: redis
description: Use when working with Redis - in-memory database, caching, pub/sub, sessions, rate limiting, RESP3, asyncio, Redis Stack
metadata:
author: mte90
version: 2.0.1
tags:
- redis
- database
- caching
- pub-sub
- sessions
- rate-limiting
- data-structures
- nosql
- resp3
- asyncio
---
# Redis (redis-py v8.0)
Redis - in-memory data structure store, used as database, cache, and message broker.
**redis-py v8.0 changes:**
- Python 3.10+ required
- RESP3 is the default protocol (was RESP2 in 7.x)
- Async API: `redis.asyncio` module (not `.asyncio` property)
## Installation
```bash
pip install redis
```
## Basic Usage
### Synchronous Client
```python
import redis
# Default protocol is now RESP3
r = redis.Redis(host='localhost', port=6379, db=0)
# Basic commands
r.set('foo', 'bar')
value = r.get('foo') # b'bar'
# Close connection
r.close()
```
- ### RESP2 vs RESP3
-
- ```python
- # RESP3 (default in v8.0)
- r = redis.Redis(protocol=3)
-
- # Opt back into RESP2
- r = redis.Redis(protocol=2)
- ```
-
### Async Client
```python
import redis.asyncio as redis
async def example():
# Default protocol is RESP3
r = redis.Redis(host='localhost', port=6379, db=0)
await r.set('foo', 'bar')
value = await r.get('foo') # b'bar'
# Close connection
await r.aclose()
```
**Migration from v7.x:**
```python
# OLD (v7.x and earlier)
from redis import Redis
r = Redis()
client = r.asyncio # Property
# NEW (v8.0+)
import redis.asyncio as redis
r = redis.Redis()
```
## Connection Pool
```python
# Create a connection pool
pool = redis.ConnectionPool(
host='localhost',
port=6379,
db=0,
max_connections=50,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
retry_on_timeout=True,
)
# Use pool with client
r = redis.Redis(connection_pool=pool)
```
### Connection Pool with Password
```python
pool = redis.ConnectionPool(
host='localhost',
port=6379,
password='your_password',
max_connections=100,
)
r = redis.Redis(connection_pool=pool)
```
## Core Commands
- ### Strings
-
```python
- # Set
- r.set('key', 'value')
- r.set('key', 'value', ex=300) # Expiration in seconds
- r.setnx('key', 'value') # Set if not exists
- r.mset({'key1': 'value1', 'key2': 'value2'})
+ # Strings
+ r.set('key', 'value', ex=300)
+ r.setnx('key', 'value')
+ r.mset({'key1': 'value1'})
+ r.get('key')
+ r.mget(['key1', 'key2'])
+ r.delete('key')
+ r.exists('key')
+ r.ttl('key')
+ r.expire('key', 300)
- # Get
- value = r.get('key')
- values = r.mget(['key1', 'key2'])
+ # Hashes
+ r.hset('user:1', mapping={'name': 'Alex', 'email': 'a@b.com'})
+ r.hget('user:1', 'name')
+ r.hgetall('user:1')
+ r.hexists('user:1', 'name')
+ r.hkeys('user:1')
+ r.hdel('user:1', 'name')
- # Delete
- r.delete('key')
- r.delete('key1', 'key2', 'key3')
+ # Lists
+ r.lpush('mylist', 'item1', 'item2')
+ r.rpush('mylist', 'item3')
+ r.lpop('mylist')
+ r.rpop('mylist')
+ r.lrange('mylist', 0, -1)
+ r.llen('mylist')
- # Exists
- exists = r.exists('key')
+ # Sets
+ r.sadd('myset', 'item1', 'item2')
+ r.sismember('myset', 'item1')
+ r.smembers('myset')
+ r.srem('myset', 'item1')
+ r.sunion('set1', 'set2')
+ r.sinter('set1', 'set2')
- # TTL
- ttl = r.ttl('key') # Time to live in seconds
- r.expire('key', 300) # Set expiration
+ # Sorted Sets
+ r.zadd('scores', {'player1': 100, 'player2': 200})
+ r.zrange('scores', 0, -1, withscores=True)
+ r.zrevrange('scores', 0, 9)
+ r.zrem('scores', 'player1')
+ r.zscore('scores', 'player1')
```
- ### Hashes
+ ## RESP3 Protocol
- ```python
- # Set hash field
- r.hset('user:1', 'name', 'Alex')
- r.hset('user:1', mapping={'name': 'Alex', 'email': 'alex@example.com'})
+ RESP3 (Redis Serialization Protocol version 3) is the default in redis-py v8.0. It introduces push messages, better type support, and client-side caching.
- # Get hash field
- name = r.hget('user:1', 'name')
- user = r.hgetall('user:1') # All fields
+ ### Negotiating RESP3
- # Check field exists
- exists = r.hexists('user:1', 'name')
+ ```python
+ # Explicitly request RESP3 (default in v8.0)
+ r = redis.Redis(protocol=3)
- # Get all fields
- fields = r.hkeys('user:1')
- values = r.hvals('user:1')
+ # Check negotiated protocol
+ info = r.info('server')
+ print(info['redis_version'])
- # Delete field
- r.hdel('user:1', 'name')
+ # Verify protocol version
+ client_info = r.client_info()
+ # Look for 'proto' field in response
```
- ### Lists
+ ### RESP3 Features
+ **Push Messages:**
```python
- # Push to list
- r.lpush('mylist', 'item1', 'item2')
- r.rpush('mylist', 'item3')
-
- # Pop from list
- item = r.lpop('mylist')
- item = r.rpop('mylist')
-
- # Get range
- items = r.lrange('mylist', 0, -1)
+ # Enable push message handling
+ r = redis.Redis(protocol=3, push_response_callback=lambda msg: print(msg))
- # List length
- length = r.llen('mylist')
+ # Client-side caching with CLIENT TRACKING
+ r.execute_command('CLIENT TRACKING', 'ON', 'REDIRECT', r.connection_pool.get_connection('_').pid)
```
- ### Sets
-
+ **New Commands/Options:**
```python
- # Add to set
- r.sadd('myset', 'item1', 'item2', 'item3')
-
- # Check membership
- exists = r.sismember('myset', 'item1')
+ # GETDEL - atomic get and delete (RESP3-friendly pattern)
+ value = r.execute_command('GETDEL', 'key')
- # Get all members
- members = r.smembers('myset')
+ # EXPIRE with NX/XX/GT/LT options
+ r.expire('key', 300, nx=True) # Only set if not exists
+ r.expire('key', 300, xx=True) # Only set if exists
- # Remove from set
- r.srem('myset', 'item1')
+ # OBJECT introspection
+ memory_usage = r.object_encoding('key')
+ idletime = r.object_idletime('key')
- # Set operations
- r.sunion('set1', 'set2')
- r.sinter('set1', 'set2')
+ # COMMAND introspection
+ commands = r.command()
```
- ### Sorted Sets
-
- ```python
- # Add to sorted set
- r.zadd('scores', {'player1': 100, 'player2': 200})
-
- # Get range by score
- top_players = r.zrange('scores', 0, -1, withscores=True)
-
- # Get range by rank
- top_players = r.zrevrange('scores', 0, 9) # Top 10
+ ### Operational Caution
- # Remove from sorted set
- r.zrem('scores', 'player1')
+ > **Proxy/Managed Service Downgrade:** A proxy (Twemproxy, Codis) or managed service in front of Redis may downgrade to RESP2 and silently remove RESP3 capabilities. Always verify the negotiated protocol before relying on RESP3-only behavior.
- # Score of member
- score = r.zscore('scores', 'player1')
+ **Check before using RESP3 features:**
+ ```python
+ def verify_resp3_support(client):
+ """Verify RESP3 is actually available."""
+ try:
+ # Try a RESP3-specific command
+ client.execute_command('HELLO', 3)
+ return True
+ except Exception:
+ # Downgraded to RESP2
+ return False
```
+ Source: [RESP3 Protocol Specification](https://github.com/antirez/redis-doc/blob/master/RESP3.md)
+
## Redis Stack Modules
### Redis Search
```python
- # Create index
- r.ft().create_index((
- TextField('name'),
- NumericField('age'),
- TagField('tags'),
- ))
-
- # Search
+ r.ft().create_index((TextField('name'), NumericField('age'), TagField('tags')))
results = r.ft().search('@name:alex @age:[20 30]')
-
- # Drop index
r.ft().dropindex()
```
### Redis JSON
```python
from redis.commands.json.path import Path
- # Set JSON document
- r.json().set('/user:1', Path.root_path(), {
- 'name': 'Alex',
- 'age': 30,
- 'tags': ['developer', 'python'],
- })
-
- # Get JSON document
- user = r.json().get('/user:1')
-
- # Get specific field
- name = r.json().get('/user:1', Path('$.name'))
-
- # Merge update
+ r.json().set('/user:1', Path.root_path(), {'name': 'Alex', 'age': 30})
+ r.json().get('/user:1')
+ r.json().get('/user:1', Path('$.name'))
r.json().merge('/user:1', Path.root_path(), {'age': 31})
-
- # Delete field
r.json().del_('/user:1', Path('$.age'))
```
### Redis TimeSeries
```python
- # Create time series
r.ts().create('sensor:1')
-
- # Add data point
- r.ts().add('sensor:1', '*', 25.3) # Current timestamp
- r.ts().add('sensor:1', 1234567890, 25.5) # Specific timestamp
-
- # Get range
- data = r.ts().range('sensor:1', 0, '*')
-
- # Add with aggregation
r.ts().add('sensor:1', '*', 25.3)
+ r.ts().range('sensor:1', 0, '*')
aggregated = r.ts().range('sensor:1', 0, '*', agg_type='avg', bucket_size=60000)
```
## Pipelines
```python
# Basic pipeline
pipe = r.pipeline()
pipe.set('key1', 'value1')
pipe.set('key2', 'value2')
pipe.get('key1')
results = pipe.execute() # [True, True, b'value1']
# Watch for transactions
pipe = r.pipeline(True)
pipe.watch('mykey')
val = pipe.get('mykey')
pipe.multi()
pipe.set('mykey', val + 1)
pipe.execute()
```
- ## Django Integration
+ ## Rules That Prevent Incidents
- ### django-redis (still works with v8.0)
+ These guidelines prevent specific failure modes in production.
- ```bash
- pip install django-redis
- ```
+ ### TTL Discipline Prevents Unbounded Key Growth
+ **Failure prevented:** Memory exhaustion from accumulating keys with no expiration.
+
+ Always set TTL on cache keys. Redis is an in-memory store; keys without TTL persist forever and accumulate.
+
```python
- # settings.py
- CACHES = {
- 'default': {
- 'BACKEND': 'django_redis.cache.RedisCache',
- 'LOCATION': 'redis://127.0.0.1:6379/1',
- 'OPTIONS': {
- 'CLIENT_CLASS': 'django_redis.client.DefaultClient',
- 'CONNECTION_POOL_KWARGS': {
- 'max_connections': 50,
- 'retry_on_timeout': True,
- 'protocol': 3, # RESP3
- },
- 'SOCKET_CONNECT_TIMEOUT': 5,
- 'SOCKET_TIMEOUT': 5,
- },
- 'KEY_PREFIX': 'myapp',
- 'VERSION': 1,
- }
- }
+ # BAD - key persists forever
+ r.set('user:session:123', session_data)
+
+ # GOOD - key expires after 1 hour
+ r.set('user:session:123', session_data, ex=3600)
```
- ```python
- from django.core.cache import cache
+ ### Never Use KEYS/SCAN Patterns on Large Keyspaces
- # Set with expiration
- cache.set('key', 'value', timeout=300)
+ **Failure prevented:** Single-threaded Redis blocking for seconds during KEYS * on millions of keys.
- # Get
- value = cache.get('key', 'default_value')
+ ```python
+ # BAD - blocks Redis for seconds
+ keys = r.keys('user:*')
- # Delete
- cache.delete('key')
- cache.delete_pattern('user_*')
+ # GOOD - non-blocking scan
+ cursor = 0
+ while True:
+ cursor, keys = r.scan(cursor, match='user:*', count=100)
+ # Process keys
+ if cursor == 0:
+ break
```
- ### Celery Broker
+ ### Pipeline vs Transaction vs Lua for Atomicity
+ **Failure prevented:** Race conditions when multiple operations must be atomic.
+
+ - **Pipeline:** Batch commands for single round-trip, not atomic
+ - **Transaction (WATCH/MULTI/EXEC):** Optimistic locking, fails if key changed
+ - **Lua script:** True atomicity, server-side execution
+
```python
- # settings.py
- CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
- CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/1'
- CELERY_ACCEPT_CONTENT = ['json']
- CELERY_TASK_SERIALIZER = 'json'
- CELERY_RESULT_SERIALIZER = 'json'
+ # Pipeline - fast but NOT atomic
+ pipe = r.pipeline()
+ pipe.incr('counter')
+ pipe.set('flag', 'done')
+ pipe.execute() # Another client can interleave
+
+ # Transaction - atomic if key unchanged
+ pipe = r.pipeline(True)
+ pipe.watch('counter')
+ pipe.multi()
+ pipe.incr('counter')
+ pipe.execute() # Raises WatchError if counter changed
+
+ # Lua - truly atomic
+ script = """
+ local current = redis.call('INCR', KEYS[1])
+ redis.call('SET', KEYS[2], 'done')
+ return current
+ """
+ r.eval(script, 2, 'counter', 'flag')
```
- ### Django Channels
+ ### Cache-Aside vs Write-Through Failure Behavior
+ **Failure prevented:** Stale data after cache invalidation.
+
+ - **Cache-aside:** App loads cache, misses, reads DB, writes cache. On write, invalidate cache. Risk: stale window between invalidate and next read.
+ - **Write-through:** App writes cache, cache writes DB. Risk: write latency doubles, cache write failure may lose data.
+
```python
- # settings.py
- CHANNEL_LAYERS = {
- 'default': {
- 'BACKEND': 'channels_redis.core.RedisChannelLayer',
- 'CONFIG': {
- 'hosts': [('127.0.0.1', 6379)],
- },
- }
- }
+ # Cache-aside pattern
+ def get_user(user_id):
+ key = f'user:{user_id}'
+ user = r.get(key)
+ if user is None:
+ user = db.get_user(user_id) # Slow path
+ r.setex(key, 3600, user) # Populate cache
+ return user
+
+ # Invalidate on write
+ def update_user(user_id, data):
+ db.update_user(user_id, data)
+ r.delete(f'user:{user_id}') # Invalidate, don't write
```
- ## Redis CLI
+ ### Cache Stampede Prevention
- ```bash
- # Connect
- redis-cli
+ **Failure prevented:** Thundering herd when popular key expires—thousands of requests hit DB simultaneously.
- # Check keys
- KEYS *
+ Use distributed lock or jittered TTL.
- # Delete by pattern (use with caution)
- SCAN 0 MATCH user:* COUNT 1000 | xargs redis-cli DEL
+ ```python
+ # Lock-based refresh
+ import redis.lock
- # Clear current database
- FLUSHDB
+ def get_user_safe(user_id):
+ key = f'user:{user_id}'
+ user = r.get(key)
+ if user is None:
+ lock = r.lock(f'lock:{key}', timeout=10)
+ if lock.acquire(blocking=True):
+ try:
+ # Double-check after acquiring lock
+ user = r.get(key)
+ if user is None:
+ user = db.get_user(user_id)
+ r.setex(key, 3600, user)
+ finally:
+ lock.release()
+ else:
+ # Wait and retry
+ time.sleep(0.1)
+ user = r.get(key) or db.get_user(user_id)
+ return user
- # Monitor commands
- MONITOR
+ # Jittered TTL approach
+ import random
+ ttl = 3600 + random.randint(-300, 300) # ±5 min jitter
+ r.setex(key, ttl, user)
```
- ## Best Practices
+ ### Never Cache Per-Request Volatile Data
- ### Connection Management
+ **Failure prevented:** Cross-request data leakage, memory waste from unique keys.
```python
- from redis import ConnectionPool
-
- # Reuse connection pool globally
- pool = ConnectionPool.from_url('redis://localhost:6379/0', max_connections=50)
+ # BAD - each request creates unique key
+ cache.set(f'token:{request_id}', token, timeout=300)
- def get_redis():
- return redis.Redis(connection_pool=pool)
+ # GOOD - session-scoped data stays in memory/session store
+ request.session['token'] = token
```
- ### Key Design
+ ## Production Failure Modes
+ Understanding these failure modes helps diagnose production issues.
+
+ ### Connection Exhaustion Under Burst
+
+ **Symptom:** `Connection refused` or timeout errors during traffic spikes.
+
+ Redis uses a single thread for command processing. Too many concurrent connections exhaust file descriptors and cause connection failures.
+
+ **Fix:** Use a connection pooler or enforce bounded concurrency.
+
```python
- # Use namespaced keys
- KEY_PREFIX = 'myapp'
- VERSION = 1
- # Keys become: myapp:v1:user:123
+ # Bounded connection pool
+ pool = redis.ConnectionPool(
+ max_connections=50, # Cap connections
+ socket_connect_timeout=5,
+ socket_timeout=5,
+ retry_on_timeout=True,
+ )
+ r = redis.Redis(connection_pool=pool)
+ ```
- # Add TTL to all keys
- cache.set('temp_token', token, timeout=300) # 5 min
+ ### maxmemory Eviction Policy Silently Dropping Keys
- # Use consistent naming
- cache.set('user:profile:123', data)
- cache.set('user:session:123', data)
+ **Symptom:** Cache returns missing keys despite recent writes; data appears to "disappear".
+
+ When Redis hits `maxmemory`, it evicts keys based on the configured policy. The wrong policy turns a cache into a source of data loss.
+
+ **Check eviction policy:**
+ ```bash
+ redis-cli CONFIG GET maxmemory-policy
```
- ### Performance
+ **Common policies:**
+ - `noeviction`: Returns error when memory full (default for persistence)
+ - `allkeys-lru`: Evict least recently used keys (good for cache)
+ - `volatile-lru`: Evict LRU keys with TTL set (good for mixed workloads)
+ - `allkeys-lfu`: Evict least frequently used keys (Redis 4.0+)
```python
- # Batch operations
- r.mset({'key1': 'value1', 'key2': 'value2', 'key3': 'value3'})
+ # Set appropriate eviction policy
+ r.config_set('maxmemory-policy', 'allkeys-lru')
+ ```
- # Use pipeline for multiple ops
- pipe = r.pipeline()
- pipe.set('a', 1)
- pipe.set('b', 2)
- pipe.execute() # Atomic, single round-trip
+ ### Replication Lag Causing Stale Read After Write
- # Use SCAN instead of KEYS * in production
+ **Symptom:** Write succeeds, immediate read from replica returns old value.
+
+ In replicated setups, writes go to primary, reads may hit replica. Replication lag causes stale reads.
+
+ **Fix:** Read from primary after writes, or use `WAIT` to confirm replication.
+
+ ```python
+ # Wait for replica acknowledgment
+ result = r.set('key', 'value')
+ replicas_acked = r.wait(1, timeout=1000) # Wait for 1 replica
+ if replicas_acked == 0:
+ log.warning("Replication lag detected")
+ ```
+
+ ### Blocking Commands Freezing Single-Threaded Server
+
+ **Symptom:** All clients experience latency spike; Redis appears frozen.
+
+ Redis is single-threaded. Blocking commands like `KEYS`, `SORT`, `HGETALL` on large hashes freeze the server.
+
+ **Check slowlog:**
+ ```bash
+ redis-cli SLOWLOG GET 10
+ ```
+
+ ```python
+ # BAD - blocks server
+ all_keys = r.keys('*')
+
+ # GOOD - use SCAN
cursor = 0
while True:
- cursor, keys = r.scan(cursor, match='user:*', count=100)
- # Process keys
+ cursor, keys = r.scan(cursor, count=100)
if cursor == 0:
break
```
- ### Error Handling
+ ### Slowlog as First Diagnostic for Latency
+ **Symptom:** Gradual latency increase; occasional spikes.
+
+ The slowlog captures commands exceeding a threshold. Inspect it first when diagnosing latency.
+
```python
- from redis.exceptions import ConnectionError, TimeoutError
+ # Set slowlog threshold (microseconds)
+ r.config_set('slowlog-log-slower-than', 10000) # 10ms
- try:
- value = r.get('key')
- except ConnectionError:
- # Fallback to DB or default
- value = get_from_db()
- except TimeoutError:
- # Handle timeout
- value = get_cached_default()
+ # Get slow queries
+ slow_queries = r.slowlog_get(10)
+ for query in slow_queries:
+ print(f"Command: {query['command']}")
+ print(f"Duration: {query['duration']}µs")
+ print(f"Timestamp: {query['timestamp']}")
```
- ### Do:
+ ## When Not to Use Redis
- - Always set TTL (expire keys)
- - Use key prefixes for namespacing
- - Reuse connections (use connection pool)
- - Use pipeline for batch operations
- - Use SCAN instead of KEYS * in production
+ Not every workload benefits from Redis. Know when to choose alternatives.
- ### Don't:
+ ### Relational Database Is Enough
- - Use KEYS * in production (blocks Redis)
- - Store large values (>1MB consider compression)
- - Use Redis as primary data store (it's a cache)
- - Forget connection timeout settings
- - Create new connection per request
+ Many caching use cases don't need Redis. If your access patterns are simple and data volume fits in database memory, use the database cache.
- ## Migration from v7.x
+ **Use database instead of Redis when:**
+ - Query patterns are straightforward (primary key lookups, indexed searches)
+ - Data consistency is critical over speed
+ - You already have connection pooling and query caching in place
- ### Breaking Changes
+ ### Job Queues Belong to a Dedicated Broker
- 1. **Python 3.10+ required**
- - Drop support for Python 3.9 and earlier
+ Redis can power job queues, but dedicated brokers (RabbitMQ, SQS, Bull) provide:
+ - Message acknowledgment and retry
+ - Dead letter queues
+ - Priority queues
+ - Better visibility and monitoring
- 2. **RESP3 is default**
- ```python
- # v7.x defaulted to RESP2
- # v8.0 defaults to RESP3
- r = redis.Redis() # Now RESP3
- r = redis.Redis(protocol=2) # Opt into RESP2
- ```
+ **Use a dedicated broker when:**
+ - Jobs must not be lost
+ - Complex routing or priority is needed
+ - You need visibility into queue depth and processing
- 3. **Async import path changed**
- ```python
- # v7.x
- from redis import Redis
- client = Redis().asyncio
-
- # v8.0
- import redis.asyncio as redis
- client = redis.Redis()
- ```
+ ### Caching Pages Nobody Requests
- ## References
+ Caching costs memory. If a page is rarely accessed, caching it wastes memory and adds complexity.
+ **Cache only when:**
+ - Hit rate is high (>80%)
+ - Computation/cache miss is expensive
+ - Data is read-heavy, write-light
+
+ ### Multi-Region Writes Need Different Consistency
+
+ Redis is primarily single-primary. Multi-region active-active setups require different consistency models.
+
+ **Consider alternatives when:**
+ - You need true multi-region writes with low latency
+ - Strong consistency across regions is required
+ - Conflict resolution is complex
+
+ ## Django Integration
+
+ Django-specific patterns (django-redis cache backend, invalidation on model save, per-user rate limiting, Celery broker, Channels) live in `references/django-integration.md`.
+
+ ## Deep Dives
+
+ Load these reference files on demand for specific use cases:
+
+ - `references/django-integration.md` — When working with Django projects (cache backend, invalidation, rate limiting, Celery, Channels)
+
+ ## CLI Operations
+
+ ```bash
+ # Connect
+ redis-cli
+
+ # Check keys (use with caution)
+ KEYS *
+
+ # Delete by pattern (use with caution)
+ SCAN 0 MATCH user:* COUNT 1000 | xargs redis-cli DEL
+
+ # Clear current database
+ FLUSHDB
+
+ # Monitor commands
+ MONITOR
+
+ # Check slowlog
+ SLOWLOG GET 10
+
+ # Check eviction policy
+ CONFIG GET maxmemory-policy
+ ```
+
+ ## References
- [redis-py Official Documentation](https://redis.readthedocs.io/)
- [redis-py v8.0 Release Notes](https://github.com/redis/redis-py/releases)
- [Redis Stack Documentation](https://redis.io/docs/stack/)
- [RESP3 Protocol Specification](https://github.com/antirez/redis-doc/blob/master/RESP3.md)
+