More Than Just a Cache

Redis Data Structures

Andy Snell

@andrewsnell

Longhorn PHP

October 16, 2021

Setting Expectations

Expect wildly inconsistent  pronunciation

Expectation #1

"Data"

Setting Expectations

Expect oversimplification and some handwaving

Expectation #2

This is an hour long talk

About Me

  • Contract PHP Developer and Consultant
  • Currently Living in Dallas
  • Never Intended to Be a Developer
  • Consider that my real PHP career started in 2016
wkdb.ty/redis
wkdb.ty/redis

Fast Foward: Rediscovering Redis

Tasked to Store Phone Numbers to Not Call 

MySQL Not Performant Enough

 

From >12 GB (on disk) to < 400 mb (in memory)

 

 

wkdb.ty/redis

What is Redis?

REmote DIctionary Server
wkdb.ty/redis

Why Redis?

Redis is an open-source, in-memory, persistent data storage engine that associates string keys with values using a variety of data structures and atomic operations.

 

 

toolbox outline icon toolbox outline icon illustration
wkdb.ty/redis

Why Redis?

Great Documentation => Happy Developer

 

wkdb.ty/redis

Why Redis

Redis is a great cache

wkdb.ty/redis

If you just used Redis as a NoSQL key-value store with key expiration as a cache, there is still significant value there.

 

Why Redis

  • PHP Session Handler
  • Server Clusters
  • Pub/Sub Message Brokering
  • Transactions & Command Pipeline
  • Lua Scripting
  • Extensibility with C Modules
  • High Availability with Redis Sentinel
  • Replication & Partitioning of Data
  • Access Control Lists
  • etc...
wkdb.ty/redis

Why Redis

php -r "echo 'Hello, World';"
redis-cli SET foo "Hello World"
redis-cli GET foo

Redis Does Not Let Its Features Get in Your Way

wkdb.ty/redis

Setup & Connect to Redis

1. Spin Up a Redis Server

2. Run Commands on the Command Line

docker exec redis-server redis-cli ping
docker run --rm --name redis-server -d -p 6379:6379 redis 
wkdb.ty/redis

Setup & Connect to Redis

3. Install PHP Extension with PECL

4. Instantiate and Connect the Redis Client

pecl install redis
$redis = new Redis();
$redis->connect('127.0.0.1');
$pong = $redis->ping();
var_dump($pong);
wkdb.ty/redis

Docker Compose Stack & Examples

wkdb.yt/redis
wkdb.ty/redis

RedisInsight

localhost:8001
wkdb.ty/redis

RedisInsight

localhost:8001
wkdb.ty/redis

Data Structures

A data structure defines how to organize a collection of data and what operations can be performed on it.

wkdb.ty/redis
wkdb.ty/redis

Data Structures

A single collection of data could be represented by several data structures, as needed to provide the most efficient operations for accessing and manipulating the data.

wkdb.ty/redis

Measuring Complexity

wkdb.ty/redis

Redis Data Structures

string
hash
list
set
sorted set
stream

Basic Structures

wkdb.ty/redis

Redis Data Structures

bitmap
geospatial
hyperloglog

Derived Structures

wkdb.ty/redis

Redis Data Structures

hash
list
set
sorted set
stream
=>
=>
=>
=>
=>
mapping
sequences
membership
ranking
log-like

Optimizations

wkdb.ty/redis

Keys

  • Keys are binary-safe strings
  • Maximum Key Size: 512 Mb
  • Maximum Number of Keys: 4,294,967,296
  • Keys can either persist or expire
  • Keyspacing is left up to you.
Key Value
user.12343.greeting Hello, Andy
user.12343.lastlogin 1566423852
import.532 {"import_id":53233,"import_time":"2019-08-23...
wkdb.ty/redis
wkdb.ty/redis

String

A singular value of any kind of binary data.

Value
Hello, Andy
greeting:andy
Value
50 4b 03 04 14 00 00 00 08 00 39 18 f9 4e 8f 59 95 1d...
files:archive.zip
Value
2503234
usercount
Value
{"host":"192.168.1.70","last_updated": "2021-10-16...
cache:config
  • Operations that get the length or a substring are highly optimized
  • Integer and float values are can be easily incremented/decremented
  • Each string value can be up to 512Mb in size
wkdb.ty/redis

String

$redis->set('foo', 'hello, world');
echo $redis->get('foo');
wkdb.ty/redis
hello, world

String

$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found

String: Get & Set

$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found

String: Debounce

if(!$redis->get('api_debouncer')){
    $redis->set('api_debouncer', true, [
        'nx',
        'ex' => 2,
    ]);

    // do api stuff...
}
wkdb.ty/redis

String: Lock

$quicklock = new class($redis) {
    public function __construct(private Redis $redis){}

    public function lock($name): string
    {
        $random_bytes = \random_bytes(16);
        $this->redis->set('lock.' . $name, $random_bytes, [
            'nx',
            'ex' => 2,
        ]);
        return $random_bytes;
    }

    public function unlock($name, $key = null): void
    {
        $lock = $this->redis->get('lock.' . $name);
        if (!$lock || $lock === $key) {
            $this->redis->del('lock.' . $name);
        }
    }
};
wkdb.ty/redis

Bitmap

wkdb.ty/redis

Act upon a String value as if it were packed binary data

  • Allows extremely compact storage of continuous or time-series data
  • Redis works with arbitrary bitfield sizes and does the math for you.
Value
111110011000001000000000000000000001...
user:23453:flags
Value
111110011000001
user:23454:flags
Value
...02 01 02 03 00 00 00 01 00 02 00 ...
user:23453:site:perminute:20210823

 unoptimized worse case: 1.44KB per user per day

Value
...000111101110000110101000001111101...
user:23453:site:engaged:20210823

 unoptimized worse case: 0.14KB per user per day

Bitmap

class CheckZipCode
{
    public function __construct(private Redis $redis)
    {
    }

    public function add(string $zip_code): void
    {
        $key = substr($zip_code, 0, 5);
        $offset = (int)substr($zip_code, 5, 4);
        $this->redis->setBit($key, $offset, 1);
    }

    public function check(string $zip_code): bool
    {
        $key = substr($zip_code, 0, 5);
        $offset = (int)substr($zip_code, 5, 4);
        return $this->redis->getBit($key, $offset);
    }
}
wkdb.ty/redis

Hash

wkdb.ty/redis

A collection of field and value pairs

Field Value
firstname
lastname
email
lastlogin
logincount
Andy
Snell
andy@example.com
2021-08-23T13:10:56+00:00
1232
user:12343
  • Can be used to map objects when the values need to be used
  • Hashes are like sub-key/value pairs and have string-like methods
  • Field lookups occur at constant time
  • Caution: hashes expire based on the key and are schema-less

Hash

// Set a field on a hash
$redis->hSet('import:33342', 'job_id', 12345); // 1
$redis->hSet('import:33342', 'import_count', 0); // 1

// Set multiple fields on a hash
$redis->hMSet('import:73723', ['job_id' => 234323, 'import_count' => 0]); // true

// Get the value of a hash field
$redis->hGet('import:73723', 'job_id'); // '234323'

// Get all the fields/values of a hash -- caution: order is not guaranteed.
$redis->hGetAll('import:73723'); // [ 'import_count' => '0', 'job_id' => '234323']

// Increment the value of a hash field 
$redis->hIncrBy('import:73723', 'import_count', 1); // 1
wkdb.ty/redis

List

wkdb.ty/redis

A collection of ordered values

  • A deque data structure, allowing push/pop operations from both sides
  • You can get a value by index, but probably should use a Hash instead
  • Lists can pop-push directly to other lists.
  • A connection can block pop or block push on multiple lists at a time 
Index Value
0
1
2
3
4
5
 
12324422
23423432
23432343
23432345
23423432
23423432
user:12343:next_action_id
Index Value
0
1
2
3
 
{"job_id":53233,"email":"andy@ex...
{"job_id":53234,"email":"jeff@p...
{"job_id":53235,"email":"megan@...
{"job_id":53236,"email":"sarah@...
 
emailqueue

List

// Add values to either side of the list
$redis->lPush('queued_email', 8976454); // 1 (length of list)
$redis->rPush('queued_email', 3423432); // 2
$redis->rPush('queued_email', 3423432); // 3 duplicates are ok
$redis->rPush('queued_email', 5675456); // 4
$redis->rPush('queued_email', 3423411); // 5

// Get the length of the list
$redis->llen('queued_email'); // 5

// Remove from either side of the list
$redis->lPop('queued_email'); // '8976454'
$redis->rPop('queued_email'); // '3423411'

// Remove from either side while blocking for up to 30 seconds
$redis->blPop('queued_email', 30); // '3423432'
$redis->brPop('queued_email', 30); // '5675456'

// Remove from right side of one list and push it onto a different list.
$redis->rpoplpush('queued_contacts', 'dequeued_email'); // 3423432
$redis->brpoplpush(queued_contacts, dequeued_email, 30); // false
wkdb.ty/redis

Set

wkdb.ty/redis

An unordered collection of unique values

Value
andy@example.com
me@example.net
andysnell@example.com
contact:435433:emails
  • Sets are used when membership and cardinality are important.
  • Operations return intersection, union, and difference of multiple sets
Value
andy@example.com
kevin@example.com
user:234329:emails
Value
andy@example.com
me@example.net
andysnell@example.com
kevin@example.com
union:emails
Value
andy@example.com
inter:emails
Value
me@example.net
andysnell@example.com
diff:emails

Set

$redis->sAdd('emails', 'andy@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 0 (already exists)

// Add more items to the set using an Array
$redis->sAddArray('emails', ['rachel@example.com', 'jc@example.com']); // true

// Get the cardinality (count) of a set
$redis->sCard('emails'); // 5

// Check if an element is a member of the set
$redis->sIsMember('emails', 'kevin@example.com'); // true
$redis->sIsMember('emails', 'jack@example.com'); // false

// Get all members of the set
$redis->sMembers('emails'); // [ array containing all the emails ]
wkdb.ty/redis

Set

// Get the intersection of multiple sets
$redis->sInter('emails', 'other_emails');
$redis->sInterStore('inter_list', 'emails', 'other_emails');

// Get the union of multiple sets
$redis->sUnion('emails', 'other_emails');
$redis->sUnionStore('union_list', 'emails', 'other_emails');

// Get the difference of multiple sets
$redis->sDiff('emails', 'other_emails');
$redis->sDiffStore('diff_list', 'emails', 'other_emails');
wkdb.ty/redis

Set

class CombinedPosts
{
    public function __construct(private Redis $redis) {}

    public function add(int $user_id, int $post_id): bool
    {
        $personal_key = "user:{$user_id}:posts";
        return (bool)$this->redis->sAdd($personal_key, $post_id);
    }

    public function check(int $user_id, int $team_id, int $post_id): bool
    {
        $personal_key = "user:{$user_id}:posts";
        $team_key = "team:{$team_id}:posts";
        $union_key = "temp:allposts:{$user_id}";
        $pipe = $this->redis->multi();
        $pipe->sInterStore($union_key, $personal_key, $team_key);
        $pipe->sIsMember($union_key, $post_id);
        $pipe->del($union_key);
        $result = $pipe->exec();
        return (bool)$result[1];
    }
}
wkdb.ty/redis

Sorted Set

wkdb.ty/redis

A collection of unique values ordered by a user defined score

Score Value
1
1
2
2
3
+13145551234
+13145553943
+13433431232
+19193332347
+12165553211
user:12343:callcount
Score Value
1
1
1
1
1
Adam
Beth
Charlie
Dan
Erica
user:12343:names
  • The score can be any 64-bit float/integer, including a Unix timestamp
  • When scores are identical, members are sorted lexicographically
  • Provides combined functionality similar to both Sets and Lists
  • Scores can be aggregated between set operations (min, max, sum)
Score Value
1566448434
1566448304
1566448134
1566448027
1566447927
+13145551234
+13145553943
+13433431232
+19193332347
+12165553211
user:12343:recentcalls

Sorted Set

class RedisRateLimit
{
    public function __construct(private Redis $redis) {}

    public function check(string $request_source_id, $requests_per_minute = 120): void
    {
        $key = "ratelimit:" . $request_source_id;

        $pipe = $this->redis->multi(Redis::PIPELINE);
        $pipe->zAdd($key, time(), bin2hex(\random_bytes(16)));
        $pipe->expire($key, 60);
        $pipe->zRemRangeByScore($key, 0, time() - 60);
        $pipe->zCard($key);
        $pipe_results = $pipe->exec(); // [1, true, 0, 1]
        if ($pipe_results[3] > $requests_per_minute) {
            throw new RateLimitExceeded();
        }
    }
}
wkdb.ty/redis

Stream

wkdb.ty/redis

Stream is a fairly recent addition to Redis

An append-only log data structure used for cooperative message stream consumption, with applications for event sourcing.

If the "pub/sub" functionality is a push of message data to the consumer,

using the stream type is a pull.

Conclusion

wkdb.ty/redis

If Redis is so great, should I replace my existing relational database with it?

Can you:

Maybe?

Should you:

Probably Not.

Conclusion

wkdb.ty/redis

Use the best tool you can for the job at hand.

 

Using Redis to its fullest, as a data structure server, gives you a toolbox from which to build solutions that work together with your existing infrastructure.

Please rate our talks...

joind.in/talk/83825
wkdb.ty/redis

Additional Resources

wkdb.yt/redis
wkdb.ty/redis

More Than Just a Cache

By Andy Snell

More Than Just a Cache

Redis is a popular key-value store, commonly used as a cache or message broker service. However, it can do so much more than just hold string values in memory! -- Redis is a full featured “data structure server”. As PHP developers, we typically don’t think about data structures other than our jack-of-all-trades array, but Redis can store hashes, lists, sets, and sorted sets, in addition to operating on string values. In this talk, we’ll explore these basic data structures in Redis and look at how we can apply them to solve problems like rate limiting, creating distributed locks, or efficiently checking membership in a massive set of data.

  • 249