Skip to content

[REDIS] Use SCAN instead of KEYS to find all matching keys#13405

Merged
markstory merged 10 commits into
cakephp:masterfrom
annuh:master
Jul 12, 2019
Merged

[REDIS] Use SCAN instead of KEYS to find all matching keys#13405
markstory merged 10 commits into
cakephp:masterfrom
annuh:master

Conversation

@annuh

@annuh annuh commented Jul 8, 2019

Copy link
Copy Markdown
Contributor

This PR changes the way how cache is cleared. First the inefficient KEYS command was used to find all matching keys in Redis.

According to the Redis docs, this doesn't perform well and they advice to use the SCAN command.

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using SCAN or sets.

Fixes: #13404

Used sample code from PhpRedis https://github.com/phpredis/phpredis#scan

Comment thread src/Cache/Engine/RedisEngine.php Outdated
@codecov

codecov Bot commented Jul 8, 2019

Copy link
Copy Markdown

Codecov Report

Merging #13405 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #13405      +/-   ##
============================================
+ Coverage     90.94%   90.94%   +<.01%     
- Complexity    13997    14000       +3     
============================================
  Files           523      523              
  Lines         36318    36323       +5     
============================================
+ Hits          33029    33034       +5     
  Misses         3289     3289
Impacted Files Coverage Δ Complexity Δ
src/Cache/Engine/RedisEngine.php 94.44% <100%> (+0.32%) 38 <0> (+3) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76de47e...d6cf2ed. Read the comment docs.

@codecov

codecov Bot commented Jul 8, 2019

Copy link
Copy Markdown

Codecov Report

Merging #13405 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #13405      +/-   ##
============================================
+ Coverage     90.94%   90.94%   +<.01%     
- Complexity    13997    14000       +3     
============================================
  Files           523      523              
  Lines         36318    36325       +7     
============================================
+ Hits          33029    33036       +7     
  Misses         3289     3289
Impacted Files Coverage Δ Complexity Δ
src/Cache/Engine/RedisEngine.php 94.56% <100%> (+0.44%) 38 <0> (+3) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c122d6...bb7b632. Read the comment docs.

Comment thread src/Cache/Engine/RedisEngine.php Outdated
@annuh

annuh commented Jul 10, 2019

Copy link
Copy Markdown
Contributor Author

I needed to remove the composer.lock file (in appveyor.yml) in order to fix issues with Appveyor (#13385)

Comment thread src/Cache/Engine/RedisEngine.php
@saeideng saeideng added this to the 3.8.1 milestone Jul 11, 2019
@saeideng saeideng added the cache label Jul 11, 2019
@markstory markstory merged commit 90440a6 into cakephp:master Jul 12, 2019
$result = [];
foreach ($keys as $key) {
$result[] = $this->_Redis->del($key) > 0;
$this->_Redis->setOption(Redis::OPT_SCAN, Redis::SCAN_RETRY);

@ADmad ADmad Aug 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is causing type errors in 4.x https://travis-ci.org/cakephp/cakephp/jobs/567855601#L822.
I am not sure what version of redis/predis travis is using. But even if it only occurs for older versions it could be an issue for our users.

@ADmad ADmad Aug 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the issue has been fixed phpredis/phpredis#1540. Not sure in which version.

@ADmad ADmad Aug 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix is available in 5.0.0. So essentially one won't be able to use Redis 4.x with Cake 4.0.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do (string)Redis::SCAN_RETRY typecast would help?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So essentially one won't be able to use Redis 4.x with Cake 4.0.

You mean the client library cannot be 4.x, not that you cannot use Redis 4.x.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@garas Maybe, I haven't checked.

@josegonzalez Yes, I should have worded it better.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Redis] Optimize Cache::clear by using SCAN instead of KEYS

6 participants