1 配置文件中的最大內(nèi)存刪除策略
在redis的配置文件中,可以設(shè)置redis內(nèi)存使用的最大值,當(dāng)redis使用內(nèi)存達(dá)到最大值時(shí)(如何知道已達(dá)到最大值?),redis會(huì)根據(jù)配置文件中的策略選取要?jiǎng)h除的key,并刪除這些key-value的值。若根據(jù)配置的策略,沒有符合策略的key,也就是說內(nèi)存已經(jīng)容不下新的key-value了,但此時(shí)有不能刪除key,那么這時(shí)候?qū)懙脑挘瑢?huì)出現(xiàn)寫錯(cuò)誤。
1.1 最大內(nèi)存參數(shù)設(shè)置
若maxmemory參數(shù)設(shè)置為0,則分兩種情況:
*在64位系統(tǒng)上,表示沒有限制。
*在32為系統(tǒng)上,是3G,redis官方文檔的說法是,32位系統(tǒng)最大內(nèi)存是4G,預(yù)留1G給系統(tǒng)。而且策略會(huì)自動(dòng)設(shè)置為noeviction。
也就是說在32位系統(tǒng)上,若maxmemory設(shè)置為0,則默認(rèn)是3G,當(dāng)?shù)竭_(dá)3G,再往reidis中寫入時(shí),則會(huì)報(bào)錯(cuò)。
1.2 到達(dá)最大內(nèi)存時(shí)的幾種刪除key的策略
* volatile-lru -> remove the key with an expire set using an LRU algorithm
按照LRU算法(最少最近沒使用)來選取,并刪除已經(jīng)設(shè)置了expire時(shí)間的key。
* allkeys-lru -> remove any key accordingly to the LRU algorithm
根據(jù)LRU算法,刪除任意的key。不論這個(gè)key是否設(shè)置了expire時(shí)間。
* volatile-random -> remove a random key with an expire set
隨機(jī)刪除一個(gè)設(shè)置了expire時(shí)間的key。
* allkeys-random -> remove a random key, any key
隨機(jī)刪除任意一個(gè)key,不論這個(gè)key是否設(shè)置了expire時(shí)間。
* volatile-ttl -> remove the key with the nearest expire time (minor TTL)
刪除具有最近終止時(shí)間值(TTL)的key。
* noeviction -> don't expire at all, just return an error on write operations
若沒有設(shè)置終止時(shí)間,返回一個(gè)錯(cuò)誤。
1.3 配置內(nèi)存最大值策略
以下這些命令的默認(rèn)策略是:volatile-lru
# At the date of writing this commands are: set setnx setex append
# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd
# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby
# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby
# getset mset msetnx exec sort
#
# The default is:
# maxmemory-policy volatile-lru
1.4 配置要?jiǎng)h除key的檢測(cè)樣本個(gè)數(shù)
maxmemory-samples
由于LRU和最小TTL算法都是不是精確的算法。因此你可以選擇要檢測(cè)樣本的個(gè)數(shù)。例如,默認(rèn)情況下redis將會(huì)檢查3個(gè)key,并從這3個(gè)key中選取一個(gè)最近沒有使用的key。當(dāng)然你可以修改檢查樣本的個(gè)數(shù)的值。
要修改這個(gè)值,可以通過在配置文件中設(shè)置參數(shù):
maxmemory-samples 3
2 實(shí)現(xiàn)
這幾種刪除策略的實(shí)現(xiàn)都是在函數(shù) freeMemoryIfNeeded(void) 中完成的。下面具體講解每種策略是如何實(shí)現(xiàn)的。
2.1 什么時(shí)候去刪除key-value
當(dāng)設(shè)置了maxmemory-policy策略后,什么時(shí)候會(huì)去刪除key呢?
實(shí)際上,當(dāng)設(shè)置了maxmemory參數(shù)后,在處理每個(gè)命令的時(shí)候都會(huì)根據(jù)maxmemory-policy去刪除對(duì)應(yīng)的key值。
代碼如下:
// 處理客戶端的每個(gè)命令,都會(huì)調(diào)用這個(gè)函數(shù)
int processCommand(redisClient *c) {
... ...
/* Handle the maxmemory directive.
*
* First we try to free some memory if possible (if there are volatile
* keys in the dataset). If there are not the only thing we can do
* is returning an error. */
// 以上意思是:若存在可以刪除的key,就釋放一些內(nèi)存,若不存在,給客戶端返回一個(gè)錯(cuò)誤。
if (server.maxmemory) { // 若maxmemory不為0,則調(diào)用以下函數(shù),釋放其中一些key
int retval = freeMemoryIfNeeded(); // 根據(jù)配置策略刪除key
if ((c->cmd->flags REDIS_CMD_DENYOOM) retval == REDIS_ERR) { // 若出錯(cuò),就終止處理命令,把錯(cuò)誤返回給客戶端
flagTransaction(c);
addReply(c, shared.oomerr);
return REDIS_OK;
}
}
... ...
}
實(shí)戰(zhàn)1:若沒有設(shè)置maxmemory變量,即使設(shè)置了maxmemory-policy,也不會(huì)起作用。
實(shí)戰(zhàn)2:若沒有設(shè)置maxmemory變量,在處理命令時(shí)將不會(huì)調(diào)用釋放策略,會(huì)加速命令的處理過程。
2.2 刪除key的總體流程
當(dāng)內(nèi)存達(dá)到最大值時(shí)需要按策略刪除老的key,所有的刪除操作和刪除策略的實(shí)現(xiàn)都是在函數(shù)freeMemoryIfNeeded()中實(shí)現(xiàn)的。
在執(zhí)行刪除策略之前,先要選取db和查找key。
總體步驟如下:
int freeMemoryIfNeeded(void) {
size_t mem_used, mem_tofree, mem_freed;
int slaves = listLength(server.slaves);
mstime_t latency;
/* Remove the size of slaves output buffers and AOF buffer from the
* count of used memory. */
mem_used = zmalloc_used_memory();
if (slaves) {
listIter li;
listNode *ln;
listRewind(server.slaves,li);
while((ln = listNext(li))) {
redisClient *slave = listNodeValue(ln);
unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave);
if (obuf_bytes > mem_used)
mem_used = 0;
else
mem_used -= obuf_bytes;
}
}
if (server.aof_state != REDIS_AOF_OFF) {
mem_used -= sdslen(server.aof_buf);
mem_used -= aofRewriteBufferSize();
}
/* Check if we are over the memory limit. */
// 檢查目前系統(tǒng)是否超過內(nèi)存的限制
if (mem_used = server.maxmemory) return REDIS_OK;
if (server.maxmemory_policy == REDIS_MAXMEMORY_NO_EVICTION)
return REDIS_ERR; /* We need to free memory, but policy forbids. */
/* Compute how much memory we need to free. */
mem_tofree = mem_used - server.maxmemory;
mem_freed = 0;
latencyStartMonitor(latency);
while (mem_freed mem_tofree) {
int j, k, keys_freed = 0;
// 遍歷16個(gè)數(shù)據(jù)庫
for (j = 0; j server.dbnum; j++) {
long bestval = 0; /* just to prevent warning */
sds bestkey = NULL;
struct dictEntry *de;
redisDb *db = server.db+j;
dict *dict;
// 這里要注意,若是ALLKEYS_xx策略,則直接在對(duì)應(yīng)庫結(jié)構(gòu)的dict中查找key。
// 若是非ALLKEYS_xx策略,也就是可能是 volatile-xxx等策略,操作的庫結(jié)構(gòu)將設(shè)置成expires結(jié)構(gòu)。
if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU ||
server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_RANDOM)
{
// 若設(shè)置了
dict = server.db[j].dict;
} else {
dict = server.db[j].expires;
}
// 若數(shù)據(jù)庫的大小為0,說明沒有key存在,繼續(xù)在下一個(gè)數(shù)據(jù)庫中查找
if (dictSize(dict) == 0) continue;
... ...
}
2.2 volatile-lru機(jī)制和allkeys-lru的實(shí)現(xiàn)
2.2.1 redis中的LRU機(jī)制
對(duì)于LRU機(jī)制,redis的官方文檔有這樣的解釋:
Redis LRU algorithm is not an exact implementation. This means that Redis is not able to pick the best candidate for eviction, that is, the access that was accessed the most in the past. Instead it will try to run an approximation of the LRU algorithm, by sampling a small number of keys, and evicting the one that is the best (with the oldest access time) among the sampled keys.
However since Redis 3.0 (that is currently in beta) the algorithm was improved to also take a pool of good candidates for eviction. This improved the performance of the algorithm, making it able to approximate more closely the behavior of a real LRU algorithm.
What is important about the Redis LRU algorithm is that you are able to tune the precision of the algorithm by changing the number of samples to check for every eviction. This parameter is controlled by the following configuration directive:
maxmemory-samples 5
The reason why Redis does not use a true LRU implementation is because it costs more memory. However the approximation is virtually equivalent for the application using Redis. The following is a graphical comparison of how the LRU approximation used by Redis compares with true LRU.
大意是說,redis的LRU算法不是正真意思上的LRU。而是使用另外一種方式實(shí)現(xiàn)的。也就意味著,redis并不能每次都選擇一個(gè)最好的key來刪除。沒有使用正真的LRU算法的原因是,它可能會(huì)消耗更多的內(nèi)存。該算法和正真的LRU算法效果大概相同。
redis是在一小部分key中選擇最優(yōu)的要?jiǎng)h除的key。這一小部分key的個(gè)數(shù)可以指定,可以在配置文件中設(shè)置參數(shù)maxmemory-samples 。
2.2.2 LRU機(jī)制的實(shí)現(xiàn)
freeMemoryIfNeeded()函數(shù),首先要計(jì)算最大空余內(nèi)存和目前已經(jīng)使用的內(nèi)存大差值,若不夠了,就要釋放老的key-value。
若使用的是LRU策略,就會(huì)走以下代碼,先進(jìn)行最優(yōu)刪除key的選擇,然后進(jìn)行刪除操作:
int freeMemoryIfNeeded(void) {
size_t mem_used, mem_tofree, mem_freed;
int slaves = listLength(server.slaves);
mstime_t latency;
/* Remove the size of slaves output buffers and AOF buffer from the
* count of used memory. */
mem_used = zmalloc_used_memory(); // 計(jì)算目前使用的內(nèi)存大小,要排除slave和AOF使用的buffer大小
if (slaves) { //遍歷slaves鏈表,減去slave使用的內(nèi)存數(shù)量
listIter li;
listNode *ln;
listRewind(server.slaves,li);
while((ln = listNext(li))) {
redisClient *slave = listNodeValue(ln);
unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave);
if (obuf_bytes > mem_used)
mem_used = 0;
else
mem_used -= obuf_bytes;
}
}
if (server.aof_state != REDIS_AOF_OFF) { //減去AOF使用的內(nèi)存大小
mem_used -= sdslen(server.aof_buf);
mem_used -= aofRewriteBufferSize();
}
/* Check if we are over the memory limit. */ //檢查是否達(dá)到設(shè)置的內(nèi)存上限
if (mem_used = server.maxmemory) return REDIS_OK;
// 不釋放內(nèi)存
if (server.maxmemory_policy == REDIS_MAXMEMORY_NO_EVICTION)
return REDIS_ERR; /* We need to free memory, but policy forbids. */
/* Compute how much memory we need to free. */ //計(jì)算要釋放的內(nèi)存量
mem_tofree = mem_used - server.maxmemory;
mem_freed = 0;
latencyStartMonitor(latency);
while (mem_freed mem_tofree) { //已經(jīng)釋放的內(nèi)存小于要釋放的內(nèi)存量
int j, k, keys_freed = 0;
for (j = 0; j server.dbnum; j++) { //遍歷所有數(shù)據(jù)庫開始釋放內(nèi)存
long bestval = 0; /* just to prevent warning */
sds bestkey = NULL;
struct dictEntry *de;
redisDb *db = server.db+j;
dict *dict;
// 這一步要先選擇淘汰取值的數(shù)據(jù)庫的dict
if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU ||
server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_RANDOM)
{ //若maxmemory-policy的值是LRU或RANDOM時(shí),直接在主數(shù)據(jù)庫中進(jìn)行淘汰
dict = server.db[j].dict;
} else { // 其他策略,在已經(jīng)設(shè)置了終止時(shí)間的key中間進(jìn)行淘汰。
dict = server.db[j].expires;
}
if (dictSize(dict) == 0) continue; //當(dāng)前數(shù)據(jù)庫沒有數(shù)據(jù)跳過
/* volatile-random and allkeys-random policy */ //若是RANDOM策略中的一個(gè)
if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_RANDOM ||
server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_RANDOM)
{
de = dictGetRandomKey(dict);
bestkey = dictGetKey(de);
}
/* volatile-lru and allkeys-lru policy */// 若刪除策略是LRU策略中的一個(gè)
else if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU ||
server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU)
{
// 根據(jù)配置文件中maxmemory_samples的值,決定做幾次選擇,刪除的key要從這些key中選出來。
for (k = 0; k server.maxmemory_samples; k++) {
sds thiskey;
long thisval;
robj *o;
// 從庫中隨機(jī)選取一個(gè)key-value結(jié)構(gòu)(dictEntry類型)的節(jié)點(diǎn)
de = dictGetRandomKey(dict);
thiskey = dictGetKey(de); // // 從該節(jié)點(diǎn)中獲取key的字符串地址
/* When policy is volatile-lru we need an additional lookup
* to locate the real key, as dict is set to db->expires. */
// 若最大內(nèi)存刪除策略是volatile-lru,則需要從db中查找thiskey。
// 若是VOLATILE-xx策略,則目前操作的庫的存儲(chǔ)結(jié)構(gòu)是expires,此時(shí)需要從dict中找到該key
if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU)
de = dictFind(db->dict, thiskey);
// 獲取key de的value值
o = dictGetVal(de);
// 查看該key的剩下的生存時(shí)間
thisval = estimateObjectIdleTime(o);
/* Higher idle time is better candidate for deletion */
// 每次都從遍歷的幾個(gè)Key中選出lru最長的key。
// 那么如何更新key的lru值呢?每次查找該key的時(shí)候就會(huì)更新該key的lru值,該值是系統(tǒng)的時(shí)間戳。
if (bestkey == NULL || thisval > bestval) {
bestkey = thiskey;
bestval = thisval;
}
}
}
/* volatile-ttl */
else if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_TTL) {
for (k = 0; k server.maxmemory_samples; k++) {
sds thiskey;
long thisval;
de = dictGetRandomKey(dict);
thiskey = dictGetKey(de);
thisval = (long) dictGetVal(de);
/* Expire sooner (minor expire unix timestamp) is better
* candidate for deletion */
if (bestkey == NULL || thisval bestval) {
bestkey = thiskey;
bestval = thisval;
}
}
}
... ...
// 到這里,要?jiǎng)h除的最優(yōu)key已經(jīng)選出來了?,F(xiàn)在進(jìn)入刪除階段。
// 不論哪種策略,只要選出了最優(yōu)key,就會(huì)執(zhí)行以下刪除流程。
/* Finally remove the selected key. */
if (bestkey) {
long long delta;
robj *keyobj = createStringObject(bestkey,sdslen(bestkey));
propagateExpire(db,keyobj);
/* We compute the amount of memory freed by dbDelete() alone.
* It is possible that actually the memory needed to propagate
* the DEL in AOF and replication link is greater than the one
* we are freeing removing the key, but we can't account for
* that otherwise we would never exit the loop.
*
* AOF and Output buffer memory will be freed eventually so
* we only care about memory used by the key space. */
// 刪除該bestkey對(duì)應(yīng)的key-value值。注意這里既要從dict中刪除,還要從expires中刪除。
delta = (long long) zmalloc_used_memory();
dbDelete(db,keyobj);
delta -= (long long) zmalloc_used_memory();
mem_freed += delta;
server.stat_evictedkeys++;
notifyKeyspaceEvent(REDIS_NOTIFY_EVICTED, "evicted",
keyobj, db->id);
decrRefCount(keyobj);
keys_freed++;
/* When the memory to free starts to be big enough, we may
* start spending so much time here that is impossible to
* deliver data to the slaves fast enough, so we force the
* transmission here inside the loop. */
if (slaves) flushSlavesOutputBuffers();
}
}
if (!keys_freed) {
latencyEndMonitor(latency);
latencyAddSampleIfNeeded("eviction-cycle",latency);
return REDIS_ERR; /* nothing to free... */
}
}
latencyEndMonitor(latency);
latencyAddSampleIfNeeded("eviction-cycle",latency);
return REDIS_OK;
}
以上這篇關(guān)于redis Key淘汰策略的實(shí)現(xiàn)方法就是小編分享給大家的全部內(nèi)容了,希望能給大家一個(gè)參考,也希望大家多多支持腳本之家。
您可能感興趣的文章:- Redis中LRU淘汰策略的深入分析
- 淺談redis的maxmemory設(shè)置以及淘汰策略
- 淺談Redis緩存有哪些淘汰策略