redis Summary of knowledge points
Full name ：remoteDictionaryservice Remote Dictionary Service
Main application ： cache
Other applications ： According to remote storage 、 High performance can realize distributed lock 、

Its data structure -5 Kind of :String( character string )、list( list )、set( aggregate )、hash( Hashtable )、zset( Ordered set )

Redis All data structures are unique key String as name , And then through this one and only key To get the corresponding value（Map<String,Object>）

Redis The string of is a dynamic string , Is a string that can be modified , The internal structure is similar to Java Of ArrayList, Reduce the frequent allocation of memory by pre allocating redundant space , As shown in the picture , The internal space actually allocated for the current string capacity Generally higher than the actual string length len. When the string length is less than 1M when , Expansion is to double the existing space , If exceeded 1M, When expanding, it will only expand more at one time 1M Space . Note that the maximum string length is 512M.

It can be done to key Set expiration time , Delete automatically at point , This function is often used to control the cache expiration time .

Redis The list of is equivalent to Java Inside LinkedList, Notice that it’s a linked list, not an array . It means list The insert and delete operations of are very fast , The time complexity is O(1), But index positioning is slow , The time complexity is O(n). After the last element pops up in the list , The data structure is automatically deleted , Memory is recycled .Redis The list structure of is often used for asynchronous queues

Redis The dictionary of is equivalent to Java Inside HashMap, It’s an unordered dictionary . among Redis The value of the dictionary for can only be a string

When hash After removing the last element , The data structure is automatically deleted , Memory is recycled .hash Structure can also be used to store user information , Unlike strings, you need to serialize the entire object all at once ,hash Each field in the user structure can be stored separately . So when we need to get user information, we can do some
obtain . If the user information is saved in the form of an entire string, it can only be read all at once , This will waste network traffic .hash There are also shortcomings. ,hash Structure consumes more storage than a single string , To use hash Or string , It needs to be weighed again and again according to the actual situation .

Redis The set of is equivalent to Java In language HashSet, Its internal key value pair is unordered and unique .

zset May be Redis The most characteristic data structure provided , It’s also the interviewer’s favorite data structure in an interview . It is similar to Java Of SortedSet and HashMap The combination of , On the one hand, it is a set, Guaranteed the interior value Uniqueness , On the other hand, it can give each value Give one score, On behalf of the value Sort weight for . Its internal implementation uses a method called 「 Skip list 」 Data structure of .

list/set/hash/zset These four data structures are container data structures
1、createifnotexists
If the container does not exist , Then create a , Do it again
2、dropifnoelements
If there are no elements in the container , Delete the element now , Free memory
Expiration time
Redis Expiration time can be set for all data structures , time out ,Redis The corresponding objects are automatically deleted . It should be noted that expiration is in object units , For example, a hash Expiration of structure is the whole hash Expiration of objects , Not one of them key. Another important thing to note is that if a string has an expiration time set , And then you call set Method modified it , Its expiration time will disappear .

Application scenarios – Distributed lock
Through in business initiation is in redis Set a number in , Identification lock . And set the failure time . stay 2.8 Before the release, we need to introduce lock related lib. stay 2.8 After the version , Provides set Instructions . take setnx And expire combination . We need to pay attention to ：Redis Distributed locks should not be used for longer tasks . If it does happen occasionally , Count
According to the emergence of wavelet disorder may need to be solved by human intervention . Another way is through extra logical judgment key Whether the corresponding values are consistent
Application scenarios – Asynchronous message queuing 、 Delay queue
Redis Of list( list ) Data structures are often used as asynchronous message queues , Use rpush/lpush Operation in queue , Use lpop and rpop Come out of the line .blpop/brpop Block reading ; If the thread is stuck all the time ,Redis The client connection of becomes idle connection , Idle for too long , The server usually disconnects itself , Reduce the use of idle resources . This is the time blpop/brpop Throw an exception .

Redis As a message queue, why not guarantee 100% The reliability of the ？
Redis To do message queuing, using Publishing – A subscription model . It’s a one to many relationship , That is, a message will be consumed by multiple consumers . There is no guarantee that every consumer will receive a message , No, ACK Mechanism , Can’t ask the consumer to do it after receiving the message ACK confirm . If the message is lost 、Redis If some data is not persistent, even network jitter may lead to data loss .

The delay queue can pass Redis Of zset( Ordered list ) To achieve . We serialize the message into a string as zset Of value, The expiration processing time of this message is used as score, Then poll with multiple threads zset Get due tasks to process , Multiple threads are for availability

How to ensure that the task is performed only once ？
Redis Provide zrem Methods are the key to multithreading and multiprocessing , Its return value determines whether the current instance has preempted the task , because loop Methods can be used by multiple threads 、 Multiple process calls , The same task may be preempted by multiple process threads , adopt zrem To decide the only owner .

Application scenarios – Bitmap
A way of information compression to realize data storage identification , such as ： The information record of punch in . By using a bit To mark whether to punch in .

Bitmap is not a special data structure , Its content is actually ordinary string , That is to say byte Array .Redis The digit group of is automatically expanded , If an offset is set beyond the existing content range , Will automatically expand the bit array to zero .

Redis Provides bitmap statistics instructions bitcount And bitmap lookup instructions bitpos,bitcount It is used to count the number of 1 The number of ,bitpos Used to find the first… In the specified range 0 or 1, You can specify a range parameter [start,end], Unfortunately ,start and end Parameter is byte index , That is to say, the specified bit range must be 8 Multiple , It cannot be arbitrarily specified .

Application scenarios -HyperLogLog
HyperLogLog Provide an imprecise de counting scheme , Although imprecise, it is not very imprecise , The standard error is 0.81%
Single HyperLogLog Will occupy 12K Storage space , When the count is small , Its storage space uses sparse matrix storage , It takes up very little space , Just as the count grows , When the space occupied by sparse matrix gradually exceeds the threshold, it will be transformed into dense matrix at one time , Will occupy 12k Space .
Relevant command ：pfadd、pfcount、pfmerge

Application scenarios – The bloon filter

The bloon filter can be understood as an imprecise set structure , When you use it contains Method to determine whether an object exists , It could be misjudged . But the bloom filter is not particularly imprecise , As long as the parameter setting is reasonable , Its accuracy can be controlled relatively accurately , There is only a small probability of miscalculation . When the bloom filter says that a value exists , This value may not exist ; When it says it doesn’t exist , Then there must be no .

Redis4.0 after

bf.add,bf.exists,bf.madd,bf.mexists

bf.reserve There are three parameters , Namely key,error_rate and initial_size. The lower the error rate , The more space you need .initial_size The parameter represents the number of elements expected to be put in , When the actual quantity exceeds this value , The rate of miscalculation will rise . Therefore, it is necessary to set a large value in advance to avoid the increase of misjudgment rate . If not used bf.reserve, default error_rate yes 0.01, default initial_size yes 100.

principle ： Each bloom filter corresponds to Redis The data structure of is a large group of digits and a few different unbiased hash function .

Application scenarios – Simple current limiting

adopt zset data-structured score Circle the time window . And we just need to keep this time window , Data outside the window can be cut off . So this one zset Of value Use the millisecond timestamp .

Application scenarios – Funnel restriction
funnel The remaining space in the funnel represents the amount of current behavior that can continue , The flow rate of the leak represents the maximum frequency that the system allows for this behavior .
Redis4.0 after Redis-Cell modular ,cl.throttlekeyinit-sizemax_countdurationtokenNo example ：CL.THROTTLEtest100400603

Application scenarios -GeoHash

GeoHash The algorithm will continue to do this integer once base32 code (0-9,a-z Get rid of a,i,l,o Four letters ) It becomes a string . stay Redis Inside , Latitude and longitude use 52 The whole number of bits is encoded , Put it in zset Inside ,zset Of value It’s elemental key,score yes GeoHash Of 52 Bit integer value .zset Of score Although it’s a floating-point number , But for 52 Integer value of bit , It can store without damage .

Relevant command ：geoadd、geodist、geopos、geohash、georadiusbymember

Be careful ：Geo Using separate Redis Instance deployment , No cluster environment ; If the business is big . It needs to be split in the business layer .

Application scenarios -scan
keys No, offset、limit Parameters , Spit out all that meet the conditions at one time key.
keys The algorithm is a traversal algorithm , Complexity is O(n). because Redis It’s a single threaded program , Execute all instructions in sequence . It’s easy to have delays 、 Overtime .
scan The complexity is O(n), But it’s done step by step with cursors , Will not block threads ; Provide limit Parameters ; Provide pattern matching function ; The returned results may be repeated , Need client to repeat , This is very important ; If there is data modification during traversal , It is uncertain whether the modified data can be traversed ; Just because the result of a single return is empty doesn’t mean the end of the traversal , It depends on whether the returned cursor value is zero

redis How can single thread provide high concurrency ？
Through non blocking IO、 Event polling API（ Multiplexing ）. In addition, configure the instruction queue （ Used to receive instructions ）, Response queue （ For feedback results ）;

Redis The timing tasks of are recorded in a data structure called the minimum heap . In this pile , The most immediate tasks are at the top of the heap . In every cycle ,Redis All tasks that have arrived in the minimum heap will be processed immediately . After processing , Make a note of the time needed for the task to be performed , This time is select System called timeout Parameters . because Redis Know the future timeout Within time , There are no other scheduled tasks to deal with , So you can sleep at ease timeout Time for

RESP yes Redis Shorthand for serialization protocol . It’s an intuitive text protocol , The advantage is that it’s very easy to implement , Excellent parsing performance .

Redis There are two persistence mechanisms , The first is a snapshot , The second is AOF journal . Snapshot is a full backup ,AOF Log is a continuous incremental backup . Snapshot is a binary serialization of memory data , Very compact on storage , and AOF What the log records is the instruction record text of memory data modification .AOF The log will become huge in the long run , The database needs to be loaded when it is restarted AOF Log for instruction replay , This time will be very long . So it needs to be done on a regular basis AOF rewrite , to AOF Keep your weight down .

We know Redis It’s a single threaded program , This means that single thread requests on the service line at the same time have to be processed as files IO operation , file IO Operations can seriously degrade the performance of server requests . Another important problem is not to block the online business , You need to respond to client requests while persisting .Redis Multi process using the operating system COW(Copy On Write) Mechanism to achieve snapshot persistence .

glibc fork Generate a subprocess , Using the operating system COW Mechanism to separate data segment pages . Data segments are a combination of pages from many operating systems , When the parent process modifies the data of one of the pages , A copy of the shared page will be copied and separated , Then modify the copied page . At this time, the corresponding page of the subprocess does not change , Or the number of the moment when the process is generated .

Redis Provides bgrewriteaof Instructions are used for AOF Keep your weight down . Its principle is to open up a subprocess to traverse memory and convert it into a series of Redis Operation instructions of , Serialize to a new AOF Log file . Increment occurred during operation after serialization AOF The log is appended to this new AOF Log file , Replace the old one immediately after the addition AOF The log file is missing , The job of slimming is done .

Linux Of glibc Provides fsync(int fd) Function to force the contents of the specified file from kernel cache to disk .

Redis It looks similar in form , Namely multi/exec/discard.multi Indicates the beginning of the transaction ,exec Indicates the execution of the transaction ,discard Indicates the discard of the transaction .

Why? Redis Transaction cannot support rollback ？
Redis The command may fail in the transaction , however Redis Transactions do not roll back , It will continue to execute the remaining orders . If you have knowledge of a relational database , This may seem strange to you , Because relational data will roll back in this case .Redis To do so , Mainly because : Only when there is a syntax error ( This problem cannot be detected in the command queue ) 了 ,Redis The command fails , Or right keys Given a wrong type of data ： This means that these are procedural errors , This kind of error can be found and solved in the process of development , Hardly ever in a production environment . Because there is no need to roll back , This makes Redis The interior is simpler , And it runs faster .

Master slave synchronization
CAP principle （C:Consistent Uniformity ;A：Availability Usability ;P :Partition tolerance Zone tolerance ）

Network partition ： The nodes of distributed system are often distributed on different machines for network isolation , This means there is a risk of network disconnection .

 CAP The principle is —— When network partition happens , Consistency and availability cannot be guaranteed at the same time .
 
 Redis The master-slave data of is asynchronously synchronous , Not satisfied 「 Uniformity 」 requirement . When the client is in Redis After the master node modifies the data , Return immediately , Even when the primary and secondary networks are disconnected , The master node can still provide external modification services normally , Satisfy 「 Usability 」.Redis Guarantee 「 Final consistency 」.
 
redis Incremental synchronization is instruction flow , The instructions that the master node will modify are stored in local memory buffer in , Asynchronously synchronize it to the slave node . Feedback synchronization status and synchronization position from nodes . The master node uses a circular array to store instructions .

 Codis Use Go Language development , It’s a proxy middleware , It and Redis Also use Redis Agreement to provide external services , When the client Codis When sending instructions ,Codis Responsible for forwarding instructions to the following Redis Instance to execute , And return the result back to the client .Codis All that’s hanging on Redis Examples make up a Redis colony , When the cluster space is insufficient , It can be done by dynamically increasing Redis Instance to realize the expansion demand .
 
Codis How to make a specific key Forward to a specific Redis What’s the example? ？
Codis Will all key The default division is 1024 Slots (slot), It first passed to the client key Conduct crc32 Compute hash value , then hash The integer value after is equal to 1024 This integer is modulated to get a remainder , This remainder corresponds to key Slot of .

After expansion KEY transfer
Codis Yes Redis Have been transformed , Added SLOTSSCAN Instructions , Can traverse specified slot All of them key.Codis adopt SLOTSSCAN Scan out all the key, And then move each one… One by one key To the new Redis node . During the migration ,Codis You will still receive a new request to print on the currently migrating slot .Codis It is not possible to determine the key In which instance , So it takes a completely different approach . When Codis Received… In the migrating slot key after , Will immediately force the current single key Migration , After migration , Forward the request to the new Redis example .