Redis AOF - When It's Okay to Turn It Off for Performance

Redis is a powerful in-memory data store. While its speed is crucial, due to its "in-memory" nature, there is a risk of losing all data when the server restarts. To prevent this, Redis offers two persistence options.

1. RDB (Snapshotting): It takes a snapshot of all data at a certain point in time and saves it to disk.

2. AOF (Append Only File): It logs all write commands sequentially in a log file.

AOF provides strong durability compared to RDB, with almost no risk of data loss (usually within 1 second). However, recording every write operation to a file incurs a definite performance cost.

"Is AOF necessary for every Redis use case?"

In short, "No, not at all."

Today, we will clearly distinguish between when you don't need to use AOF and when it is essential.

When AOF Isn't Necessary

The core value of AOF is in data durability. If durability is not crucial for the data, AOF can become an unnecessary 'burden' that only degrades performance.

1. When Used as a Cache

This is the most typical case.

• Reason: The data in a cache is not the 'original'. There exists a separate 'source of truth' such as a database or another API.

• If data is lost: Even if Redis restarts and the cache is emptied, the application only experiences temporary slowness (Cache Miss), and can refill the cache by reading the original data again (Cache-Warm up). The data does not get permanently lost.

• Conclusion: For cache purposes, AOF and even RDB is often turned off. Eliminating disk I/O completely maximizes Redis's pure in-memory speed.

2. Celery Broker / Result Backend (Low Importance Tasks)

Redis is often used with Celery. The AOF setting in Redis as a Celery broker and result backend depends on the importance of the tasks.

• Reason: If the task being processed by Celery can 'fail without issue' or can be 'easily retried', then AOF is unnecessary.

• Examples:

  • User log analysis (losing a few seconds of logs is not a major issue)

  • Periodic data cleanup (it can simply be executed again in the next cycle)

  • Thumbnail image generation (the original image is available for regeneration)

• Conclusion: In such scenarios, if the Redis server goes down, tasks or results in the queue could be lost. However, if this loss can be 'tolerated' from a business perspective, turning off AOF to secure the broker's performance is a wise choice.

3. Simple Statistics and Real-Time Data (Volatile)

This applies when dealing with short-term trends or rapidly changing data.

• Reason: Data is updated very frequently, or a snapshot of a specific point in time holds little significance.

• Examples:

  • Number of website visitors in the last minute

  • Real-time game rankings (changing on a second-by-second basis)

  • Temporary session data (expires within a few minutes)

• Conclusion: The data is anyway going to disappear or be updated soon. Prioritizing the ability to handle many write requests quickly is much more important than losing a few seconds' worth of data. AOF is not suitable for such environments.

When AOF Is Strongly Recommended

Conversely, when losing Redis data would cause serious issues, AOF is not an option but a necessity.

1. 'Reliable' Message Queue (Critical Celery Jobs)

This is the exact opposite of the earlier Celery examples.

• Reason: The task itself is connected to money or core business logic.

• Examples:

  • Payment processing requests

  • Completion of user registration and sending authentication emails

  • Order processing

• Conclusion: If a user clicks the 'payment' button and Redis restarts, resulting in the loss of that task request, that would be a serious failure. In such scenarios, AOF should be activated with the fsync everysec (default) option, ensuring that tasks are written to disk as soon as they enter the queue.

2. When Using Redis as a Primary Database

This is when Redis is used not just as a simple cache but as the source of truth.

• Reason: There is no separate database, and Redis itself is storing the original data, so losing data means permanent loss.

• Examples:

  • User profile information

  • User sessions that need to be stored permanently

  • In-game currency (gold, item) information

• Conclusion: In this case, AOF is mandatory, and it should be used alongside RDB snapshots to establish backup and recovery strategies.

3. Transaction and Aggregation Data

This is when the consistency and accuracy of the data is very important.

• Reason: RDB saves a 'specific point' in time, meaning any data after the last snapshot may be lost. AOF saves 'all change history', which minimizes data loss (up to 1 second).

• Examples:

  • Follow/unfollow relationships

  • Accurate counting of votes

  • Financial data (simple deposits and withdrawals)

• Conclusion: In scenarios where final data consistency is crucial, the 'Append-Only' characteristic of AOF is far more suitable than RDB.

Conclusion: The Trade-off Between Durability and Performance

AOF is a powerful feature that provides durability to Redis. However, this feature comes at the cost of performance.

If you're unsure whether to turn AOF on or off, developers should consider the following question.

"If the Redis server unexpectedly crashes right now, would we face serious issues if data from the last few minutes (RDB) or 1 second (AOF) goes missing?"

• "No, we can recalculate (cache)." -> Turn off AOF.

• "Yes, the payment history would disappear." -> Make sure to turn on AOF.

As with all technical decisions, there is no definitive answer. It is vital to accurately understand the characteristics and requirements of your service's data and choose an appropriate trade-off.