It's been a little while since I've had time to write. Work and other committments have been getting in the way for the last few months, but I'm back at it now, and today I'm going to describe a little bit about my experience with Upstash and Redis Cloud.
Working with Redis
First off, a bit of context. What is Redis?
Redis is a simple key-value database that is optimised for speed in terms of reading data. It's often used in applications that need to be low-latency (meaning that return results quickly without much of a delay) as a cache. Using the key to look up a certain piece of data and returning the value stored next with it. It's ultra fast at simple operations.
I haven't yet delved into the world of Redis. There's a lot to learn and a whole load of commands that do various things. I don't know the full extent of it, so I couldn't really give much more detail than that. It's open source and the Redis community is quite large, you can read more about it on the Redis website.
One use case of Redis in particular has also become quite common, at least in the Ruby on Rails world, which is to handle background jobs. For this, it's used in combination with a gem called Sidekiq, which I wrote about previously. Even though it's not the only tool that can do the job, for example Resque is a common alternative, it's become a mainstay in the Gemfile of many Rails developers.
I gained some experience with using these two tools in combination while building the web application for Joinative in 2019/2020. There I employed them for processing background jobs, and it was a similar problem that I wanted to solve here.
The feature
The feature I wanted to create was a reminder system for Forever Diary, an online journaling tool that I built last year and continue to develop to this day. To make journaling as easy as possible for the users, they can configure email reminders to update their entry. The issue? For this, I needed to send emails in a background process. Rails is set up well to handle this scenario, with ActiveJob doing a pretty good job of supporting it out of the box, but you still need the process to run the background jobs and actually send the emails. This means:
- a) creating a Sidekiq worker which runs a separate process to the server
- b) creating a Redis database
Very briefly, the way this works is the Rails application calls a Sidekiq worker (a simple Ruby class with a perform method), this adds an entry into Redis database to be read out at a future date. The Sidekiq worker process then checks the Redis database on a regular basis (every few seconds) to see if there are any unprocessed jobs. If there are, it pulls out the details and executes them.
With the Forever Diary application already hosted on fly.io, a developer-friendly hosting provider that has done a good job at replacing Heroku as my go-to provider since Heroku shut off its free tier, I first looked for an out-of-the-box solution that would work with my existing setup.
There I found Upstash. With a free tier and a seamless Fly integration, it looked like the perfect fit. I got it all set up in minutes and tested my production emails. It worked! What had seemed like a difficult feature in my mind was actually very simple, and the deployment was done in no time.
The issue
After the deployment, I made the cardinal sin of not diligently checking back in with the email reminders following the release. It wasn't until a few days later that I realised the reminder I'd set on my account wasn't being delivered. I dug into it.
I was reaching the daily limit of the Redis usage on Upstash within a few hours and for the rest of the day it was just throwing errors causing the Sidekiq worker process to stop working. The monitoring on Fly's dashboard is quite good and makes problems quite easy to diagnose if you actually bother to check it!
I saw quickly that the limit was being reached every day, but I wasn't quite sure how that was happening. My knowledge of the internal workings of Sidekiq and Redis is not great, so I started naively digging around on Google and Stack Overflow to see whether others had faced this issue and what they'd done to fix it - the Upstash documentation was not very illuminating on this issue, so I was pretty much on my own.
I found a few interesting threads with suggestions from other developers who had tried to combat the issue. Increasing the poll interval for Sidekiq (the number of seconds, on average, between one check of the database and the next). I figured this might help to reduce the load... I was wrong. The following day, the same issue came up. I tried a few other configuration changes, but none of them worked.
The real underlying issue, it turns out, is that the Upstash limit that I was reaching is not on data stored, connections, bandwidth or anything like that, but rather total number of commands. Commands, from Redis' perspective, seem to be executed frequently from Sidekiq. They could be as simple as checking that the database still exists, checking whether a key is in the database, writing a record, reading a record, starting a transaction and committing a transaction. Each of these individual actions count as a single command to the database. Each time Sidekiq checks for a new job, it sends 10-15 commands to Redis. So, with Sidekiq checking for jobs once, twice or up to 12 times per minute, it's clear that the 10,000 daily command limit that Upstash has will be quickly exhausted. Very little tends to happen between midnight and 5am in the application, so the Redis database was pretty much useless, and along with it the Sidekiq background jobs.
The solution
The solution was simple in principle. Even though Fly integrates the Redis database very easily with Upstash, configuring an external Redis database (one hosted elsewhere) was not too difficult.
I opted for Redis Cloud. Redis Cloud also provides a free tier, which, most usefully, doesn't have any command limit at all. It has a data limit, but one that Forever Diary won't be approaching too soon.
After creating the new Redis database, getting the URL and the security credentials and configuring the change in the Rails application, it turned out that the version of Redis running on Redis Cloud is incompatible with the latest version of Sidekiq (version 7), so I had to downgrade it.
But that was it! With my newly minted Redis database churning away, and Sidekiq merrily executing all the background jobs from the Redis database, the application was ready to go and the reminders functionality works perfectly. It's the same integration that allows for welcome emails to be sent, and for email updates to be sent to the admin team of Forever Diary, meaning we never miss any of the goings-on. Job done.
If you want to check out Forever Diary, sign up for a free account and reap the mental health benefits that daily journaling can bring. The team and I are always open to new feature requests, feedback and questions. Don't hesitate to get in touch.