Rate limiting mirroring jobs

Hi! I have a question about the mirroring plugin.

At the moment I am using Scaleway 's S3 compatible storage service for user uploads because it’s cheap etc. However I have just installed Minio in my Kubernetes cluster and would like to use that instead as primary storage for performance and because Scaleway (like the other S3 compatible services) has a limit to the number of req/sec that can be made for a bucket.

Since my app can potentially generate lots of user uploads, I am planning for the future. In Kubernetes I can easily resize disks for Minio as needed so scaling shouldn’t be a big issue at least up to a point.

However I would like to still use Scaleway as a mirror/backup just in case, but if the mirroring is done in realtime I could still hit the aforementioned limit. I can see that Shrine can use background jobs to do the mirroring, but is it possible to rate limit these operations so to prevent hitting that limit?

Thanks in advance!

You can implement rate limiting at the level of your background workers. I don’t know which library you’re using, but Sidekiq Enterprise has a rate limiting feature. In my previous job we’ve implemented a Redis-based concurrency limiter, and it worked ok. I expect there are also several Ruby libraries for rate limiting.

Hi, I saw the Enterprise option but it’s expensive :stuck_out_tongue:

I’ll see if I can find an alternative with Sidekiq if there is no way to handle this in Shrine. Thanks

Just added the mirroring, works great! Thanks for this fantastic library :slight_smile:

1 Like