I have rencently created an uploader allowing my users to upload up to 50 images at once into Uppy, presigned into Rails and shrine, uploaded to S3, and then the image details passed back to the Rails controller.
The Rails controller fire each image into a Sidekiq worker and the results are broadcasted back to the user through Actioncable.
Sidekiq worker looks like this :
It works perfectly in development but when more than a few images are dropped by the user, the RAM usage on my Heroku Dyno (512MB) skyrockets from 150MB to more than 512MB and sometimes up to more than 1GB and crashes the Sidekiq Dyno.
The Heroku Dyno restarts and finishes the job, sometime dropping one image.
The Sidekiq Dyno is set up with only one sidekiq thread. So files are basically processed one after the other.
I was wondering if there were obvious code smells that I could check easily ? Also I am not using Backgrounding plugin as it works finepassing the image details to the Sidekiq worker directly …
(I did the usual tricks to save RAM like MALLOC_ARENA_MAX or Jemalloc which have slightly improved the upload but problems still persists in production. Also the uploader use VIPS )