Ram usage on Heroku

I have rencently created an uploader allowing my users to upload up to 50 images at once into Uppy, presigned into Rails and shrine, uploaded to S3, and then the image details passed back to the Rails controller.

The Rails controller fire each image into a Sidekiq worker and the results are broadcasted back to the user through Actioncable.

Sidekiq worker looks like this :

if @image.valid?

Actioncable.broadcast …


It works perfectly in development but when more than a few images are dropped by the user, the RAM usage on my Heroku Dyno (512MB) skyrockets from 150MB to more than 512MB and sometimes up to more than 1GB and crashes the Sidekiq Dyno.
The Heroku Dyno restarts and finishes the job, sometime dropping one image.
The Sidekiq Dyno is set up with only one sidekiq thread. So files are basically processed one after the other.

I was wondering if there were obvious code smells that I could check easily ? Also I am not using Backgrounding plugin as it works finepassing the image details to the Sidekiq worker directly …

(I did the usual tricks to save RAM like MALLOC_ARENA_MAX or Jemalloc which have slightly improved the upload but problems still persists in production. Also the uploader use VIPS )

Have you done some memory profiling? I would recommend MemoryProfiler gem, I’ve used it a lot in the past.

I’ve spent a lot of time reducing memory usage in anything Shrine-related, so it’s difficult to tell just from this information where the increased memory allocations are coming from. It could be from downloading original & uploading derivatives, these will allocate strings in the process, but both Down and aws-sdk-s3 should be deallocating those strings. Also, there were some memory optimizations in net/http (which aws-sdk-s3 uses) in recent Ruby versions, I’m not sure if you’re running at least Ruby 2.6.

Thanks Janko.
I wasn’t using Ruby 2.6 but it is now fixed (2.6.6)
Also I have installed MemoryProfiler, MiniProfiler and DerailedBenchmarks and experencing a bit with it but I need to really dive in it to get consistent results.
What I am sure is that it is not a leak. Memory comes back to (nearly) initial level in Heroku, and localy it comes back to initial value (by monitoring HTOP while files are uploaded, attached and versionned).
I come back to you (or the discourse more generally) when I know more