Use backgrounding plugin only for some uploaders

#1

Hi!

At the moment I am using an uploader for profile pictures, and when a picture (avatar) is uploaded, two derivatives are created in background. A “medium” one for use on pages, and a “small” one for the navbar.

This is working fine. I also need to implement image uploads with Imperavi Article rich text editor (I’m building a CMS). In this case, I don’t think I can leverage backgrounding because otherwise the images would be referenced in the rich text as coming from the cache. So even if the derivates are created, loading that text would mean always loading images from the cache. Not only can those images be too big, the cache is also going to be cleared periodically.

Questions:

  1. is it possible for this specific image uploader to disable backgrounding and have derivatives created right away when uploading and make sure that the URL returned after uploading is already the URL for the “store” and not the cache? This way when the rich text is rendered I can be sure that the images are loaded from the store and not from the cache.

2.I’m going to try and limit the max file size when a user attaches an image to a text (not sure of how to do that with the Imperavi editor yet), but anyway I want users to be able to upload images of up to a few megabytes at least, for convenience, so that they don’t have to resize the images themselves each time before uploading. Then Shrine would resize the images to a smaller file size that can be more efficient for downloading when the rich text with the images is rendered. Is it possible to have Shrine store a derivative as the default image, instead of the original bigger image, in the “store” storage, so to avoid waste of space, and then return the URL of that image?

I’m still new to Shrine and am replacing ActiveStorage. So please be patient if I’m asking dumb questions :smiley:

Thanks!

#2

And… 3: I guess there is no point I waste time with direct upload to S3 if the image is going to be downloaded right away by the app for resizing, right? Does it make sense to have the files uploaded to the app first, then Shrine resizes them and uploads them to S3?

#3

And… 4 :smiley:

I am using DigitalOcean Spaces for the storage, and so far I’ve configured a Cloudflare hosted domain to point to DigitalOcean, so the images are loaded from Cloudflare, which in turn pulls them from DigitalOcean. However this means that the derivatives need to be stored in DigitalOcean already, which is one problem with the background jobs etc.

So I know that Shrine can do on-the-fly processing as well, and I could avoid processing derivatives upfront, so I was thinking that perhaps the Cloudflare domain (cdn.myapp.net) could point to the app and use OTF processing instead? Problem with this, from I’ve seen, is that the parameters for resizing are added to the URL, right? So if someone wants, they can write a script or something that makes tons of requests triggering lots of file processing which may end up in DoS.

I think on the fly processing may make some things easier, but how do I prevent that sort of DoS-like problem? Thanks!

#4

Interesting… I get Provided signature does not match the calculated signature if I try to just change the width and height from the URL. So I guess it’s not possible to perform that kind of attack?

#5

Hi :wave:

  1. Yes, the following should do it:

    class MyUploader < Shrine
  Attacher.promote_block(&:promote)
end

  2. If I understood correctly, you want to replace the main file with a resized version? This is not recommended, it’s always good to keep the original file in case you want to re-process later. That being said, this thread has some tips.

  3. The direct upload is still useful for the user experience, e.g. displaying a progress bar. That being said, you can choose to use FileSystem storage for :cache and use simple direct upload to upload_endpoint. That way your server will just read the local file when processing, which should generally be faster.

    However, FileSystem storage will only work if you’re using a single server, S3 will work with multiple servers.

  4. The derivation_endpoint signs its URLs, so it’s not possible for someone to change the URL parameters, as that would invalidate the URL. That’s mentioned in the “How it works” section of the plugin documentation.

#6

Hi!

yeah I’m using multiple servers. The signed URL thing sounds really cool. I am now using derivation_url and am passing the cdn host to the host option. The URLs generated to have the CDN host in them, however for some reason images load still slowly and it seems requests still go through my app. I have configured Cloudflare to cache for 1 year. Any idea of what could cause requests to still go through the app? Thanks!

#7

My guess is that there is something missing in your CDN settings. Or derivation_endpoint is not returning the HTTP headers that Cloudflare expects to trigger caching (I haven’t tested with Cloudflare specifically). The endpoint returns Cache-Control: public, max-age=#{365*24*60*60}, which I expected to be enough for any CDN, but you could check in the Cloudflare docs whether it is.

#8

If I make a CURL request to one derivation I get this: https://pastebin.com/raw/hibHqUgq

I wonder what that Connection state changed (MAX_CONCURRENT_STREAMS == 256)! means because the request seems to pause there for one second and then the rest of the headers are shown. Have you seen this before?

#9

BTW I’ve added Attacher.promote_block(&:promote) to the uploader and I get this error:

NoMethodError (undefined method `promote' for {}:Hash):
#10

BTW I’ve added Attacher.promote_block(&:promote) to the uploader and I get this error:

My bad, it needs to be

Attacher.promote_block { promote }

I wonder what that Connection state changed (MAX_CONCURRENT_STREAMS == 256)! means because the request seems to pause there for one second and then the rest of the headers are shown.

I’m not sure what might that mean. The request will pause before sending response headers because the uploaded file first needs to be opened from the storage (in S3 storage case the HTTP request to S3 needs to start) in order for the correct response headers to be generated.

However, I would expect the CDN not to even make the request to the app on the 2nd request.

#11

Yep the promote thing now works without errors.

I was able to fix the caching issue. I think it’s caused by the lack of extension in the file name when using the on-the-fly derivations. Is there any particular reason why it’s missing?

Cloudflare caches static content according to extensions, I think that’s why it wasn’t working. So to work around it I created a page rule which forces caching for everything under /derivations/*. It seems to work, the first request is of course slower but subsequent requests are instant and do not reach my app.

Hey thanks a lot for your help even on a Sunday! I should have listened to you on Reddit :slight_smile:
I started with ActiveStorage mainly because I was going to use ActionText and the two are integrated. But I ended up with issues with both. They are both limited and it doesn’t seem like the Rails team cares much about those issues, from reading Github threads. So I am now using Shrine instead of AS and Imperavi Article editor (which is a paid product) instead of AT.

Shrine has been quite a bit more code/work to set up… but it seems so much more complete than ActiveStorage will likely be for the foreseeable future in my opinion.

Now I need to figure out how to properly use it with the Imperavi editor with direct uploads and all the rest.

Thanks again :slight_smile:

1 Like
#12

I was able to fix the caching issue. I think it’s caused by the lack of extension in the file name when using the on-the-fly derivations. Is there any particular reason why it’s missing?

Shrine doesn’t know which format the result of the derivation will be in, so it doesn’t put any extension. It’s possible we’ll add support for adding filenames to the URL, like Active Storage has, but this hasn’t been requested yet.

Glad you were able to determine the issue :slightly_smiling_face:

Shrine has been quite a bit more code/work to set up… but it seems so much more complete than ActiveStorage will likely be for the foreseeable future in my opinion.

Yeah, Shrine assumes a lot less about your environment and setup, and requires you to string together different pieces. I’m glad you find it more complete :slightly_smiling_face: