Shrine

Gotcha on s3, upload_options, and tags

I was trying to use the upload_options plugin with an S3 storage, to dynamically add some S3 tags based on the specific file – say, a mime_type tag. Using the tagging option to S3.

Because I use a “direct upload to cache” architecture, there aren’t any tags applied to cache, and the first time ruby code has the opportunity to intervene is on promotion to storage.

I could not figure out why it was not working! After a bunch of debugging, I realized that:

  • Shrine S3 would use the copy_from method on promotion, if copying from S3 to S3, same AWS account.
  • The S3 copy_from operation would ignore tagging option supplied with new tags, instead insisting on copying tags from source to destination (in my case, that’s no tags at all.).
  • UNLESS you also pass tagging_directive: "REPLACE" option.
    • This was hard to figure out, because none of this, including the tagging_directive option, is mentioned in the AWS SDK method-level documentation for the copy_from method! Grr! It is oddly mentioned in the AWS::S3::Client#copy_object docs though, and turns out to apply to AWS::S3::Object.copy_from too.

So, okay, once I figured that out, I could just add tagging_directive: "REPLACE" to my upload_options and it worked. A bit tricky to figure out, so I post it here in case someone else running into troubles might find it. You might need it on upload_options passed directly to the storage or even directly to a put or upload method, in addition to my use of the upload_options plugin.

But also…

I notice that Shrine S3 sets a similar metadata_directive: "REPLACE" option automatically when doing a copy, to avoid this same problem with metadata.

It turns out this doesn’t cover tagging, tagging needs it’s own tagging_directive: "REPLACE". I’m thinking that probably the S3 adapter should just set this for tagging_directive too?

Since whether Shrine actually uses an AWS copy operation or not can depend on a number of factors including size of file… it should probably just never try to copy tags, like it never tries to copy metadata?

If someone agrees, I’ll PR it!

And actually… adding the tagging_directive param to upload_options myself isn’t a good solution!

Because that’s an illegal param for AWS SDK in some cases it winds up getting applied, it’s only legal on a copy operation!

I have PR’d. https://github.com/shrinerb/shrine/pull/569