Variable storage

I looked around in Issues on Github and didn’t see anything like this, but it’s possible I missed it. I’m trying to see if it’s possible to have more than one storage per storage type in a single attachment.

For instance, in my case I’m using Wasabi which has an S3-compatible API. I know that I can use direct-to-S3 multipart uploads from the browser and save the completed file data to a Shrine column. But what if I dynamically upload to the closest bucket, such that users in the US would save direct to a US East bucket, and users overseas would save direct to an EU bucket?

This is just for the cache phase, as no matter where it gets uploaded initially, I’d be using one store location, promoting the file from one of two or three cache locations to one centralized store location. Utilizing dynamically chosen global buckets for the initial cache is just very helpful for speeding up large uploads around the world. It’s essentially replicating a push zone CDN.

Is this doable right now, such that cache storage can be variable to some extent?

Yes, you should be able to do that. Given that on the client side you’ve specified the correct storage identifier on the uploaded file data, in your controller you can set the desired cache storage on the attacher before assigning the file:

photo_params[:image] #=> '{"id":"...","storage":"cache","metadata":{...}}'

file = Shrine.uploaded_file(photo_params[:image])

# whitelist allowed storage identifiers for security
unless [:cache, :other_cache, ...].include?(file.storage_key.to_sym)
  fail "invalid storage identifier: #{file.storage_key.to_sym}"
end

photo.image_attacher(cache: file.storage_key.to_sym) # select temporary storage
photo.image = file.to_json # now this should work

Excellent! Maybe I’m misunderstanding, but wouldn’t the storage key always just be cache? For instance, when declaring an attachment on a model, you can specify an alternate store or cache than the default, but the value in the data that’s saved is still only either cache or store I believe. So in this case, if the data stored is just storing cache, how is it supposed to know which cache storage was used? On promotion won’t it just try to retrieve the file from whatever the attachment’s declared cache was in the model?

The storage key will match the registered storage identifier. You have to have the possible cache storage objects registered in Shrine.storages, then select the correct one based on the file data submitted to your app.

I’ll try to illustrate with a code example:

Shrine.storages = {
  cache:    Shrine::Storage::S3.new(region: "us-east-1",    **s3_options),
  cache_eu: Shrine::Storage::S3.new(region: "eu-central-1", **s3_options),
  store:    Shrine::Storage::S3.new(**s3_options)
}
photo_params[:image] #=> '{ "id": "...", "storage": "cache_eu", "metadata": { ... } }'

photo.image_attacher(cache: :cache_eu)
photo.image = photo_params[:image]

photo.image.storage_key.to_sym #=> :cache_eu
photo.image_data #=> '{ "id": "...", "storage": "cache_eu", "metadata": { ... } }'

Got it. I misunderstood how that worked. Thanks a ton!

By the way, I’ve just added a new multi_cache plugin to master, which will allow you to whitelist additional temporary storages upfront.

Shrine.plugin :multi_cache, additional_cache: [:cache_eu, ...]

So, it should help remove the additional boilerplate in the controller. It will be released with Shrine 3.0.

Very cool! That’s awesome. Thanks so much