Derivatives are present in data field but in UploadedFile object it's missing

I’ve been working on a task to “Move away from Paperclip to Shrine”, but facing a issue (defiantly silly one, cause no one maybe encountered it).

Right now we are using paperClip in S3 storage. So, I’ve migrated all paperClip data in <field>_data field and it’s done successfully (and really quite happy with it, super simple and secure).

And I’ve checked all PaperClip styles are present in Shrine data field as json data as derivatives. But when I hit Uploader::UploadedFile object, derivatives are missing there. And naturally specific versioned image are nil.

Could you guys please help me, what am I missing?
Thanks!

Hmm, that is strange, it looks like it should work :thinking:

Could I see your Shrine setup and any code you think might be relevant? Also, could you create a self-contained example that reproduces the problem? I’m asking because when I try to set this up myself, fetching the derivative works for me:

require "shrine"
require "shrine/storage/memory"

Shrine.storages = {
  cache: Shrine::Storage::Memory.new,
  store: Shrine::Storage::Memory.new,
}

Shrine.plugin :model
Shrine.plugin :derivatives

class Photo
  include Shrine::Attachment(:image)
  attr_accessor :image_data
end

photo = Photo.new
photo.image_data = JSON.generate(
  id: "original",
  storage: "store",
  metadata: { "size" => 123, "filename" => "image.png", "mime_type" => "image/png" },
  derivatives: {
    high_resolution:     { id: "high_resolution.png",     storage: "store", metadata: {} },
    standard_resolution: { id: "standard_resolution.png", storage: "store", metadata: {} },
    low_resolution:      { id: "low_resolution.png",      storage: "store", metadata: {} },
    thumb:               { id: "thumb.png",               storage: "store", metadata: {} },
  }
)

p photo.image_data
# "{\"id\":\"original\",\"storage\":\"store\",\"metadata\":{\"size\":123,\"filename\":\"image.png\",\"mime_type\":\"image/png\"},\"derivatives\":{\"high_resolution\":{\"id\":\"high_resolution.png\",\"storage\":\"store\",\"metadata\":{}},\"standard_resolution\":{\"id\":\"standard_resolution.png\",\"storage\":\"store\",\"metadata\":{}},\"low_resolution\":{\"id\":\"low_resolution.png\",\"storage\":\"store\",\"metadata\":{}},\"thumb\":{\"id\":\"thumb.png\",\"storage\":\"store\",\"metadata\":{}}}}"
p photo.image
# #<Shrine::UploadedFile storage=:store id="original" metadata={"size"=>123, "filename"=>"image.png", "mime_type"=>"image/png"}>
p photo.image(:high_resolution)
# #<Shrine::UploadedFile storage=:store id="high_resolution.png" metadata={}>

Sure… (yeah it’s quite strange and already spent couple of hours)
My paperClip s3 config:

application.rb

    config.paperclip_defaults = {
      storage: :s3,
      s3_protocol: 'https',
      s3_credentials: Rails.application.secrets.s3_credentials,
      url: ':s3_alias_url',
      s3_host_alias: Rails.application.secrets.s3_host_alias,
      s3_region: Rails.application.secrets.s3_region,
    }

config/initializers/shrine.rb

    # frozen_string_literal: true

    require "shrine"
    require "shrine/storage/s3"
    require "shrine/storage/file_system"

    Shrine.storages = {
      cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"), # temporary
      store: Shrine::Storage::S3.new(
               bucket: Rails.application.secrets.s3_credentials[:bucket], # required
               region: Rails.application.secrets.s3_region, # required
               access_key_id: Rails.application.secrets.s3_credentials[:access_key_id],
               secret_access_key: Rails.application.secrets.s3_credentials[:secret_access_key],
              )
    }

    Shrine.plugin :activerecord           # loads Active Record integration
    Shrine.plugin :cached_attachment_data # enables retaining cached file across form redisplays
    Shrine.plugin :restore_cached_data    # extracts metadata for assigned cached files

    Dir["#{Rails.root}/app/uploaders/*.rb"].each { |file| require file }

custom_photo_uploader.rb

    # frozen_string_literal: true

    require "image_processing/mini_magick"

    class CustomPhotoForPostUploader < Shrine
      plugin :keep_files
      plugin :default_url
      plugin :validation_helpers
      plugin :derivatives
      plugin :pretty_location

      Attacher.default_url do |derivative: nil, **|
        'https://dak7yvjkjihidn8i2d8.cloudfront.net/default_images/icon.png' if derivative
      end

      Attacher.derivatives_processor do |original|
        magick = ImageProcessing::MiniMagick.source(original)

        if %w[video/mp4].include? file.mime_type
          {
            high_resolution: { geometry: magick.resize_to_limit!(1080, nil), format: 'jpg', frame_index: 10 },
            standard_resolution: { geometry: magick.resize_to_limit!(640, nil), format: 'jpg', frame_index: 10 },
            low_resolution: { geometry: magick.resize_to_limit!(320, nil), format: 'jpg', frame_index: 10 },
            thumb: { geometry: magick.resize_to_limit!(150, 150), format: 'jpg', frame_index: 10 },
          }
        else
          {
            high_resolution: magick.resize_to_limit!(1080, nil),
            standard_resolution: magick.resize_to_limit!(640, nil),
            low_resolution: magick.resize_to_limit!(320, nil),
            thumb: magick.resize_to_limit!(150, 150),
            original: magick.resize_to_limit!(2000, 2000),
          }
        end
      end

      Attacher.validate do
        validate_mime_type %w[image/jpeg image/gif image/png image/jpg video/mp4]
        validate_max_size 50*1024*1024 if file.mime_type.include? "video"
      end

      def generate_location(io, record: nil, derivative: nil, **)
        return super unless record

        table  = record.class.table_name
        id     = record.id
        prefix = derivative || "original"

        "uploads/#{table}/#{id}/#{prefix}/#{super}"
      end
    end

I think somehow I’ve missed in custom_photo_uploader.rb. Please let me know if anything else, I should share. Any help will be appreciated.
Thanks a lot!

Nothing looks out of the ordinary to me. If you’re getting this error for attachments migrated from Paperclip, then the derivatives/validation/location configuration probably doesn’t even matter, as that code is used only for new attachments uploaded by Shrine.

  1. Were you able to produce an isolated Ruby script that reproduces the bug, like the one I’ve shown? This would guarantee that I would identify the bug. Can you tell what might be different in that script compared to your code?

  2. Are you getting the missing derivative when you load persisted records from the database? That should make it easier to debug, as you can just copy-paste the attachment JSON data you’ve shown into a self-contained script. I cannot do so because you’ve posted it as an image :wink:

  3. When you call p.custom_photo_derivatives, do you get the hash of derivatives back, or you get an empty hash instead?

  4. When you reload the record (p.reload), are you able to retrieve the derivative afterwards?

Any additional information would be useful. Honestly, I highly doubt this is a bug in Shrine, as the derivatives plugin has been thoroughly tested.

Yeah, I’m quite sure it’s not a bug of Shrine but I definitely missing something to configure.

  1. No, I wasn’t able to product any bug! I’ve used my real data (custom_photo_data) and it’s works well. Difference is I’m using custom uploader and including it to the model (e.g. include CustomPhotoForPostUploader::Attachment(:custom_photo)).

  1. Yeah, that is my issue. I’ve migrated all data using include PaperclipShrineSynchronization and data successfully persisted on custom_photo_data field. And when I’m going to retrieve specific versioned image, I’m getting nil. Here is the content of custom_photo_data:
>> p.custom_photo_data
=> "{\"id\":\"/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png\",\"storage\":\"store\",\"metadata\":{\"size\":141207,\"filename\":\"Screenshot_2020-01-10_at_10.51.03.png\",\"mime_type\":\"image/png\"},\"derivatives\":{\"high_resolution\":{\"id\":\"/posts/6534/high_resolution/284dc222bf965ba93038b02266598393ea7cd323.png\",\"storage\":\"store\",\"metadata\":{}},\"standard_resolution\":{\"id\":\"/posts/6534/standard_resolution/62bb7d7395e9fcf0cc5fe5596fd1e88af5cc1806.png\",\"storage\":\"store\",\"metadata\":{}},\"low_resolution\":{\"id\":\"/posts/6534/low_resolution/5e99d63846fa8a8afeccd4630873d22a7933a4ae.png\",\"storage\":\"store\",\"metadata\":{}},\"thumb\":{\"id\":\"/posts/6534/thumb/26285bb3f8daa99dba1b940ad608544450af1776.png\",\"storage\":\"store\",\"metadata\":{}},\"original\":{\"id\":\"/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png\",\"storage\":\"store\",\"metadata\":{}}}}"
  1. Here is the output:
>> p.custom_photo_derivatives
=> {} 
  1. I reloaded the record and there is no difference.

Most importantly, it’s really strange case. This problem I’m facing just for migrating data. I’ve uploaded a file (using Shrine) and it works absolutely fine with derivatives.

It’s really great that you are guiding me to resolve it. Thank you so much!

So, when you’re loading a record from the database which has attachment data migrated from Paperclip, that record has empty derivatives, but when you’re loading a record from a database which has attachment data written directly by Shrine, derivatives are retrieved? And there is no visible difference between the two formats?

That’s really a mystery then. When you’re loading records from the database, Shrine just parses what’s in the column, it doesn’t matter how that data was written. There must be some state corruption happening somewhere. You’re using a text column, right?

When you execute the following, does it manage to successfully retrieve derivatives?

attacher = CustomPhotoForPostUploader::Attacher.from_model(post, :custom_photo)
p attacher.file
p attacher.derivatives

# OR

attacher = CustomPhotoForPostUploader::Attacher.from_column(post.custom_photo_data)
p attacher.file
p attacher.derivatives

Here is the output:

 attacher = CustomPhotoForPostUploader::Attacher.from_model(post, :custom_photo)
 => #<CustomPhotoForPostUploader::Attacher:0x00007ffd5f7f1070 @file=#<CustomPhotoForPostUploader::UploadedFile storage=:store id="/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png" metadata={"size"=>141207, "filename"=>"Screenshot_2020-01-10_at_10.51.03.png", "mime_type"=>"image/png"}>, @cache=:cache, @store=:store, @context={:record=>#<Post id: 6534, data_type: "image", source: "file", source_id: "7-1578887671674-Screenshot 2020-01-10 at 10.51.03....", source_user_id: nil, created_time: "2020-01-13 03:54:31", thumbnail: nil, low_resolution: nil, standard_resolution: nil, link: nil, caption: nil, created_at: "2020-01-13 03:54:31", updated_at: "2020-01-25 03:15:13", source_user_picture: nil, custom_photo_file_name: "Screenshot_2020-01-10_at_10.51.03.png", custom_photo_content_type: "image/png", custom_photo_file_size: 141207, custom_photo_updated_at: "2020-01-13 03:54:31", shop_id: 7, custom_cover_photo_file_name: nil, custom_cover_photo_content_type: nil, custom_cover_photo_file_size: nil, custom_cover_photo_updated_at: nil, custom_photo_processing: true, custom_cover_photo_processing: nil, video_low_resolution: nil, video_standard_resolution: nil, carousel_media: {}, source_user_email: nil, photo_request_id: nil, is_customer_post: false, instagram_account_id: nil, custom_photo_data: "{\"id\":\"/posts/6534/original/993d5cdb9a410ed7ff8310...", custom_cover_photo_data: nil>, :name=>:custom_photo}, @previous=nil, @column_serializer=Shrine::Plugins::Column::JsonSerializer, @model_cache=true, @model=true, @derivatives={}, @derivatives_mutex=#<Thread::Mutex:0x00007ffd5f7f0c88>, @errors=[], @record=#<Post id: 6534, data_type: "image", source: "file", source_id: "7-1578887671674-Screenshot 2020-01-10 at 10.51.03....", source_user_id: nil, created_time: "2020-01-13 03:54:31", thumbnail: nil, low_resolution: nil, standard_resolution: nil, link: nil, caption: nil, created_at: "2020-01-13 03:54:31", updated_at: "2020-01-25 03:15:13", source_user_picture: nil, custom_photo_file_name: "Screenshot_2020-01-10_at_10.51.03.png", custom_photo_content_type: "image/png", custom_photo_file_size: 141207, custom_photo_updated_at: "2020-01-13 03:54:31", shop_id: 7, custom_cover_photo_file_name: nil, custom_cover_photo_content_type: nil, custom_cover_photo_file_size: nil, custom_cover_photo_updated_at: nil, custom_photo_processing: true, custom_cover_photo_processing: nil, video_low_resolution: nil, video_standard_resolution: nil, carousel_media: {}, source_user_email: nil, photo_request_id: nil, is_customer_post: false, instagram_account_id: nil, custom_photo_data: "{\"id\":\"/posts/6534/original/993d5cdb9a410ed7ff8310...", custom_cover_photo_data: nil>, @name=:custom_photo> 
2.6.5 :003 > p attacher.file
#<CustomPhotoForPostUploader::UploadedFile storage=:store id="/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png" metadata={"size"=>141207, "filename"=>"Screenshot_2020-01-10_at_10.51.03.png", "mime_type"=>"image/png"}>
 => #<CustomPhotoForPostUploader::UploadedFile storage=:store id="/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png" metadata={"size"=>141207, "filename"=>"Screenshot_2020-01-10_at_10.51.03.png", "mime_type"=>"image/png"}> 
2.6.5 :004 > p attacher.derivatives
{}
 => {}

AND

2.6.5 :005 > attacher = CustomPhotoForPostUploader::Attacher.from_column(post.custom_photo_data)
 => #<CustomPhotoForPostUploader::Attacher:0x00007ffd68013b60 @file=#<CustomPhotoForPostUploader::UploadedFile storage=:store id="/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png" metadata={"size"=>141207, "filename"=>"Screenshot_2020-01-10_at_10.51.03.png", "mime_type"=>"image/png"}>, @cache=:cache, @store=:store, @context={}, @previous=nil, @column_serializer=Shrine::Plugins::Column::JsonSerializer, @model_cache=true, @model=nil, @derivatives={}, @derivatives_mutex=#<Thread::Mutex:0x00007ffd680138e0>, @errors=[]> 
2.6.5 :006 > p attacher.file
#<CustomPhotoForPostUploader::UploadedFile storage=:store id="/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png" metadata={"size"=>141207, "filename"=>"Screenshot_2020-01-10_at_10.51.03.png", "mime_type"=>"image/png"}>
 => #<CustomPhotoForPostUploader::UploadedFile storage=:store id="/posts/6534/original/993d5cdb9a410ed7ff83105194ba10388967b188.png" metadata={"size"=>141207, "filename"=>"Screenshot_2020-01-10_at_10.51.03.png", "mime_type"=>"image/png"}> 
2.6.5 :007 > p attacher.derivatives
{}
 => {}

And yeah, maybe it’s somehow my database state related issue. I’ll test creating new DB.

Thank you for your time and it was really helpful.