Shrine

Derivatives missing on newly created, reloaded entity in the same session

Bug
#1

I have a Photo ActiveRecord class. The attribute associated with the attachment is :photo, and it’s supposed to have two derivatives - :medium and :thumb. I eagerly generate the derivatives before save.

class Photo < ApplicationRecord
  before_save     :process_styles

  include ShrineUploader::Photo::Attachment(:photo)
  validates_presence_of :photo

  def process_styles
    return unless changes.include?(:photo_data)

    photo_derivatives!
  end
end

Creating a new Photo instance with a file attached to :photo works fine, and derivatives are generated.

However, I realized that in the same ‘session’, if one were to

  • reload that instance, or
  • load another instance with same ID from DB

The derivatives are ‘gone’. Even though the metadata in :photo_data indicates otherwise. Throughout the entire exercise below, the metadata is always:

# Converted the JSON to hash to make it easier to look at 
photo_data = 
{
  "id" => "v2/themes/base/icons/icon_original.png",
  "storage" => "public_storage",
  "metadata" => {
    "filename" => "icon.png", "size" => 918, "mime_type" => "image/png", "md5" => "9604d337e69de8d7a98a22ea761ede31", "width" => 75, "height" => 86
  },
  "derivatives" => {
    "thumb" => {
      "id" => "v2/themes/base/icons/icon_thumb.png",
      "storage" => "public_storage",
      "metadata" => {
        "filename" => "image_processing20200430-18457-12fvyxv.png", "size" => 1621, "mime_type" => "image/png", "md5" => "72d1ff986d3ccc2901333a87cb840bd6", "width" => 52, "height" => 60
      }
    },
    "medium" => {
      "id" => "v2/themes/base/icons/icon_medium.png",
      "storage" => "public_storage",
      "metadata" => {
        "filename" => "image_processing20200430-18457-z5u1iv.png", "size" => 788, "mime_type" => "image/png", "md5" => "eaca4980e678f587ef7daf7df0ed1895", "width" => 75, "height" => 86
      }
    }
  }
}

Another finding - If one were to fire up another rails console - the previously problematic instance lists derivatives just fine, even with reloads. This leads me to suspect that it also has something to do with the session / connection? And that somewhere there’s some invalid state.

The following would illustrate the points above:

> rails c
# Create new, valid instance
[9] pry(main)> photo = FactoryBot.create(:photo)
[10] pry(main)> photo.id
=> 8010
[11] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]
# Load another instance using the same ID, observe they have the same metadata
[12] pry(main)> photo.attributes == Photo.find(8010).attributes      
=> true
# However the loaded instance don't show any derivatives
[13] pry(main)> Photo.find(8010).photo_derivatives.keys
=> []
# And the existing instance also loses its derivatives once reloaded
[14] pry(main)> photo.reload
[15] pry(main)> photo.photo_derivatives.keys
=> []
[16] pry(main)> exit

# New console session
> rails c
# Reload the previously problematic instance
[1] pry(main)> photo = Photo.find(8010)
# Derivatives are now available
[2] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]
# Derivatives are still available with reload
[3] pry(main)> photo.reload
[4] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]

Lastly, if we manually create the attacher in the same ‘faulty’ session, the attacher is still able to list the derivatives correctly.

[4] pry(main)> photo = FactoryBot.create(:photo)
# Shows derivatives right after it has been created
[15] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]
# Derivatives gone missing after a reload
[16] pry(main)> photo.reload
[17] pry(main)> photo.photo_derivatives.keys
=> []
# A manually instantiated Shrine::Attacher from the same, 'faulty' photo instance however shows derivatives still
[18] pry(main)> attacher = Shrine::Attacher.from_model(photo, :photo)
[19] pry(main)> attacher.derivatives.keys
=> [:thumb, :medium]

My plugins are

  plugin :default_storage
  plugin :model, cache: false
  plugin :activerecord
  plugin :derivatives
  plugin :validation
  plugin :validation_helpers
  plugin :signature
  plugin :add_metadata
  plugin :determine_mime_type
  plugin :url_options
  plugin :default_url
#2

Ok I think I figured it out (kinda).

What’s happening is that I am trying to migrate to shrine from paperclip. There’s a few models involved, so at this stage we’re seeing both

  • models with dual-write migration. This has Shrine.plugin :derivatives
  • models purely on shrine (using shrine’s Uploader). The Uploader also has plugin :derivatives

Am not familiar with the internals when the plugins are included, but basically having derivatives plugin defined twice causes the issue. Once I commented out the derivatives plugin in dual-write, we no longer have the issue above.

#3

Hi, thank you for the detailed analysis, and apologies for the late response.

Loading the same plugin the second time should be a no-op, if it isn’t, then it’s considered a bug. Unfortunately, I’m not able to tell from this why did loading derivatives the second time cause the problem.

It would be very helpful if you could reproduce these problems in a self-contained script, without FactoryBot or Rails. For example, when I try to run a script based on your console code, I’m not able to reproduce the issue.

require "shrine"
require "shrine/storage/memory"
require "active_record"
require "stringio"

ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
ActiveRecord::Base.connection.create_table(:photos) { |t| t.text :photo_data }

Shrine.storages[:cache] = Shrine.storages[:store] = Shrine::Storage::Memory.new

Shrine.plugin :activerecord
Shrine.plugin :derivatives # first time

class ImageUploader < Shrine
  plugin :derivatives # second time

  Attacher.derivatives do
    { thumb: StringIO.new, medium: StringIO.new } # fake derivatives
  end
end

class Photo < ActiveRecord::Base
  include ImageUploader::Attachment[:photo]

  before_save :photo_derivatives!, if: -> (r) { r.changes.include?(:photo_data) }
end

photo = Photo.create(photo: StringIO.new)
photo.photo_derivatives.keys # => [:thumb, :medium]

Photo.find(photo.id).photo_derivatives.keys # => [:thumb, :medium]

photo.reload
photo.photo_derivatives.keys # => [:thumb, :medium]

I think the issue is that the derivatives you created somehow weren’t persisted. It could be because Active Record failed validations for whatever reason.

For creating derivatives automatically I would recommend this patch. In Shrine 3.3 this functionality will come in form of a :create_on_promote plugin option.

#4

Thanks Janko!

I gave this one another go, and I think I found a way to reproduce it.

In my paperclip dual write model, I have

Shrine.plugin :model
Shrine.plugin :derivatives

In my uploader, I have

class BaseUploader < Shrine
  plugin :activerecord
  plugin :derivatives
end

I have to specifically comment out Shrine.plugin #derivatives on the paperclip dual write module to make the errors go away, like the previous finding. I realized there’s a slight difference from my code to your test code.

Mine has :activerecord plugin in the uploader, yours have it outside of the uploader.

Yours:

Shrine.plugin :activerecord
Shrine.plugin :derivatives

class ImageUploader < Shrine
  plugin :derivatives
end

Note that I also tried commenting out the derivatives plugin in the uploader, but it breaks, even though the Shrine.plugin should ideally cover for it?

So in short:

# :derivatives plugin declared twice
# This causes derivatives to go missing on reload
Shrine.plugin :model
Shrine.plugin :derivatives

class BaseUploader < Shrine
  plugin :activerecord
  plugin :derivatives
end

# :derivatives plugin declared once outside of Uploader
# This causes derivatives processing to break - undefined method `derivatives' 
Shrine.plugin :model
Shrine.plugin :derivatives

class BaseUploader < Shrine
  plugin :activerecord
  # plugin :derivatives
end

# :derivatives plugin declared once inside of Uploader
# This fixes the problem, but dual write paperclip no longer works for other models
Shrine.plugin :model
# Shrine.plugin :derivatives

class BaseUploader < Shrine
  plugin :activerecord
  plugin :derivatives
end
#5

Thanks @usamah82, your discovery helped fix the same issue.

@janko I also had this same issue. By commenting out the derivatives import in PaperclipShrineSynchronization and keeping the one in the uploader I was able to get the derivates after an Active Object reload.

# Shrine.plugin :derivatives

module PaperclipShrineSynchronization
...

class UserImageUploader < Shrine
  plugin :determine_mime_type
  plugin :validation_helpers
  plugin :derivatives
  plugin :data_uri
...