Derivatives missing on newly created, reloaded entity in the same session

usamah82 · April 30, 2020, 9:15am

I have a Photo ActiveRecord class. The attribute associated with the attachment is :photo, and it’s supposed to have two derivatives - :medium and :thumb. I eagerly generate the derivatives before save.

class Photo < ApplicationRecord
  before_save     :process_styles

  include ShrineUploader::Photo::Attachment(:photo)
  validates_presence_of :photo

  def process_styles
    return unless changes.include?(:photo_data)

    photo_derivatives!
  end
end

Creating a new Photo instance with a file attached to :photo works fine, and derivatives are generated.

However, I realized that in the same ‘session’, if one were to

reload that instance, or
load another instance with same ID from DB

The derivatives are ‘gone’. Even though the metadata in :photo_data indicates otherwise. Throughout the entire exercise below, the metadata is always:

# Converted the JSON to hash to make it easier to look at 
photo_data = 
{
  "id" => "v2/themes/base/icons/icon_original.png",
  "storage" => "public_storage",
  "metadata" => {
    "filename" => "icon.png", "size" => 918, "mime_type" => "image/png", "md5" => "9604d337e69de8d7a98a22ea761ede31", "width" => 75, "height" => 86
  },
  "derivatives" => {
    "thumb" => {
      "id" => "v2/themes/base/icons/icon_thumb.png",
      "storage" => "public_storage",
      "metadata" => {
        "filename" => "image_processing20200430-18457-12fvyxv.png", "size" => 1621, "mime_type" => "image/png", "md5" => "72d1ff986d3ccc2901333a87cb840bd6", "width" => 52, "height" => 60
      }
    },
    "medium" => {
      "id" => "v2/themes/base/icons/icon_medium.png",
      "storage" => "public_storage",
      "metadata" => {
        "filename" => "image_processing20200430-18457-z5u1iv.png", "size" => 788, "mime_type" => "image/png", "md5" => "eaca4980e678f587ef7daf7df0ed1895", "width" => 75, "height" => 86
      }
    }
  }
}

Another finding - If one were to fire up another rails console - the previously problematic instance lists derivatives just fine, even with reloads. This leads me to suspect that it also has something to do with the session / connection? And that somewhere there’s some invalid state.

The following would illustrate the points above:

> rails c
# Create new, valid instance
[9] pry(main)> photo = FactoryBot.create(:photo)
[10] pry(main)> photo.id
=> 8010
[11] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]
# Load another instance using the same ID, observe they have the same metadata
[12] pry(main)> photo.attributes == Photo.find(8010).attributes      
=> true
# However the loaded instance don't show any derivatives
[13] pry(main)> Photo.find(8010).photo_derivatives.keys
=> []
# And the existing instance also loses its derivatives once reloaded
[14] pry(main)> photo.reload
[15] pry(main)> photo.photo_derivatives.keys
=> []
[16] pry(main)> exit

# New console session
> rails c
# Reload the previously problematic instance
[1] pry(main)> photo = Photo.find(8010)
# Derivatives are now available
[2] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]
# Derivatives are still available with reload
[3] pry(main)> photo.reload
[4] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]

Lastly, if we manually create the attacher in the same ‘faulty’ session, the attacher is still able to list the derivatives correctly.

[4] pry(main)> photo = FactoryBot.create(:photo)
# Shows derivatives right after it has been created
[15] pry(main)> photo.photo_derivatives.keys
=> [:thumb, :medium]
# Derivatives gone missing after a reload
[16] pry(main)> photo.reload
[17] pry(main)> photo.photo_derivatives.keys
=> []
# A manually instantiated Shrine::Attacher from the same, 'faulty' photo instance however shows derivatives still
[18] pry(main)> attacher = Shrine::Attacher.from_model(photo, :photo)
[19] pry(main)> attacher.derivatives.keys
=> [:thumb, :medium]

My plugins are

  plugin :default_storage
  plugin :model, cache: false
  plugin :activerecord
  plugin :derivatives
  plugin :validation
  plugin :validation_helpers
  plugin :signature
  plugin :add_metadata
  plugin :determine_mime_type
  plugin :url_options
  plugin :default_url

usamah82 · April 30, 2020, 9:57am

Ok I think I figured it out (kinda).

What’s happening is that I am trying to migrate to shrine from paperclip. There’s a few models involved, so at this stage we’re seeing both

models with dual-write migration. This has Shrine.plugin :derivatives
models purely on shrine (using shrine’s Uploader). The Uploader also has plugin :derivatives

Am not familiar with the internals when the plugins are included, but basically having derivatives plugin defined twice causes the issue. Once I commented out the derivatives plugin in dual-write, we no longer have the issue above.

janko · May 8, 2020, 7:59am

Hi, thank you for the detailed analysis, and apologies for the late response.

Loading the same plugin the second time should be a no-op, if it isn’t, then it’s considered a bug. Unfortunately, I’m not able to tell from this why did loading derivatives the second time cause the problem.

It would be very helpful if you could reproduce these problems in a self-contained script, without FactoryBot or Rails. For example, when I try to run a script based on your console code, I’m not able to reproduce the issue.

require "shrine"
require "shrine/storage/memory"
require "active_record"
require "stringio"

ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
ActiveRecord::Base.connection.create_table(:photos) { |t| t.text :photo_data }

Shrine.storages[:cache] = Shrine.storages[:store] = Shrine::Storage::Memory.new

Shrine.plugin :activerecord
Shrine.plugin :derivatives # first time

class ImageUploader < Shrine
  plugin :derivatives # second time

  Attacher.derivatives do
    { thumb: StringIO.new, medium: StringIO.new } # fake derivatives
  end
end

class Photo < ActiveRecord::Base
  include ImageUploader::Attachment[:photo]

  before_save :photo_derivatives!, if: -> (r) { r.changes.include?(:photo_data) }
end

photo = Photo.create(photo: StringIO.new)
photo.photo_derivatives.keys # => [:thumb, :medium]

Photo.find(photo.id).photo_derivatives.keys # => [:thumb, :medium]

photo.reload
photo.photo_derivatives.keys # => [:thumb, :medium]

I think the issue is that the derivatives you created somehow weren’t persisted. It could be because Active Record failed validations for whatever reason.

For creating derivatives automatically I would recommend this patch. In Shrine 3.3 this functionality will come in form of a :create_on_promote plugin option.

usamah82 · May 11, 2020, 4:55am

Thanks Janko!

I gave this one another go, and I think I found a way to reproduce it.

In my paperclip dual write model, I have

Shrine.plugin :model
Shrine.plugin :derivatives

In my uploader, I have

class BaseUploader < Shrine
  plugin :activerecord
  plugin :derivatives
end

I have to specifically comment out Shrine.plugin #derivatives on the paperclip dual write module to make the errors go away, like the previous finding. I realized there’s a slight difference from my code to your test code.

Mine has :activerecord plugin in the uploader, yours have it outside of the uploader.

Yours:

Shrine.plugin :activerecord
Shrine.plugin :derivatives

class ImageUploader < Shrine
  plugin :derivatives
end

Note that I also tried commenting out the derivatives plugin in the uploader, but it breaks, even though the Shrine.plugin should ideally cover for it?

So in short:

# :derivatives plugin declared twice
# This causes derivatives to go missing on reload
Shrine.plugin :model
Shrine.plugin :derivatives

class BaseUploader < Shrine
  plugin :activerecord
  plugin :derivatives
end

# :derivatives plugin declared once outside of Uploader
# This causes derivatives processing to break - undefined method `derivatives' 
Shrine.plugin :model
Shrine.plugin :derivatives

class BaseUploader < Shrine
  plugin :activerecord
  # plugin :derivatives
end

# :derivatives plugin declared once inside of Uploader
# This fixes the problem, but dual write paperclip no longer works for other models
Shrine.plugin :model
# Shrine.plugin :derivatives

class BaseUploader < Shrine
  plugin :activerecord
  plugin :derivatives
end