After promotion, Shrine randomly returns :cache or :store even though only :store in DB

Bug
#1

I’m using Shrine for the first time, and I’ve even stopped using derivatives to try and track down this bug. I’m utterly confused why it happens and I’ve tried everything from the docs.

My setup:

  • rails API only
  • backgrounding with sidekiq
  • direct uploads to S3 bucket
  • not using Uppy

Expected behaviour:

  • user async uploads image via upload endpoint
  • UI confirms saves back to server with cached file data
  • finalize image
  • user profile API returns the promoted image

minimal API to show reproduction of issue

def profile
      json_response({ cover_image: current_user.cover_image_url }, :ok)
end

Actual behaviour

  • repeated requests to the profile API returns the URL of the :cache file on every 5th request. Otherwise it returns the :store file. And it is exactly every 5th request. It’s BIZARRE.

1st, 2nd, 3rd, 4th response response

{
  "cover_image": "https://media.entertwine.online/dev/user/1/cover_image/e9b9f669250348c06597293cd5e5bb86.jpg"
}

5th response (reverts to cache?!)

{
  "cover_image": "https://media.entertwine.online/dev/cache/9d0f6bfb5dcf604d472fe402c89073d9.jpg"
}

6, 7 8, 9th response

{
  "cover_image": "https://media.entertwine.online/dev/user/1/cover_image/e9b9f669250348c06597293cd5e5bb86.jpg"
}

10th response (back to cache?!!?)

{
  "cover_image": "https://media.entertwine.online/dev/cache/9d0f6bfb5dcf604d472fe402c89073d9.jpg"
}

I don’t know what to do?!?! This seems REALLY ODD behaviour.

I’ve checked the DB, and the ONLY value in the cover_image_data column is like this. So how is it even possible for the app to know the :cache URL and return it??~?~?

{"id":"user/1/cover_image/e9b9f669250348c06597293cd5e5bb86.jpg","storage":"store","metadata":{"filename":"example.jpg","size":440686,"mime_type":"image/jpeg","width":1667,"height":1667,"crop":{"x":0,"y":0,"height":200,"width":200}}}

Additional info

ImageUploader code

I’ve removed the derivatives because I was worried this was the source of the bug

require 'image_processing/vips'
require 'shrine/plugins/store_dimensions'

class DocumentUploader < BaseUploader
  include ImageProcessing::Vips 
  plugin :store_dimensions, analyzer: :ruby_vips

  # doc related
  DOC_EXTS = %w[doc docx ppt pptx xls xlsx pdf odt ods]
  DOC_MIME_TYPES = %w[
    application/pdf
    application/msword
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
    application/vnd.ms-powerpoint
    application/vnd.openxmlformats-officedocument.presentationml.presentation
    application/vnd.ms-excel
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    application/vnd.oasis.opendocument.presentation
    application/vnd.oasis.opendocument.spreadsheet
    application/vnd.oasis.opendocument.text
  ]

  # image related
  IMG_EXTS = %w[jpg jpeg png webp]
  IMG_MIME_TYPES = %w[image/jpeg image/png image/webp]
  MAX_IMG_DIMENSIONS = [2048, 2048]
  THUMBNAILS = {
    large: [720, 720],
    medium: [576, 576],
    small: [240, 240],
    thumbnail: [128, 128]
  }

  # validate
  Attacher.validate do
    validate_extension_inclusion IMG_EXTS + DOC_EXTS
    if validate_mime_type IMG_MIME_TYPES
      validate_max_dimensions MAX_IMG_DIMENSIONS
    end
  end

  # derivatives processing for images only
  # Attacher.derivatives do |original|
  #   case file.mime_type
  #   when *IMG_MIME_TYPES
  #     process_derivatives(:image, original)
  #   end
  # end
  #
  # Attacher.derivatives :image do |original|
  #   i = ImageProcessing::Vips.source(original)
  #   i = i.crop(*file.crop_points)
  #   THUMBNAILS.transform_values do |width, height|
  #     i.resize_to_limit(width, height)
  #   end
  # end

  Attacher.default_url { |derivative: nil, **| file&.url if derivative }

  class UploadedFile
    # convenience method for fetching crop points from metadata
    def crop_points
      metadata.fetch('crop').values_at('x', 'y', 'width', 'height')
    end
  end
end

initializer /config/initalizers/shrine.rb 

#
# SHRINE FILE UPLOADER
#

require 'shrine'
require 'shrine/storage/s3'
require 'shrine/plugins/activerecord'
require 'shrine/plugins/derivatives'
require 'shrine/plugins/backgrounding'
require 'shrine/plugins/cached_attachment_data'
require 'shrine/plugins/restore_cached_data'
require 'shrine/plugins/validation'
require 'shrine/plugins/validation_helpers'
require 'shrine/plugins/remove_invalid'
require 'shrine/plugins/data_uri'
require 'shrine/plugins/instrumentation'
require 'shrine/plugins/pretty_location'
require 'shrine/plugins/default_url'

require 'shrine/plugins/upload_endpoint'
require 'shrine/plugins/determine_mime_type'
require 'shrine/plugins/url_options'

# plugins
Shrine.plugin :activerecord # loads Active Record integration
Shrine.plugin :derivatives # Save a file in multiple versions
Shrine.plugin :backgrounding # Background processing

Shrine.plugin :cached_attachment_data # enables retaining cached files for async uploader
Shrine.plugin :restore_cached_data # extracts metadata for assigned cached files

Shrine.plugin :validation
Shrine.plugin :validation_helpers # validation methods
Shrine.plugin :remove_invalid
Shrine.plugin :data_uri # allows uploading of base64 images
Shrine.plugin :instrumentation
Shrine.plugin :pretty_location
Shrine.plugin :default_url

Shrine.plugin :upload_endpoint, url: true
Shrine.plugin :determine_mime_type, analyzer: :marcel
Shrine.plugin :url_options,
              store: { host: ENV['MEDIA_URL'] },
              cache: { host: ENV['MEDIA_URL'] }

# logging
Shrine.logger = Rails.logger

# setup

def production
  s3_options = Rails.application.credentials.aws
  {
    cache:
      Shrine::Storage::S3.new(
        prefix: 'uploads/cache',
        public: true,
        upload_options: { acl: 'public-read' },
        **s3_options
      ), # temporary
    store:
      Shrine::Storage::S3.new(
        prefix: 'uploads',
        public: true,
        upload_options: { acl: 'public-read' },
        **s3_options
      )
  }
end

def development
  s3_options = Rails.application.credentials.aws
  {
    cache:
      Shrine::Storage::S3.new(
        prefix: 'dev/cache',
        public: true,
        upload_options: { acl: 'public-read' },
        **s3_options
      ), # temporary
    store:
      Shrine::Storage::S3.new(
        prefix: 'dev',
        public: true,
        upload_options: { acl: 'public-read' },
        **s3_options
      )
  }
end

Shrine.storages = Rails.env.production? ? production : development

Shrine::Attacher.promote_block do
  ShrineBackgrounding::PromoteJob.perform_async(
    self.class.name,
    record.class.name,
    record.id,
    name,
    file_data
  )
end

Shrine::Attacher.destroy_block do
  ShrineBackgrounding::DestroyJob.perform_async(self.class.name, data)
end

promote job

# frozen_string_literal: true

module ShrineBackgrounding
  class PromoteJob
    include Sidekiq::Worker

    def perform(attacher_class, record_class, record_id, name, file_data)
      logger.debug 'promoting!'
      attacher_class = Object.const_get(attacher_class)
      record = Object.const_get(record_class).find(record_id) # if using Active Record
      logger.debug 'got record!'
      attacher =
        attacher_class.retrieve(model: record, name: name, file: file_data)
      logger.debug 'got attacher!'
      attacher.atomic_promote
    rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound => e
      # attachment has changed or the record has been deleted, nothing to do
      logger.debug 'error message: ' + e.message
      logger.debug 'Job abandoned. Something changed or record not found'
    end
  end
end

image controller

# frozen_string_literal: true

module Api
  class ImagesController < ApiController
    before_action :authenticate_request

    USER_IMAGE_TYPES = {
      user_cover_image: { model_attribute: 'cover_image', size: :large },
      user_avatar: { model_attribute: 'avatar', size: :thumbnail }
    }

    def promote_user_media
      image = image_params
      final_type = final_type_params

      unless current_user and final_type and
               USER_IMAGE_TYPES.key?(final_type.to_sym)
        json_response({ status: 'ERROR' }, :bad_request)
      end
      attribute = USER_IMAGE_TYPES[final_type.to_sym].fetch(:model_attribute)
      size = USER_IMAGE_TYPES[final_type.to_sym].fetch(:size)
      save_user_image(image, attribute)
      current_user.reload
      final_url = current_user.send attribute + '_url', size
      json_response({ final_url: final_url }, :ok)
    rescue StandardError => e
      json_response({ status: e.message }, :internal_server_error)
    end

    private

    def final_type_params
      params.permit(:final_type)
      params[:final_type]
    end

    def image_params
      params.require(:image).permit!
      params[:image]
    end

    def save_user_image(upload = nil, attribute = nil)
      unless upload and attribute and current_user
        raise StandardError.new 'Missing arguments to save image'
      end
      u = {}
      u[attribute] = upload.to_json
      current_user.update(u)
    end
  end
end

routes

mount DocumentUploader.upload_endpoint(:cache) => '/images/uploads'
put '/images/finalize_user_image' => 'images#promote_user_media'
#2

Can anyone help? :pensive: I have no way of resolving this, and it’s becoming mission critical

#3

I know I’m kind of late to the party, and I assume you were either able to resolve the problem or moved away from Shrine.

Just wanted to say that there is nothing in Shrine should be causing this behaviour. Once the cached file is promoted to permanent storage, it stays there. And Shrine will return whichever data it found in the database.

My only guess that there is some caching involved, but I cannot image which kind. I’m sorry you’re having these problems, but as a maintainer I wouldn’t know where to start looking.