I’m using Shrine for the first time, and I’ve even stopped using derivatives to try and track down this bug. I’m utterly confused why it happens and I’ve tried everything from the docs.
My setup:
- rails API only
- backgrounding with sidekiq
- direct uploads to S3 bucket
- not using Uppy
Expected behaviour:
- user async uploads image via upload endpoint
- UI confirms saves back to server with cached file data
- finalize image
- user profile API returns the promoted image
minimal API to show reproduction of issue
def profile
json_response({ cover_image: current_user.cover_image_url }, :ok)
end
Actual behaviour
- repeated requests to the profile API returns the URL of the
:cache
file on every 5th request. Otherwise it returns the:store
file. And it is exactly every 5th request. It’s BIZARRE.
1st, 2nd, 3rd, 4th response response
{
"cover_image": "https://media.entertwine.online/dev/user/1/cover_image/e9b9f669250348c06597293cd5e5bb86.jpg"
}
5th response (reverts to cache?!)
{
"cover_image": "https://media.entertwine.online/dev/cache/9d0f6bfb5dcf604d472fe402c89073d9.jpg"
}
6, 7 8, 9th response
{
"cover_image": "https://media.entertwine.online/dev/user/1/cover_image/e9b9f669250348c06597293cd5e5bb86.jpg"
}
10th response (back to cache?!!?)
{
"cover_image": "https://media.entertwine.online/dev/cache/9d0f6bfb5dcf604d472fe402c89073d9.jpg"
}
I don’t know what to do?!?! This seems REALLY ODD behaviour.
I’ve checked the DB, and the ONLY value in the cover_image_data
column is like this. So how is it even possible for the app to know the :cache
URL and return it??~?~?
{"id":"user/1/cover_image/e9b9f669250348c06597293cd5e5bb86.jpg","storage":"store","metadata":{"filename":"example.jpg","size":440686,"mime_type":"image/jpeg","width":1667,"height":1667,"crop":{"x":0,"y":0,"height":200,"width":200}}}
Additional info
ImageUploader code
I’ve removed the derivatives because I was worried this was the source of the bug
require 'image_processing/vips'
require 'shrine/plugins/store_dimensions'
class DocumentUploader < BaseUploader
include ImageProcessing::Vips
plugin :store_dimensions, analyzer: :ruby_vips
# doc related
DOC_EXTS = %w[doc docx ppt pptx xls xlsx pdf odt ods]
DOC_MIME_TYPES = %w[
application/pdf
application/msword
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/vnd.ms-powerpoint
application/vnd.openxmlformats-officedocument.presentationml.presentation
application/vnd.ms-excel
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.oasis.opendocument.presentation
application/vnd.oasis.opendocument.spreadsheet
application/vnd.oasis.opendocument.text
]
# image related
IMG_EXTS = %w[jpg jpeg png webp]
IMG_MIME_TYPES = %w[image/jpeg image/png image/webp]
MAX_IMG_DIMENSIONS = [2048, 2048]
THUMBNAILS = {
large: [720, 720],
medium: [576, 576],
small: [240, 240],
thumbnail: [128, 128]
}
# validate
Attacher.validate do
validate_extension_inclusion IMG_EXTS + DOC_EXTS
if validate_mime_type IMG_MIME_TYPES
validate_max_dimensions MAX_IMG_DIMENSIONS
end
end
# derivatives processing for images only
# Attacher.derivatives do |original|
# case file.mime_type
# when *IMG_MIME_TYPES
# process_derivatives(:image, original)
# end
# end
#
# Attacher.derivatives :image do |original|
# i = ImageProcessing::Vips.source(original)
# i = i.crop(*file.crop_points)
# THUMBNAILS.transform_values do |width, height|
# i.resize_to_limit(width, height)
# end
# end
Attacher.default_url { |derivative: nil, **| file&.url if derivative }
class UploadedFile
# convenience method for fetching crop points from metadata
def crop_points
metadata.fetch('crop').values_at('x', 'y', 'width', 'height')
end
end
end
initializer /config/initalizers/shrine.rb
#
# SHRINE FILE UPLOADER
#
require 'shrine'
require 'shrine/storage/s3'
require 'shrine/plugins/activerecord'
require 'shrine/plugins/derivatives'
require 'shrine/plugins/backgrounding'
require 'shrine/plugins/cached_attachment_data'
require 'shrine/plugins/restore_cached_data'
require 'shrine/plugins/validation'
require 'shrine/plugins/validation_helpers'
require 'shrine/plugins/remove_invalid'
require 'shrine/plugins/data_uri'
require 'shrine/plugins/instrumentation'
require 'shrine/plugins/pretty_location'
require 'shrine/plugins/default_url'
require 'shrine/plugins/upload_endpoint'
require 'shrine/plugins/determine_mime_type'
require 'shrine/plugins/url_options'
# plugins
Shrine.plugin :activerecord # loads Active Record integration
Shrine.plugin :derivatives # Save a file in multiple versions
Shrine.plugin :backgrounding # Background processing
Shrine.plugin :cached_attachment_data # enables retaining cached files for async uploader
Shrine.plugin :restore_cached_data # extracts metadata for assigned cached files
Shrine.plugin :validation
Shrine.plugin :validation_helpers # validation methods
Shrine.plugin :remove_invalid
Shrine.plugin :data_uri # allows uploading of base64 images
Shrine.plugin :instrumentation
Shrine.plugin :pretty_location
Shrine.plugin :default_url
Shrine.plugin :upload_endpoint, url: true
Shrine.plugin :determine_mime_type, analyzer: :marcel
Shrine.plugin :url_options,
store: { host: ENV['MEDIA_URL'] },
cache: { host: ENV['MEDIA_URL'] }
# logging
Shrine.logger = Rails.logger
# setup
def production
s3_options = Rails.application.credentials.aws
{
cache:
Shrine::Storage::S3.new(
prefix: 'uploads/cache',
public: true,
upload_options: { acl: 'public-read' },
**s3_options
), # temporary
store:
Shrine::Storage::S3.new(
prefix: 'uploads',
public: true,
upload_options: { acl: 'public-read' },
**s3_options
)
}
end
def development
s3_options = Rails.application.credentials.aws
{
cache:
Shrine::Storage::S3.new(
prefix: 'dev/cache',
public: true,
upload_options: { acl: 'public-read' },
**s3_options
), # temporary
store:
Shrine::Storage::S3.new(
prefix: 'dev',
public: true,
upload_options: { acl: 'public-read' },
**s3_options
)
}
end
Shrine.storages = Rails.env.production? ? production : development
Shrine::Attacher.promote_block do
ShrineBackgrounding::PromoteJob.perform_async(
self.class.name,
record.class.name,
record.id,
name,
file_data
)
end
Shrine::Attacher.destroy_block do
ShrineBackgrounding::DestroyJob.perform_async(self.class.name, data)
end
promote job
# frozen_string_literal: true
module ShrineBackgrounding
class PromoteJob
include Sidekiq::Worker
def perform(attacher_class, record_class, record_id, name, file_data)
logger.debug 'promoting!'
attacher_class = Object.const_get(attacher_class)
record = Object.const_get(record_class).find(record_id) # if using Active Record
logger.debug 'got record!'
attacher =
attacher_class.retrieve(model: record, name: name, file: file_data)
logger.debug 'got attacher!'
attacher.atomic_promote
rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound => e
# attachment has changed or the record has been deleted, nothing to do
logger.debug 'error message: ' + e.message
logger.debug 'Job abandoned. Something changed or record not found'
end
end
end
image controller
# frozen_string_literal: true
module Api
class ImagesController < ApiController
before_action :authenticate_request
USER_IMAGE_TYPES = {
user_cover_image: { model_attribute: 'cover_image', size: :large },
user_avatar: { model_attribute: 'avatar', size: :thumbnail }
}
def promote_user_media
image = image_params
final_type = final_type_params
unless current_user and final_type and
USER_IMAGE_TYPES.key?(final_type.to_sym)
json_response({ status: 'ERROR' }, :bad_request)
end
attribute = USER_IMAGE_TYPES[final_type.to_sym].fetch(:model_attribute)
size = USER_IMAGE_TYPES[final_type.to_sym].fetch(:size)
save_user_image(image, attribute)
current_user.reload
final_url = current_user.send attribute + '_url', size
json_response({ final_url: final_url }, :ok)
rescue StandardError => e
json_response({ status: e.message }, :internal_server_error)
end
private
def final_type_params
params.permit(:final_type)
params[:final_type]
end
def image_params
params.require(:image).permit!
params[:image]
end
def save_user_image(upload = nil, attribute = nil)
unless upload and attribute and current_user
raise StandardError.new 'Missing arguments to save image'
end
u = {}
u[attribute] = upload.to_json
current_user.update(u)
end
end
end
routes
mount DocumentUploader.upload_endpoint(:cache) => '/images/uploads'
put '/images/finalize_user_image' => 'images#promote_user_media'