Strange problem with minio, shrine and cache

Hi! I am trying to setup shrine with minio and imgproxy.

My shrine.rb looks like

require 'shrine'
require 'shrine/storage/s3'

def generate_s3_settings
  minio_settings = { endpoint: ENV['S3_HOST'], force_path_style: true }

  res = {
    access_key_id: ENV['S3_ACCESS_KEY'],
    secret_access_key: ENV['S3_SECRET'],
    bucket: ENV['S3_BUCKET'],
    region: 'us-east-1',
    public: true
  }

  if ENV['S3_HOST'].present?
    res.merge!(minio_settings)
  end

  res
end

Shrine.storages = {
  store: Shrine::Storage::S3.new(generate_s3_settings)
  cache: Shrine::Storage::S3.new(generate_s3_settings.merge(bucket: 'cache'))    
}

Shrine.plugin :activerecord # or :activerecord
Shrine.plugin :cached_attachment_data # for retaining the cached file across form redisplays
Shrine.plugin :restore_cached_data # re-extract metadata when attaching a cached file
Shrine.plugin :determine_mime_type

and uploader looks like

class ImageUploader < Shrine
  plugin :remote_url, max_size: 4 * 1024 * 1024
  plugin :determine_mime_type, analyzer: :marcel
  plugin :infer_extension, inferrer: :mini_mime
  plugin :validation_helpers

  Attacher.validate do
    validate_extension_inclusion %w[jpg jpeg png gif]
    validate_mime_type_inclusion %w[image/jpeg image/png image/gif]
  end

  def generate_uid(io)
    Digest::SHA1.hexdigest(io.read)
  end
end

When I try to upload anything through the form, I got error

puma_1         | [ca85ccdb-52bc-4845-817f-3871c0411009] Aws::S3::Errors::Forbidden ():
puma_1         | [ca85ccdb-52bc-4845-817f-3871c0411009]
puma_1         | [ca85ccdb-52bc-4845-817f-3871c0411009] app/uploaders/image_uploader.rb:13:in `generate_uid'
puma_1         | [ca85ccdb-52bc-4845-817f-3871c0411009] app/controllers/registrations_controller.rb:51:in `update_resource'

In the logs of nginx in front of minio:

172.28.0.1 - - [18/Sep/2019:13:51:10 +0000] "PUT /cache/c43db2597bda1d0c3aefdf1a00bed88ba25705c7.jpg HTTP/1.1" 200 25 "-" "aws-sdk-ruby3/3.67.0 ruby/2.6.3 x86_64-linux aws-sdk-s3/1.48.0"
172.28.0.1 - - [18/Sep/2019:13:51:10 +0000] "HEAD /cache/c43db2597bda1d0c3aefdf1a00bed88ba25705c7.jpg HTTP/1.1" 403 0 "-" "aws-sdk-ruby3/3.67.0 ruby/2.6.3 x86_64-linux aws-sdk-s3/1.48.0"

Also, in development in docker-compose environment I got this error

Aws::S3::Errors::InvalidRequest: This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.
/bundle/gems/aws-sdk-core-3.67.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/bundle/gems/aws-sdk-core-3.67.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/bundle/gems/aws-sdk-core-3.67.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/bundle/gems/aws-sdk-core-3.67.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/bundle/gems/aws-sdk-core-3.67.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/bundle/gems/aws-sdk-core-3.67.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/bundle/gems/aws-sdk-core-3.67.0/lib/seahorse/client/request.rb:70:in `send_request'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/client.rb:726:in `copy_object'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/object_copier.rb:33:in `copy_object'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/object_copier.rb:15:in `copy_from'
/bundle/gems/aws-sdk-s3-1.48.0/lib/aws-sdk-s3/customizations/object.rb:66:in `copy_from'
/bundle/gems/shrine-2.19.3/lib/shrine/storage/s3.rb:283:in `copy'
/bundle/gems/shrine-2.19.3/lib/shrine/storage/s3.rb:118:in `upload'
/bundle/gems/shrine-2.19.3/lib/shrine.rb:324:in `copy'
/bundle/gems/shrine-2.19.3/lib/shrine.rb:313:in `put'
/bundle/gems/shrine-2.19.3/lib/shrine.rb:297:in `_store'
/bundle/gems/shrine-2.19.3/lib/shrine.rb:228:in `store'
/bundle/gems/shrine-2.19.3/lib/shrine.rb:206:in `upload'
/bundle/gems/shrine-2.19.3/lib/shrine/attacher.rb:191:in `store!'
/bundle/gems/shrine-2.19.3/lib/shrine/attacher.rb:123:in `promote'
/bundle/gems/shrine-2.19.3/lib/shrine/attacher.rb:117:in `_promote'
/bundle/gems/shrine-2.19.3/lib/shrine/attacher.rb:112:in `finalize'
/bundle/gems/shrine-2.19.3/lib/shrine/plugins/activerecord.rb:41:in `block (2 levels) in included'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:429:in `instance_exec'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:429:in `block in make_lambda'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:264:in `block in conditional'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:518:in `block in invoke_after'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:518:in `each'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:518:in `invoke_after'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:136:in `run_callbacks'
/bundle/gems/activesupport-6.0.0/lib/active_support/callbacks.rb:827:in `_run_commit_callbacks'
/bundle/gems/activerecord-6.0.0/lib/active_record/transactions.rb:339:in `committed!'
/bundle/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/transaction.rb:127:in `commit_records'
/bundle/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/transaction.rb:265:in `block in commit_transaction'
/bundle/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/transaction.rb:255:in `commit_transaction'
/bundle/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/transaction.rb:293:in `block in within_new_transaction'
/bundle/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/bundle/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:277:in `transaction'
/bundle/gems/activerecord-6.0.0/lib/active_record/transactions.rb:212:in `transaction'
/bundle/gems/activerecord-6.0.0/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/bundle/gems/activerecord-6.0.0/lib/active_record/transactions.rb:315:in `save'

/bundle/gems/activerecord-6.0.0/lib/active_record/suppressor.rb:44:in `save'
/app/db/seeds.rb:414:in `<main>'
/bundle/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:54:in `load'
/bundle/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:54:in `load'
/bundle/gems/railties-6.0.0/lib/rails/engine.rb:556:in `block in load_seed'
/bundle/gems/railties-6.0.0/lib/rails/engine.rb:676:in `with_inline_jobs'
/bundle/gems/railties-6.0.0/lib/rails/engine.rb:556:in `load_seed'
/bundle/gems/activerecord-6.0.0/lib/active_record/tasks/database_tasks.rb:440:in `load_seed'
/bundle/gems/activerecord-6.0.0/lib/active_record/railties/databases.rake:328:in `block (2 levels) in <main>'
/bundle/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:54:in `load'
/bundle/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:54:in `load'
-e:1:in `<main>'
Tasks: TOP => db:seed

Any ideas how to fix this problem?

The generated upload locations should generally be unique, even when the file has the same content. What might be happening here is that when Shrine is copying cached file to permanent storage, the destination location is the same as the source location.

However, I now see that you’re using different buckets for temporary and permanent storage, so that shouldn’t be an issue. But then I don’t understand why an error is being raised. Does Minio not realize those are different buckets? :thinking:

It would be very helpful if you could isolate this issue in a minimal self-contained script, and run it on both Minio and S3. You can find a template here.

I tried narrow the problem and it looks like two things are influencing:
docker-compose environment and this code:

  def generate_uid(io)
    Digest::SHA1.hexdigest(io.read)
  end

The problem appears when running in docker-compose and with this generate_uid(io) method overriden. Same docker container with same configuration of everything in kubernetes works fine. When I comment generate_uid(io) method in my uploader everything works fine.

All that drives me craze about debugging this error. Any ideas why generate_uid can influence on this strange error?

PS. Trying to create docker-compose.yml with this error

Yes, the problem is that your #generate_uid override doesn’t produce unique locations. The digest does make it unique for files with different content, but file with same content will have same digests.

Since Shrine re-uploads the attached file from temporary to permanent storage, your #generate_uid override makes the destination location the same as the source location. I think in your development env you’re somehow using the same bucket for both temporary and permanent storage, which would cause the error with the S3 copy operation.

Hm, it looks like there are 2 different problems, one with copying file to itself. I solved the problem with copying to itself with default storage plugin and now in every environment everything works fine.

But problem with Aws::S3::Errors::Forbidden () still exists. Any ideas how to narrow it?

If you’re still generating the SHA in #generate_uid, the problem might be that you’re not rewinding the file after reading it. It should be:

class ImageUploader < Shrine
  # ...
  def generate_uid(io)
    sha = Digest::SHA1.hexdigest(io.read)
    io.rewind
    sha
  end
end

You can simplify this with the signature plugin, which does the rewinding automatically and is more memory efficient:

Shrine.plugin :signature
class ImageUploader < Shrine
  def generate_uid(io)
    Shrine.signature(io, :sha1)
  end
end

Thanks for the answer, I tried to change the method to Shrine.signature(io, :sha1), but it looks like it works only until I upload first file to the cache. After that, it will fail with forbidden error and save image url from cache bucket to the database.

I use imgproxy to resize images on the fly, so I dont need any processing.

Is there are any way to disable cache? Or is it possible to use different generate_uid(io) for cache and store buckets?

Is there are any way to disable cache?

At the moment it’s not so easy to disable cache, but it will be in Shrine 3.0, which is going to be out pretty soon.

Or is it possible to use different generate_uid(io) for cache and store buckets?

Yes:

def generate_uid(io)
  if storage_key == :cache
    # ...
  elsif storage_key == :store
    # ... 
  else
    super
  end
end
1 Like

It looks like error starts to appear while storing in store.

I added method:

class ImageUploader < Shrine
  plugin :remote_url, max_size: 4 * 1024 * 1024
  plugin :determine_mime_type, analyzer: :marcel
  plugin :infer_extension, inferrer: :mini_mime
  plugin :validation_helpers
  plugin :default_storage, store: :store, cache: :cache

  Attacher.validate do
    validate_extension_inclusion %w[jpg jpeg png gif]
    validate_mime_type_inclusion %w[image/jpeg image/png image/gif]
  end

  def generate_uid(io)
    return super if storage_key == :cache
    Shrine.signature(io, :sha1)
  end
end

And error appears on Shrine.signature(io, :sha1) line with:

[836af7f7-618c-4964-bc92-3d7c2d63ee28] Aws::S3::Errors::Forbidden ():
puma_1         | [836af7f7-618c-4964-bc92-3d7c2d63ee28]
puma_1         | [836af7f7-618c-4964-bc92-3d7c2d63ee28] app/uploaders/image_uploader.rb:15:in `generate_uid'
puma_1         | [836af7f7-618c-4964-bc92-3d7c2d63ee28] app/controllers/registrations_controller.rb:51:in `update_resource'

I think I will stop generating uniq filename depending on the content, because it pops strange errors only in one environment.

Anyway, thanks for the help.