What's the most performant way to check if a file exists on S3

I got to validate if db records after a db restore still point to existing files stored in S3.
What is the best way to do that?

I tried something like .to_io.read(5) to only fetch like the first few bytes. But that didn’t look very reliable as i got a lot of intermittent connection related errors.

I’d guess there is an easy/proper way i just couldn’t find in the docs.

Thanks!

You want to get just the meta data, without getting any bytes at all.

This can be handled with the AWS ruby SDK, without really using Shrine, per se:

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#exists%3F-instance_method

I haven’t checked how the underlying implementation works, but i’d be very surprised if they were streaming.

the following is untested, and i’m not a shrine expert. there may be a better way.

id = photo.image.id 
if photo.image.storage.object(id).exists?
    puts "hooray!"
else
end

Or the shrine gem has an exists? method which simply delegates to the aws ruby SDK:

so if you want to use the shrine gem, you could do something like this:

id = photo.image.id 
if photo.image.storage.exists?(id)
    puts "hooray!"
else
end
1 Like

You can use Shrine::UploadedFile#exists?, which internally does what @BKSpurgeon described.

photo.image.exists?

This makes a HEAD request to the object using AWS SDK, which should be the fastest and the most reliable way.

2 Likes

Thanks that did indeed work.
Kind of obvious now that i think about it. Guess i didn’t figure out the right terms to search for in the API :grinning: