Testing with Tus

Hi,

I have Tus wired in my app according to the Approach A: Downloading through tus server.

All is fine, except I am unable to directly assign a file to the uploader. I used to be able to do:

MyUploader.new(file: File.new(path_to_file.jpg))

However with Tus, I am getting:

NoMethodError (undefined method `url' for #<File:0x00005646c3daccc0>)

Actual uploading of files work just fine, but I am struggling with setting up test data. I looked into Testing with Shrine (sorry can’t link here). But the method suggested for unit tests no longer works with Tus.

Am I doing something wrong? Is there a recommended approach for applications that rely on Tus?

Thank you very much in advance,
Tomas

Hi, the Shrine::Storage::Tus storage provided by the shrine-tus storage doesn’t currently support uploading files to the tus storage, as that would require a tus client implemented in Ruby, which doesn’t exist yet (I started writing one a while go, and have gotten pretty far with the implementation, but never finished it). Note that this would mean you would also need a tus server running in test environment.

If you want to set up test uploaded file data, you can either directly assign stored uploaded file data as shown here, or you can assign a mock cached file, and then stub the tus download request before promotion to permanent storage using something like WebMock.

mock_cached_file = Shrine.uploaded_file({
  id: "https://tus-example.org/file",
  storage: "cache",
  metadata: { ... },
})

model = MyModel.new(file: mock_cached_file.data)

stub_request(:get, mock_cached_file.url).and_return(body: "some file content")

model.save # promotion happens

sorry can’t link here

Just checking, is Discourse still not allowing new users to post links? I thought I had disabled that :thinking:

Hi Janko, thanks for the response.
I ended up writing simple decorator to handle this also when creating files from console.

class TusFileDecorator < SimpleDelegator
  def url
    case __getobj__
    when Rack::Test::UploadedFile
      tempfile.read
    when File
      __getobj__.read
    else
      raise NotImplementedError
    end
  end
end

And no, Discourse did not allow me to make links as a new user :(.

Sorry, I take it back. The above solution (of course) does not work.

The method described in the test-data guide (sorry can’t add links) also does not work:

Shrine::Error: expected cached file, got UploadedFile storage=:store …

Changing the storage to :cache brings back the URL problem.

Am I missing anything please? Thank you!

The testing guide only shows creating test file data that is meant to be assigned directly to the attachment column (see the Factory Bot example). You can certainly do this, it’s just that this way you wouldn’t be testing the shrine-tus integration.

If you wanted to test the shrine-tus integration, I would need to know what didn’t work with my last suggestion, so that we can go from there.

I’ve checked again that new users should be allowed to post emails, so it might be a bug in Discourse. So I manually upgraded your Discourse user level, hopefully this will fix it. I’ll also try upgrading the Discourse instance.

Hi Janko,
Thanks for getting back to me.

If I replicate exactly what it is in the testing with Shrine (Testing with Shrine · Shrine) and FactoryBot, I am getting:

Shrine::Error: expected cached file, got #<UploadedFile storage=:store id="…" metadata={"filename"=>"sample.jpg", "size"=>76496, "mime_type"=>nil}>

When I switch the store in the example to :cache, I am back on Down::InvalidUrl: unknown scheme ….

My storages config looks like this:

Shrine.storages ||= {}
Shrine.storages.merge!(
  cache: Shrine::Storage::Memory.new,
  store: Shrine::Storage::Memory.new,
  tus: Shrine::Storage::Tus.new
)

And the Tus storage is configured as indicated here:

Namely by assigning the :tus to :cache storage in an Uploader class:

storages[:cache] = storages[:tus]

Everything works fine when uploading through an uploader (I use Uppy), both in dev & production.

However sometime I need to upload files from console or in tests – and that’s where I am hitting a wall.

Advice how to get this to work would be much appreciated as it seems otherwise I’d have to drop Tus, which I am hesitant to do (I’d like to keep the uploading resumable etc.).

Thank you!

If you assign attachment data directly to the <attachment>_data column attribute (skipping Shrine’s attachment attribute), like the testing docs are showing, then you cannot be receiving Shrine::Error: expected cached file. That error is only raised by the attachment attribute setter.

If you switch to :cache, then you need to assign a valid URL to the id attribute of the uploaded file, and afterwards you can catch the request with WebMock like I’ve mentioned. It just depends on what you want to test (promotion from tus :cache to :store, or something else).

Hi Janko,

Thanks, you are right, my bad, I managed to get it work like this.

It works in tests, but still, making files from console or from seeds is an issue, unless I implement similar workaround as in the tests.

However I noticed, in the shrine-tus-demo app README (GitHub - shrinerb/shrine-tus-demo: Demo integrating tus resumable upload protocol with Shrine) that:

The Shrine storages can be configured in a way that allows you to use the tus server only for some attachments (normally large ones), while other attachments can still accept files in the standard way.

This could be a more long-term solution for the problems I am facing while using Tus. Could you please point me to a direction how do I configure storages based on attachment size? I thought I could do that using the default_storage plugin (Default Storage · Shrine), but it does not seem to be the case.

Thank you,
Tomas

This could be a more long-term solution for the problems I am facing while using Tus. Could you please point me to a direction how do I configure storages based on attachment size?

Different storages can only be configured for different uploaders, not dynamically based on filesize. I see how that paragraph from shrine-tus README was misleading, it was referring to scenario where you know that some uploaders will accept larger files than others.

The reason it’s not working from console or seeds is because Shrine::Storages::Tus doesn’t know how to upload to a tus storage. So there you want to skip it entirely. You can do that either by calling Shrine::Attacher#attach directly, which uploads directly to permanent storage:

record.file_attacher.attach(some_file)

or enabling that by default for seeds and console:

# db/seeds.rb
MyUploader.plugin :model, cache: false

# config/environments/development.rb
Rails.application.configure do
  # ...
  console do
    MyUploader.plugin :model, cache: false
  end
  # ...
end

Thank you, Janko, this helped me to resolve the issues.
Tomas