Was wondering about how to go about this type of work flow (see below). Any advice pertaining to the use of AWS lambda, and its suitability for the below case would be well regarded: unzipping large files. (AWS lambda is by no means a prescriptive requirement): i just want to unzip files and run validations against them.
- The user uploads files. These could be compressed files (.zip, .rar, .7zip) or uncompressed. The files may vary in size: 1 mb to perhaps a couple of gbs.
- I think if we upload these files directly to the bucket that would be good.
- If the files are zipped, then i would want them unzipped (perhaps with AWS lambda?) and presented to the user in a form. The user may then add further information regarding those files in form fields - which will be persisted by ActiveRecord.
- I do not want the user to upload duplicate files. For example, if a file has already been uploaded last week, I’d want some mechanism preventing the upload. Necessarily this might involve querying the back end, or some validation type logic happening here.
Was seeking your advice on how to go about the above - particularly that relating to unzipping files. What is the best way to handle this in a rails app given it’s hosted on heroku (with it’s limitations on being able to store data)?
any thoughts suggests would be much appreciated