Tag: s3

  • Desperately seek()ing S3 Mountpoints

    Context: images.nasa.gov metadata In our work on images.nasa.gov, we extract metadata from image, video, and audio files to make them searchable. Some of these files can be quite large, especially the 4K video files we’re seeing now.  The entire site runs on Amazon Web Services (AWS), with our media files stored on S3 object storage,…

  • Lambda-generated Presigned S3 URLs with AES encryption: CORS is Hell

    This is a followup to a previous post on how we use Lambdas to generate presigned URLs so that a user’s browser can directly upload to S3. We now want to have our S3 bucket enforce server-side encryption for all uploaded files. Getting all the pieces to work together was a bit hairy: bucket policies,…

  • Serverless Browser Uploads to S3

    We’ve been diggin’ the serverless ecosystem around AWS Lambda, API Gateway and the rest, especially using the Serverless Framework which makes it easy to define interesting applications. We frequently have a need to upload files from a user’s desktop to AWS S3 storage, and prefer to have the browser upload directly, rather than passing the…

  • Design Patterns for Serverless, Lambda, DynamoDB, S3

    Motivation We’ve been using AWS load balancers with autoscaling instances for years now and it’s great at handling load, but it’s quite a bit of infrastructure to manage (even with Troposphere + CloudFormation). We also have to manage all the data flow, queue processing and such ourselves: multiple SQS queues, EC2 polling, recording state in databases… More…