Context: images.nasa.gov metadata

In our work on images.nasa.gov, we extract metadata from image, video, and audio files to make them searchable. Some of these files can be quite large, especially the 4K video files we’re seeing now.

The entire site runs on Amazon Web Services (AWS), with our media files stored on S3 object storage, and metadata extraction running on EC2 autoscaling instances. We download the media files from S3 to EC2, then use ExifTool to pull out EXIF, IPTC, and XMP data including date, title, photographer, camera, geolocation, size, and other technical details.

This has worked fine for us, but presents two concerns: the copy can take nontrivial time, even over AWS internal network; the file may be too big for our currently configured EC2 EBS storage.

Talk Nerdy to Me

Although ExifTool can read data streamed from S3, that sequential access requires it to read the entire file; this is just as slow as copying it, and trades RAM for disk.

But once a file is copied to disk, ExifTool can use “random access” — instead of sequential access — to the file’s structure to skip through the parts it doesn’t need (e.g., large video segments), in order to find what it needs (metadata, in our case). The table below summarizes two posts on MP4 video file format, showing how it’s composed of multiple “boxes” with header containing box size, and type, and the content itself; below, the box sizes and sample content are illustrative only:

Box Header (8 bytes)		Box Data (N bytes)
Size (4 bytes)	Type (4 bytes)	Box Data (N bytes)
2 K	ftyp File Type	majorBrand(int:4), minorVersion(int:4), compatibleBrands(int:32[])
4 GB	moov Movie Boxes…	MvhdBox{Flags(int:4) Version(int:4), CreateTime(int:4), ModTime(int:4), TimeScale(int:4), Duration(int:4), Rate(int:4), Volume(int:2)}[]
…	[others]	…
16 K	meta Metadata	Width, Height, Duration, Resolution, …; Photographers, Dates, Copyright

With sequential access, we’d have to read the 2KB + 4GB data just to get at the metadata. With random access, we read the first 8 bytes to find that it’s “ftyp” and 2KB size, then use the standard UNIX seek() call to skip ahead 2K bytes. We do the same with the next, encountering the “moov” type, then doing a seek() 4GB ahead. We ultimately find our “meta” header then read the 16K metadata we want. The total we’ve had to read is 8 + 8 + 16KB, so just 16 bytes overhead instead of over 4GB.

All this seek()ing is great but our objects live on S3, rather than the local EC2 filesystem. What to do?

HTTP supports byte range requests and S3 Get Object supports this mechanism too. We could conceivably modify ExifTool to simulate seek() and its partners tell() and read() using HTTP byte range requests when talking to S3 objects. I’ve looked at hacking the ExifTool RandomAccess.pm code but it’s but it’s been eons since I’ve written any Perl code and now find it challenging to read. It’s certainly do-able, but it’s a matter of time and priorities, and I’ve just sat on this problem for quite a while.

AWS Mountpoint for Amazon S3

On August 9, 2023, AWS announced this new capability, Mountpoint for Amazon S3 – Generally Available and Ready for Production Workloads. The excerpts below made it sound exactly like what we were looking for:

“…open source file client that makes it easy for your file-aware Linux applications to connect directly to Amazon Simple Storage Service (Amazon S3) buckets.”

“… large-scale read-heavy applications: data lakes, machine learning training, image rendering, autonomous vehicle simulation, ETL, and more. It supports file-based workloads that perform sequential and random reads, sequential (append only) writes, …”

“…many customers have existing applications, commands, tools, and workflows that know how to access files in UNIX style: reading directories, opening & reading existing files, and creating & writing new ones.”

It was easy try it out, we just install an RPM to our AWS Linux EC2 instance, create a mountpoint, then mount our S3 bucket; something like:

yum install -y https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
mkdir /s3
mount-s3 –read-only my-bucket-o-videos /s3

Now we can run the unmodified ExifTool against some sample files, make sure it works, and check the timing. If everything works as we hope, it should seek() through the file and quickly give us the metadata.

Let’s see:

$ time exiftool /s3/First-8K-Video-from-Space.mp4

File Name : First-8K-Video-from-Space.mp4

Directory : /s3

File Size : 3.3 GB

Major Brand : MP4 v2 [ISO 14496-14]

Duration : 0:03:09

Compressor Name : HEVC Coding

Creator Tool : Adobe Premiere Pro CC 2018.1 (Windows)

real 0m0.709s

It took 0.7 seconds to find and extract the metadata from an 3.3GB file; clearly we weren’t naively copying, but quickly seek()ing. By comparison, a copy from S3 to EC2 (using the Python Boto3 library) took 25 seconds, with Exiftool taking about 0.6 seconds.

We can do the same with an image file that has much more user-friendly metadata; an excerpt:

File Name : alex5.jpg

Directory : /s3/image/embedded

File Size : 2.5 MB

Image Description : 12-year-old Alex Frye checks his special viewing glasses prior to viewing the partial solar eclipse from a highway overpass in Arlington, VA, Thursday, Oct. 23, 2014, . Photo Credit: (NASA/Bill Ingalls)

Make : NIKON CORPORATION

Camera Model Name : NIKON D4

X Resolution : 300

Y Resolution : 300

Software : Adobe Photoshop Lightroom 5.6 (Macintosh)

This was also fast, but not as demonstrative since the file is relatively small.

See How They Run

The release states that the S3 Mountpoint runs on EC2 and even local environments:

Mountpoint can be used from an Amazon Elastic Compute Cloud (Amazon EC2) instance, or within an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (EKS) container. It can also be installed on your existing on-premises systems, with access to S3 either directly or over an AWS Direct Connect connection via AWS PrivateLink for Amazon S3.

The test results, shown above, were from a deployment on a conventional t2.micro EC2 instance.

My first test was actually local, on my Mac. I built a Docker container based on amazonlinux, installed the S3 Mountpoint RPM and ExifTool, then issued the mount command and ran ExifTool. To run the container, I had to give it extra privileges to the FUSE filesystem, and of course supply my AWS credentials so it could access S3:

docker run -it –rm

–cap-add SYS_ADMIN –device /dev/fuse

-v $(HOME)/.aws/credentials:/root/.aws/credentials:ro

–env AWS_PROFILE

s3-mount-test

It worked great and was surprisingly fast, even over the WAN to AWS. I also tried some manual seek() and tell() calls in Python to make sure I wasn’t missing something; nope, the S3 mountpoint behaved like a local file, for my purposes anyway.

Lambda and Fargate: no soup for you!

We’re big fans of serverless and are migrating more of our existing applications to AWS Lambda. Conspicuously absent from the release text above is any mention of Lambda. I expect this is because (even when launching Lambdas based on container images) it does not have the elevated privileges it needs for the mount, or maybe the Lambda base environment doesn’t support the FUSE filesystem.

I was hoping I could use Fargate where I cannot use Lambda, especially since the release text mentions ECS and EKS, but no dice; the README.md says:

These elevated privileges aren’t available in AWS-managed container environments like [AWS Fargate](https://aws.amazon.com/fargate/).

I hope AWS will consider adding this capability to Lambda and Fargate; both already support mounting NFS-like file shares with Amazon Elastic File System (to use EFS, I’d have to copy the file, which I’m trying to avoid).

Conclusion

Mountpoint for Amazon S3 is a great new service, especially for folks that have existing code or tools they can’t change which expect UNIX file system semantics. It doesn’t support all POSIX semantics, but enough to be useful for a large range of problems.

It could be boon to scientists with huge data files, like those used by NASA and NOAA: Hierarchical Data Format (HDF), Geospatial Data Abstraction Library (GDAL), NASA Planetary Data System, Version 4 (PDS4).

If you have any insight into using S3 Mountpoints, or something like them, for Lambda or Fargate, please get in touch!

Desperately seek()ing S3 Mountpoints