Cookies on the Fluid Rock website

  • Our FR_og

    Our Fluid Rock Blog. Where we keep an eye on the latest technologies & trends that are flying about.

Amazon Web Services & Symfony2 Integration

This week, we migrated storage for an application that was built in Symfony2 from local file storage to Amazon S3. Here we outline some of the challenges we met along and the solutions that worked in this instance.

Figure 1.0

Figure 1.1

Figure 1.2

This blog post is aimed at developers who may be facing a similar task and we hope some of the solutions we adopted can help. The application in question enables authenticated users to add digital artefacts including text, images, audio and video to a user created story. The application had originally been written to upload the files to the local file system and store metadata about the files to a database table (Entity). We now wanted to store the original and processed files on the Amazon cloud.

The migration required us to:

  • Successfully upload the files to Amazon S3
  • Relate the uploaded file in S3 to a database entry
  • Process the files being uploaded especially large files (images and video) using
  • Amazon Elastic Transcoder
  • Implement CloudFront for content delivery

The application was originally developed to upload the files to our server along with the file metadata using a regular form and file input type. The biggest issue with this approach is the file size is limited by PHP server settings e.g. max_upload_size, post_max_size and memory_limit. Also when the file was uploaded to the server it then needed to be transferred to Amazon S3.

On reflection, to build in longevity, we decided that this approach was not really feasible. We wanted users to upload the original assets that we would store for archival purposes, while serving out optimised files online. This meant allowing users to upload files of up to 5Gb. Thankfully there is a JavaScript library, FineUploader that gets around these issues.

FineUploader  is a JavaScript file uploader library that supports uploading directly to S3 and supports chunk uploading. Chunking is a feature of the HTML5 File API that allows a client’s browser to split a file into multiple parts (chunks) and send them to a server. The server then needs to assemble the chunks into the file, which Amazon S3 supports. This feature allows users to get over the file size limit. It also allows for a nice progress bar.  A demo can be found on the FineUploader website as can documentation on how to implement FineUploader. Figure 1.0 describes the sequence of events that occurs in uploading a file to amazon S3

As stated earlier, the form metadata and file uploaded were submitted to the server at the same time. With the integration of FineUploader we needed to change this architecture. The user now gets presented with the file upload only, and on successful upload to Amazon S3 the server sends the file metadata form for the user to fill out. A hidden field is used to store the location off the file sent to Amazon S3. When the user submits the metadata form, the server needs to validate that the file actually exists on Amazon S3, and depending on file type, trigger a transcoding process on the file.

To access files on Amazon S3 we used a Symfony2 bundle called KnpGaufretteBundle  from KnpLabs (http://knplabs.com/en). It is a filesystem abstraction layer. This abstraction layer allows you to develop applications without needing to know where all their media files will be stored or how. This significantly speeds up development time. 

So Gaufrette was used to Read and Write the files to the filesystem, while LiipImagine was used to process the images in the required way. For best performance, defining and hardcoding the thumbnail, large, small etc., is recommended, the advantage being that the images are processed once and we could use Cloudfront to distribute the files. The disadvantage of this approach is if the website design needs to change or we want to syndicate the data, we are stuck with these image sizes unless we build a new script to process all images into the new size. In our situation, we didn't want to be restricted if the data was to be used elsewhere in the future, so we processed and stored them in a manner that they can be manipulated as required in the future using a Twig function.

So to recap, someone has already built a library that transforms images and can use KnpGaufretteBundle to access images on S3, LipImagineBundle from L//P.  LiipImagineBundle downloads the image from S3 onto the server and transforms the image into whatever configuration you would like. The transformed image can then be cached onto S3 or cached locally on the filesystem. Figure 1.1 shows the sequence of events for transforming the images

The next media format we looked at was video. Amazon Elastic Transcoder has great support for video transcoding. We wanted videos to be processed into multiple formats MP4, WebM and three versions of HLS. We also needed to create thumbnails of the video. In essence we wanted to create YouTube functionality.

To use Amazon Elastic Transcoder, you first create a Pipeline. A Pipeline tells the job what S3 bucket to get the video file from and what S3 bucket to store the processed videos and thumbnails in. Amazon Elastic Transcoder comes with a lot of preset video configurations. Amazon calls these configurations Presets. To process a video you have to manually create a Job with a pipeline and one or more presets. After a user enters the metadata for a video the web server creates a new video processing job and uses AWS SDK for PHP to connect and send the job to Amazon Elastic Transcoder service.

As the time to transcode the video is dependent on the the length and quality of the video, and the length of the queue of videos waiting, it is important for the user to know what is happening and to send appropriate messages. This leads onto the next issue.  When the job is added to a queue it can take anywhere from a minute to an hour before it is processed. We needed some way to know what the state of the processing job was and to notify the user if its ‘processing’, ‘processed’, or if an ‘error’ occurred. Amazon Elastic Transcoder, does not support notification systems, to communicate the state of the job, another Amazon service, Amazon Simple Notification Service (Amazon SNS) was employed for push notification service. SNS is a fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients. In our case the recipient will be our server. To use SNS you need to implement a listener for the push notifications. Figure 1.2 describes the sequence of events that occurs when a video is to be processed.

Avoid the Pitfalls

  • Amazons documentation is not as clear as it could be. We need to use a lot of third party documentation and blogs. Stackoverflow as always was of massive help and a big time saver.
  • One of the biggest issue we experience with AWS was that things took time to process. For example if we uploaded a thumbnail to s3 and wanted to display it straight away to the user using CloudFront, the image would not show because it has not been distributed across CloudFront yet. If you change a buckets policy or CORS Configuration, files already upload to that bucket did not get those permissions transferred to CloudFront for days. Our solution was to abandon this and store locally.
  • FineUploader aims to make file-uploading on the web possible in every browser and mobile device. It is cross-browser, dependency-free, and 100% JavaScript.
  • The S3 upload wizard wasn't great, so we sourced CloudBerry Explorer to interface with S3 and CloudFront.
  • Our chosen player, JW Player was tempermental, sometimes playing the video, and other times not. After extensive exploration of their website and code, we found a notification in a snippet that the Premium package was required to support streaming of HLS

Some Resources

 

 

Follow us :