13.10.2016 . created by Paul
This blog post is aimed at developers who may be facing a similar task and we hope some of the solutions we adopted can help. The application in question enables authenticated users to add digital artefacts including text, images, audio and video to a user created story. The application had originally been written to upload the files to the local file system and store metadata about the files to a database table (Entity). We now wanted to store the original and processed files on the Amazon cloud.
The migration required us to:
The application was originally developed to upload the files to our server along with the file metadata using a regular form and file input type. The biggest issue with this approach is the file size is limited by PHP server settings e.g. max_upload_size, post_max_size and memory_limit. Also when the file was uploaded to the server it then needed to be transferred to Amazon S3.
As stated earlier, the form metadata and file uploaded were submitted to the server at the same time. With the integration of FineUploader we needed to change this architecture. The user now gets presented with the file upload only, and on successful upload to Amazon S3 the server sends the file metadata form for the user to fill out. A hidden field is used to store the location off the file sent to Amazon S3. When the user submits the metadata form, the server needs to validate that the file actually exists on Amazon S3, and depending on file type, trigger a transcoding process on the file.
To access files on Amazon S3 we used a Symfony2 bundle called KnpGaufretteBundle from KnpLabs (http://knplabs.com/en). It is a filesystem abstraction layer. This abstraction layer allows you to develop applications without needing to know where all their media files will be stored or how. This significantly speeds up development time.
So Gaufrette was used to Read and Write the files to the filesystem, while LiipImagine was used to process the images in the required way. For best performance, defining and hardcoding the thumbnail, large, small etc., is recommended, the advantage being that the images are processed once and we could use Cloudfront to distribute the files. The disadvantage of this approach is if the website design needs to change or we want to syndicate the data, we are stuck with these image sizes unless we build a new script to process all images into the new size. In our situation, we didn't want to be restricted if the data was to be used elsewhere in the future, so we processed and stored them in a manner that they can be manipulated as required in the future using a Twig function.
So to recap, someone has already built a library that transforms images and can use KnpGaufretteBundle to access images on S3, LipImagineBundle from L//P. LiipImagineBundle downloads the image from S3 onto the server and transforms the image into whatever configuration you would like. The transformed image can then be cached onto S3 or cached locally on the filesystem. Figure 1.1 shows the sequence of events for transforming the images
The next media format we looked at was video. Amazon Elastic Transcoder has great support for video transcoding. We wanted videos to be processed into multiple formats MP4, WebM and three versions of HLS. We also needed to create thumbnails of the video. In essence we wanted to create YouTube functionality.
To use Amazon Elastic Transcoder, you first create a Pipeline. A Pipeline tells the job what S3 bucket to get the video file from and what S3 bucket to store the processed videos and thumbnails in. Amazon Elastic Transcoder comes with a lot of preset video configurations. Amazon calls these configurations “Presets”. To process a video you have to manually create a Job with a pipeline and one or more presets. After a user enters the metadata for a video the web server creates a new video processing job and uses AWS SDK for PHP to connect and send the job to Amazon Elastic Transcoder service.
As the time to transcode the video is dependent on the the length and quality of the video, and the length of the queue of videos waiting, it is important for the user to know what is happening and to send appropriate messages. This leads onto the next issue. When the job is added to a queue it can take anywhere from a minute to an hour before it is processed. We needed some way to know what the state of the processing job was and to notify the user if its ‘processing’, ‘processed’, or if an ‘error’ occurred. Amazon Elastic Transcoder, does not support notification systems, to communicate the state of the job, another Amazon service, Amazon Simple Notification Service (Amazon SNS) was employed for push notification service. SNS is a fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients. In our case the recipient will be our server. To use SNS you need to implement a listener for the push notifications. Figure 1.2 describes the sequence of events that occurs when a video is to be processed.