Varnish and FLV streaming

When doing usual website speed optimization to a website that has lots of videos, I have encountered a problem. Videos started breaking for no apparent reason, they loaded very slowly and in general were unusable.

It turned out it was because of two things. One was that Apache has been set up to compress everything except images with mod_deflate - this does not play well with streaming. Another problem was that Varnish running in front of the web server wasn't set up to do streaming, so it first fetched whole video to its cache and then sent it to the browser. It was slow.

The solution was to upgrade Varnish to 3.0.1 and add the following code to vcl_fetch:

if(beresp.http.Content-Type && beresp.http.Content-Type ~ "video") {
        set beresp.do_stream = true;
}

Optimizing website performance is not an easy job and has tons of catches like this.Order optimization now!

Celery task queue with PHP

We have released a Celery client for PHP some time ago. What does it do and why is it useful?

Celery is a piece of software that helps you easily run time-consuming tasks. If you have ever used system() or exec() to create a big ZIP file or encode a video, you probably already know what the problem is. If you didn't - well, time-consuming tasks executed from within a PHP script (or any other web application) usually generate many problems, unresponsive user interface and timeouts being two major ones. There are many ways to overcome these issues, but face it - spending many hours and hundreds of lines of code just to create a ZIP file isn't especially effective.

So how does a basic Celery application to create a ZIP file look like? Let's see:

#!/usr/bin/python
from celery.task import task
from subprocess import Popen, PIPE, STDOUT

@task
def create_zip(zip_path, files_path):
    command = ("zip", zip_path, files_path)
    return Popen(command, stdout=PIPE, stderr=STDOUT).communicate()[0]


The above code calls the command-line zip command with two arguments: path to a zip file and path of the files to be archived. "def" is Python for "function". (this is just an example, in real world you will probably want to validate paths and replace zip command with Python's zipfile module)

Now you will probably want to call that code from your PHP application. First schedule the task for execution:

$c = new Celery('localhost', 'myuser', 'mypass', 'myvhost');
$_SESSION['zip_result'] = $c->PostTask('tasks.create_zip', array('/home/user/file.zip', '/home/user/images/'));

Then you will want to somehow asynchronously display the result to user. You can use AJAX or some other technique to display the result of this script every second:

$result = $_SESSION['zip_result'];
if(!$result->isReady())
{
    echo 'Please wait...';
}
else
{
    echo '<a href="/file.zip">Ready!</a>';
}


That's it, as easy as this. Check out the Celery-PHP documentation and try it out.

Also, the examples above are oversimplified - make sure you validate your input data, handle errors and don't unnecessarily poll the server.