Pure-PHP Celery connector

Our contribution to the open source community, the Celery-PHP library, had one drawback: it needed a PECL extension that was pretty hard to compile, and not shipped with the most popular distributions. I can only imagine how hard it must have been for some people to set up a working environment.

So during the last month the library has undergone massive refactoring to abstract away the parts responsible for connecting to a queue. This allowed to create pluggable queue backend objects. Right now two are implemented: "pecl", the old code utilizing a PECL extension, and "php-amqplib", using a pure PHP AMQP library.

If you've used Celery-PHP before, you won't notice any difference. The library will just default to using PECL.

Here's how it works: I've added an extra parameter to the constructor of Celery class. If not supplied, it triggers an auto-detection system which first looks for the PECL extension, then checks if PHP-amqplib has been installed by composer, and gives up if neither of these is present. You can force the queue type by passing a string - currently 'pecl' and 'php-amqplib' are accepted.

The new PHP-AMQPLib backend passed most unit tests, but the ones checking how it behaves on long running tasks. The problem is that AMQPLib doesn't allow specifying an arbitrary interval when waiting for a new message, it's always an integer number of seconds. For this reason some tests don't behave as expected, but it shouldn't affect real-life code other than that "asynchronous" checks for result take a whole second. I've prepared a pull request to address this problem, but the developer didn't decide to pull it in yet.

If you want to try it out, pull the code from its branch on github. It will be available in the main branch once it's well tested and the php-amqplib pull request goes through (so vote for it if you want to use PECL-less Celery!).

Celery-PHP updated, works with Celery 3.0

Updated Celery binding for PHP is available from Github.

I've ran unit tests with latest Celery and RabbitMQ from Debian Squeeze. They went well without any change in code logic.

I've came across some problems during setup of testing environment, mainly during installation of php-pecl-amqp. The install_amqp.sh script is now updated with current librabbitmq-c URL and other minor changes to build properly on Squeeze. I've also updated documentation about testing and updated celeryconfig.py in test scenario to use Celery 3.0-style BROKER_URL variable instead of BROKER_HOST etc. used in Celery 2.x.

Software versions used:

• Celery 3.0.19
• RabbitMQ 2.8.4
• PHP 5.4.4
• PHP-AMQP 1.0.10
• librabbitmq 0.3.0

Celery task queue with PHP

We have released a Celery client for PHP some time ago. What does it do and why is it useful?

Celery is a piece of software that helps you easily run time-consuming tasks. If you have ever used system() or exec() to create a big ZIP file or encode a video, you probably already know what the problem is. If you didn't - well, time-consuming tasks executed from within a PHP script (or any other web application) usually generate many problems, unresponsive user interface and timeouts being two major ones. There are many ways to overcome these issues, but face it - spending many hours and hundreds of lines of code just to create a ZIP file isn't especially effective.

So how does a basic Celery application to create a ZIP file look like? Let's see:

#!/usr/bin/python
from celery.task import task
from subprocess import Popen, PIPE, STDOUT

@task
def create_zip(zip_path, files_path):
    command = ("zip", zip_path, files_path)
    return Popen(command, stdout=PIPE, stderr=STDOUT).communicate()[0]


The above code calls the command-line zip command with two arguments: path to a zip file and path of the files to be archived. "def" is Python for "function". (this is just an example, in real world you will probably want to validate paths and replace zip command with Python's zipfile module)

Now you will probably want to call that code from your PHP application. First schedule the task for execution:

$c = new Celery('localhost', 'myuser', 'mypass', 'myvhost');
$_SESSION['zip_result'] = $c->PostTask('tasks.create_zip', array('/home/user/file.zip', '/home/user/images/'));

Then you will want to somehow asynchronously display the result to user. You can use AJAX or some other technique to display the result of this script every second:

$result = $_SESSION['zip_result'];
if(!$result->isReady())
{
    echo 'Please wait...';
}
else
{
    echo '<a href="/file.zip">Ready!</a>';
}


That's it, as easy as this. Check out the Celery-PHP documentation and try it out.

Also, the examples above are oversimplified - make sure you validate your input data, handle errors and don't unnecessarily poll the server.

RabbitMQ 2.6.0

It looks like new RabbitMQ is just behind the corner.