Dropping the botnet

Google Analytics sent me an email earlier today about changes in their data retention policy:

Today we introduced granular data retention controls that allow you to manage how long your user and event data is held on our  

servers. Starting May 25, 2018, user and event data will be retained  

according to these settings; Google Analytics will automatically delete  

user and event data that is older than the retention period you select.  

Note that these settings will not affect reports based on aggregated data.

Action: Please review these data retention settings and modify as needed.

Before May 25, we will also introduce a new user deletion tool that allows  

you to manage the deletion of all data associated with an individual user  

(e.g. site visitor) from your Google Analytics and/or Analytics 360  

properties. This new automated tool will work based on any of the common  

identifiers sent to Analytics Client ID (i.e. standard Google Analytics  

first party cookie), User ID (if enabled), or App Instance ID (if using  

Google Analytics for Firebase). Details will be available on our Developers  

site  

I decided to delete Google Analytics tracking code from both Massive VPS and Massive Scale websites. It was an invaluable tool in the early days when we still had to advertise in Adwords but these days there's really no excuse for voluntarily sending our visitors' data to Google.

The first performance issue I've fixed

The first performance problem I've found and fixed in code written by someone else was in Gadu-Gadu. GG was an Windows instant messenger very popular in Poland in the early 2000s. It was closed, binary-only code but due to its popularity a modding "underground" has been formed quickly and soon, its protocol was reverse engineered.

I was about 18 years old at the time and I had a running competition with a friend to make our Windows systems look and feel like Amiga OS. We had the basic things covered, but some programs were not themeable at the time. GG was one of them and I thought: challenge accepted.

I decided to use PowerGG, a live patching framework by Ajron. I've quickly found out that the conversations were displayed in an embedded Internet Explorer control and were plain old HTML + Javascript. Then I've found a way to patch that HTML code and it looked just like Amiga.

GG always had this annoying bug that the longer your conversation was the slower the program was. And boy, our conversations were long these days, we were teens and had lots of time on our hands. Sometimes we talked so much that the program froze for several seconds after I pressed Enter.

After looking at the Javascript code it quickly became obvious why this is happening: GG was looking for a <div> containing the last message, and then appended the new message after that <div>. But instead of getting the number of <div>s and finding the last one, it iterated over all of them. And, since these were days long before the V8 engine, Javascript wasn't very fast on 400 MHz CPUs and iterating several hundred DOM elements took a lot of time. It was a three-line code change and it made the messages send instantly.

If you want to verify my story. the PowerGG plugin was briefly named "Amigo" and then released as "Foolteen".

Varnish for Joomla maintenance release

It seems that with Joomla! 3.4.3, our patch for Varnish support could not be applied, so we made a maintenance release - as usually, available for free for the existing clients. There are no new features, so you won't need it unless you're planning to upgrade to Joomla! 3.4.3 or later.

Apache mod_status page mangled by mod_rewrite?

Quicktip: if a complicated mod_rewrite setup is making your /server-status inaccessible, just paste the following two lines above all the rewrite rules:

RewriteCond %{REQUEST_URI} /server-status
RewriteRule .* - [L]

and you're ready to set up monitoring of Apache processes.
If the rules are in /etc/apache2, not in .htaccess, you need to reload the web server too:

/etc/init.d/apache2 reload

You know you shouldn't use .htaccess anyway because it hurts performance, right?

Upgrade your Varnish!

Surprisingly, Varnish Software announced that they will not update Varnish 3.0 and that users need to upgrade to 4.0 as soon as possible. This is not an expected decision, it has been stated more than once on the varnish-misc list that 3.0 will be supported for a long time in the foreseeable future. This means it's not secure to use Varnish Cache 3 on your server, if you keep running this software you may get hacked.
If you need help upgrading your setup from Varnish 3 to 4, contact us. We can convert your old configuration file in-place on your server or set up a staging environment and update your server after it's tested.
The Varnish-Joomla package includes a Varnish 4 VCL since it has been released a year ago, so the upgrade path is straightforward.

Varnish for Joomla! 3.4

The latest Varnish Cache package for Joomla! v1.3.5 is available. Now you can accelerate Joomla CMS version 3.4.
Existing clients can contact us to get the new version for free.
If you're running a high traffic website and don't want to lose clients to downtime, perhaps we can interest you in our inexpensive server monitoring services?

How not to check if Varnish is working.

Our new Varnish clients who choose to install everything themselves want a way to verify if it's working - and often fall in the same trap: they use the isvarnishworking.com website which has good SEO and promises to be an easy way to check if everything is running correctly.
Most often the answer is "Nope" - indicating that Varnish isn't operating - even thought the cache is running. I checked I can work around that - maybe they're doing some simple check that can be passed with a minor modification to the VCL file.
Unfortunately, I've found out that the data presented on this website is completely useless - the headers they show to visitors are different from actual headers sent by the server!
You can see that yourself below:

The headers isvarnishworking.com shows

Actual headers, as seen by wget

I don't understand what's the point of building a service showing fake HTTP headers.

When testing, just use the Network tab in developer tools in your browser, or wget/wbox if you're a command line user.

Patch for 3.3.1

Joomla! 3.3.1 changed some of the patched code, so the patches from 3.3.0 won't work with 3.3.1. Existing users need to ask for the new version (for free, as usually).

VividCortex review

EDIT: Baron Schwartz kindly asked me to remove this blog post. I'm not going to do this, but I agree it was too harsh, it's my fault I didn't read the terms of service I was going to review.

Back story: I reviewed free trial of a service. The legal document I accepted when registering did not allow trial users to publish their reviews. They contacted me about it and you can see my original reaction below. I don't think this is the best way to promote a product, but it's just my personal opinion, and Baron - being an experienced blogger - claims it's normal practice.

Remember, kids: use only Free Software if you don't want any trouble!

There was a 3 pages long VividCortex review here, but they said writing a public review violates their ToS unless they accept it. If they don't want free publicity, I don't care and wish them all the best promoting their product this way.

Varnish 4 support and benchmark

A Varnish4-ready configuration file for the Varnish-Joomla pack is now ready. We also made a little benchmark. TLDR: there are no significant differences between Varnish 3 and Varnish 4.

The Varnish under load has been a 2-core VPS with 1GB RAM. Varnish 3.0.4 from repos has been tested against Varnish 4.0.0 built from a source tarball. Both were configured in an identical way, except from the VCL of course. There was Joomla 3.3 behind Varnish, and cache has been pre-warmed before testing in both cases.

The machine generating load has been a 6-core hardware server with 32 GB RAM. The software used to generate load was the Apache Benchmark.

The machines were in different data centers owned by the same company, the traffic has been routed via internal network (6 hops, 2ms ping). Both machines were connected using a gigabit ethernet card. Bandwidth usage has been around 500 Mbit/s at all time. Traffic volume has been bound by CPU usage on the Varnish machine with system load approaching 20 at the end of the test for both Varnish versions.

Both Varnish versions performed similarly. It was worrying that Varnish 4 had a number of failed connections - the test has been repeated more than one time, and the failed connections always were there. This did not happen with Varnish 3.

Without further ado: Varnish 3 and Varnish 4 results from AB.

If you need help building the new Varnish from source, converting your configuration or vmods, or anything else Varnish-related, don't hesistate to contact us.

Varnish for Joomla! 3.3

The Varnish-Joomla pack has been upgraded to support Joomla! 3.3 which has been released last week. Existing clients are eligible for a free upgrade after contacting us.

Originally we planned this release to support Varnish 4 which has been released two weeks ago, but it was more work than expected and clients were more interested in Joomla 3.3 support.

Varnish 4 doesn't bring any significant performance gains for small-to-middle sized web sites, it has separate backend and frontend processing threads, which could be better in certain high-traffic scenarios. It looks like this release was more focused on cleaning up the Varnish Configuration Language and bringing log analysis features (which are awesome, by the way).

The future of Varnish-Joomla

We got a lot of questions recently about the Varnish-Joomla package and support for future Joomla versions. Of course the package is going to be still developed in foreseeable future, and Joomla 3.5 support is planned.
This larger than usual volume of questions made me try to get my hands on a copy of Joomla 3.5 development code - only to discover that there is none yet. The next release of Joomla will be version 3.3 and it will be released on the April Fool's Day. 3.5, the long-awaited Long Term Support release initially scheduled for March 2014, is postponed to October.
Long-time Joomla users are used to rapid changes of plans, and I can have a hassle-free summer (after releasing our software for 3.3 in April, of course).

e503 no service

Pound is a popular load balancer / SSL terminator. As it usually happens with open source programs, there is plenty of documentation and self-help available on the internet, but there's one error message I could not find. Pound denied access to the web site and logged the following message:

pound: (756000001700) e503 no service "GET / HTTP/1.1" from 12.34.56.78 www.thedomain.com

I've found out that access to a service has been blocked by a HeadDeny directive. The specific problem was denying requests containing the X-Forwarded-For header - which blocked access by users behind proxies. Changing HeadDeny in service to a HeadRemove in listener fixed the problem.

Speed your HikaShop with Varnish

HikaShop is an e-commerce solution for Joomla. They have a pretty usable free edition, and reasonably priced commercial editions with support.
We worked together with the Hikari team to make our Varnish-Joomla integration work with all editions of their extension. The result of this work is available in version 1.3.1 of our package, which was out in November 2013.
No matter if your shop receives thousands or millions of view every day, your prospective clients deserve fast page loads. Visitors who don't have anything in their cart will enjoy viewing pages from cache, and more server processing power will remain for these who actually need it (visitors with items in their carts, visitors doing checkouts, your staff). Everybody wins.

Joomla 1.0 spam

This news comes from our secure Joomla 1.0 hosting department. Recently some of our Joomla 1.0 sites became sources of spam. Apparently somebody (with a narrow range of IP addresses geolocating to Malaysia) has found a security problem in VirtueMart recommendation module which allows them to send emails with POST requests.
The solution was to paste the following code into Apache configuration (or .htaccess):
# vm spam patch
RewriteEngine On
RewriteCond %{THE_REQUEST} ^.*(page=shop\.recommend).* [NC]
RewriteRule ^.*$ - [F,L]


If you're fed up dealing with lack of security in old Joomla versions, give us a shot. We run Joomla on latest Apache, PHP and MySQL versions, keep a month worth of backups and have several security measures.

Varnish-Joomla Pack 1.3.1 with Joomla 3.2 support is out

New features:

• Support for Joomla! 3.2
• Support for HikaShop

Joomla! 3.2 support has been tested with core components and the following extensions:

• JomSocial
• Akeeba Backup
• Admin Tools
• Kunena Forum
• Community Builder
• HikaShop

Virtuemart does not support 3.2 (and never will), and that's the only reason why it's not included in the test. We continue to support it. As explained here, the patch for Joomla! 3.2 does not prevent creating new sessions for anonymous users. If you have a lot of cache misses caused by anonymous users, there will be an extra disk write operation per cache miss - if this concerns you, move your sessions to memcached or XCache.

Upgrade from 1.3.0 and 1.2.2: Just replace the default.vcl file. As always, the upgrade is free for existing users.

New users - order to speed up your Joomla! and its extensions.

Varnish for Joomla! 3.2 on its way

Varnish for Joomla 3.2 will be released next week. The plan was to release it this week, but Joomla 3.2 increased its dependence on sessions and the patch can not operate the same way it did before.
In previous versions, we did what we could to not even create sessions in the database, thus lowering the number of disk writes required to serve users' requests. Unfortunately it doesn't look like it's possible any more and the best we can do is not sending session cookies to the user.
This is a new approach and while it passes all internal tests against a vanilla Joomla install, it needs more testing with supported 3rd party components.
I'm sorry for the delay, we got many requests about 3.2 availability in the past two weeks.

Varnish-Joomla package 1.3.0 released, with cache purging

The new Varnish-Joomla package 1.3.0 is now available. New features include:

• Purging cache when saving articles
• Support for more Joomla! extensions and static file types
• Improved cache hit rate because of Accept-Encoding normalization
• Support for Piwik and CloudFlare
• More debugging headers

The biggest news is cache purging - this means that the moment you save your article, it is visible to your visitors. Behind the curtains, our Joomla plugin connects to your Varnish and tells it to remove cached copies of the saved page, pages of categories it belongs to, and the front page. This feature is optional, you don't need to install the plugin if this is not the desired behavior.

The package has support for Joomla 2.5.4+, 3.0.x and 3.1.x, just like the previous version.

There is good news for our existing customers: the patch to Joomla core didn't change, all features are implemented with the Varnish VCL file and an easily installable Joomla plugin.

As usually, the upgrade is free, contact us to receive the latest package if you've already purchased it.

Make Joomla's Nextend Menu faster

If you're using Nextend's Accordion Menu for Joomla, there's a good chance it's slowing you down. Adding this simple patch to make it use cache cut down average loading time by 900 ms on a Joomla 2.5 site.

• Open libraries/nextend/accordionmenu/treebase.php
• Find the following part:

        $this->renderItem();

    }

    

    function renderItem() {

• Replace it with:

        $cache = & JFactory::getCache();

        $cache->call(array($this, 'renderItem'), JURI::getInstance()->toString());

    }

    

    function renderItem($dummy=NULL) {

Alternatives to mod_rpaf

For years, mod_rpaf has been the standard module for running Apache behind a reverse proxy. Fortunately, it's not the only one and there are alternatives.

• mod_extract_forwarded which is available in EPEL, so it's available for CentOS systems without compling
• mod_remoteip which is built into Apache 2.4 (but good luck finding a production server running 2.4)
• mod_rpaf modified to support X-Forwarded-Proto/X-Forwarded-HTTPS/X-HTTPS

New software available: HTML5 prerender support for Joomla! CMS

Massive Scale is proud to announce its new software: HTML5 Prefetch/Prerender for Joomla!. It makes use of prerendering capabilities of web browsers like Chrome, Firefox or Internet Explorer 11 using machine learning techniques. You will learn more about prefetching, prerendering and Joomla! on the product page.

Thread creation error in Varnish

Today I installed Varnish on a pretty standard system (CentOS + WHM) and the service wouldn't start. The system log (/var/log/messages) contained the following:

Oct  7 04:15:46 server varnishd[20836]: Child (20846) Panic message: Assert error in WRK_BgThread(), cache_pool.c line 582:#012  Condition((pthread_create(thr, ((void *)0), wrk_bgthread, bt)) == 0) not true.#012errno = 11 (Resource temporarily unavailable)#012thread = (cache-main)#012ident = Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit,no_waiter#012

The settings for number of processes and file descriptors in /etc/security/limits.{conf,d} were correct and it still didn't start. The problem turned out to be with stack size - I have no idea why this is happening in just this one system and works with every other centos, but the solution was as following:

1. Patch /etc/init.d/varnish adding a new ulimit call in start()

        # Stack size
        ulimit -s unlimited

2. Increase system limits in /etc/security/limits.conf accordingly

varnish         stack   soft            unlimited
varnish         stack   hard            unlimited

3. Restart Varnish

After successfully running Varnish this way, be sure to change the stack limit back to a reasonable size! Unlimited is not a very wise setting, but helps debugging the problem. Now Varnish and Joomla could happily work together.

Nextend modules for Joomla behind Varnish Cache

Nextend makes pretty and well-written modules for Joomla and Wordpress. I've optimized a website which uses one of them and it had a weird issue: sometimes the module would appear unstyled and CSS/JS files in the "media/nextend/cache/" directory returned 404. After reading some code the solution was obvious: Varnish was set to cache pages for at least 2 hours, while the Nextend module only cached it for 15 minutes, and deleted old files. Because every time it creates a new directory for updated cache content, it deletes old ones - cached HTML pages were referencing URLs which did not exist any more.

The solution was very simple: in Joomla admin go to Extensions - Plugins - Nextend Library, then click Configure in the Basic Options and set the cache time to the same value your Varnish installation uses. For the best Varnish + Joomla experience, see our offer.

Zend opcode cacher in PHP 5.5: a security perspective

The new zend opcache extension built into the latest version of PHP is, as we've checked, a bit faster than the current best - XCache. There's, however, a security concern.

Both extensions make it possible to clear the whole cache programatically (from within a script). In Zend it's the opcache_reset() function and in XCache it's xcache_clear_cache(XC_TYPE_PHP).

If you run shared hosting, you allow your users to execute arbitrary code in your PHP interpreter. This, naturally, includes the two functions mentioned above. If you rely on the opcode cache to provide desired level of service for your customers, you don't want them to clear everybody's caches. If a malicious or badly written script clears your cache, say, every minute - there is no benefit from caching. The whole idea is that you cache the most used files once, and use the cached version from RAM. What this means for a shared host is increased CPU and possibly disk IO usage. In some cases, this can turn into a Denial of Service (DoS) attack.

So let's take this simple script and see what Zend Opcache will do when you execute it:

<?php

error_reporting(E_ALL);
ini_set('display_errors', 1);

opcache_reset();

echo 'OK';

Well, Zend happily cleared the cache. Now let's see what XCache does:

<?php

error_reporting(E_ALL);
ini_set('display_errors', 1);

xcache_clear_cache(XC_TYPE_PHP);

echo 'OK';

The XCache developer envisioned this problem. Certain functions, which affect other users' experience, trigger a login page. The username and password is set by an administrator system-wide in php.ini and is not viewable by regular users.

Conclusion

If you run a shared host and want to speed it up using opcode caching, you should use XCache rather than the new Zend Opcache. Zend is only better by 10%, it's not a number worth risking a DoS attack. On the other hand, if you and only you control the code that's run on your servers, this security issue won't affect you.

Possible fix

You could (and should) use the disable_functions directive in php.ini

(Need help installing XCache or securing Zend?)

PHP 5.5: Zend Optimiser+ OPcache vs XCache

PHP 5.5 has a new feature: built-in opcode cache! It's an alternative for extensions like XCache, APC or eAccelerator. Let's see how it performs compared to established solutions.

Test environment

Server: OpenVZ container running on 1 Xeon W3520 CPU core, high I/O priority (to ensure CPU-bound execution), 1 GB RAM. Requests originating from another VPS on that system.

Software: Debian 7.0 i386, nginx 1.2.1, PHP 5.5.0 from dotdeb, XCache SVN trunk from 2013-06-23 (r1269).

(This test may or may not reflect what is happening on dedicated servers. Unfortunately, we don't have a spare server-grade physical machine at the moment. That being said, the machine which held the VPS is under a small network and IO load.

PHP-FPM configuration

pm = dynamic
pm.max_children = 25
pm.start_servers = 4
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 5000

Zend Opcache has been configured as following:

root@opcode:/var/www# grep ^opcache /etc/php5/fpm/php.ini
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=4
opcache.max_accelerated_files=3907
opcache.fast_shutdown=1

XCache has been configured as following:

xcache.size = 128M
xcache.count = 3
xcache.cacher = On

Joomla

Joomla! 3.1.1. Installation was tricky, installation/application/framework.php had to be edited, this line in particular: ini_set('display_errors', true); - otherwise AJAX calls didn't succeed because JSON has been polluted with error messages.

The "Default English" sample data set has been installed. Joomla cache has been disabled because it's not what we want to test.

Test

The Apache Benchmark has been used to measure average response time and number of requests served per seconds. The command line was:

ab -n 10000 -c 20 

Ten thousands requests have been performed by 20 concurrent threads. This corresponds with the pm.max_children=25 setting in php.ini.

No cache

Performance is clearly CPU-bound. CPU usage is 100%, RAM utilization is about 880 MB out of 1GB. Disk I/O is negligible measured from both within the container (%wa 0) and the hypervisor (%wa 7).

Apache Benchmark output (Unfold)

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 176.31.28.48 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.2.1
Server Hostname:        176.31.28.48
Server Port:            80

Document Path:          /
Document Length:        8626 bytes

Concurrency Level:      20
Time taken for tests:   1743.994 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      89890000 bytes
HTML transferred:       86260000 bytes
Requests per second:    5.73 [#/sec] (mean)
Time per request:       3487.989 [ms] (mean)
Time per request:       174.399 [ms] (mean, across all concurrent requests)
Transfer rate:          50.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   656 3483 1020.6   3166    9587
Waiting:      403 2999 778.5   2812    8757
Total:        656 3483 1020.6   3166    9587

Percentage of the requests served within a certain time (ms)
  50%   3166
  66%   3528
  75%   3849
  80%   4091
  90%   4797
  95%   5540
  98%   6695
  99%   7420
 100%   9587 (longest request)

Zend OPcache enabled

It looks like the CPU hasn't been fully utilized with 20 concurrent requests. The usage fluctuated between 70% and 90%. I/O and memory usage were, as previously, within reasonable limits.

Apache Benchmark output (Unfold)

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 176.31.28.48 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.2.1
Server Hostname:        176.31.28.48
Server Port:            80

Document Path:          /
Document Length:        8626 bytes

Concurrency Level:      20
Time taken for tests:   658.408 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      89890000 bytes
HTML transferred:       86260000 bytes
Requests per second:    15.19 [#/sec] (mean)
Time per request:       1316.817 [ms] (mean)
Time per request:       65.841 [ms] (mean, across all concurrent requests)
Transfer rate:          133.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   421 1316 426.2   1214    5664
Waiting:      275 1053 355.8    986    4329
Total:        421 1316 426.2   1214    5664

Percentage of the requests served within a certain time (ms)
  50%   1214
  66%   1333
  75%   1448
  80%   1537
  90%   1826
  95%   2065
  98%   2484
  99%   2780
 100%   5664 (longest request)

XCache

Just like previously, the load has been CPU-bound even after enabling XCache.

Apache Benchmark output (Unfold)

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 176.31.28.48 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.2.1
Server Hostname:        176.31.28.48
Server Port:            80

Document Path:          /
Document Length:        8626 bytes

Concurrency Level:      20
Time taken for tests:   769.788 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      89890000 bytes
HTML transferred:       86260000 bytes
Requests per second:    12.99 [#/sec] (mean)
Time per request:       1539.576 [ms] (mean)
Time per request:       76.979 [ms] (mean, across all concurrent requests)
Transfer rate:          114.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   601 1538 431.1   1439    4401
Waiting:      393 1232 361.6   1178    4205
Total:        601 1539 431.1   1440    4401

Percentage of the requests served within a certain time (ms)
  50%   1440
  66%   1582
  75%   1696
  80%   1784
  90%   2079
  95%   2333
  98%   2847
  99%   3179
 100%   4401 (longest request)

Graphs

Time graph: less is better

Requests per second graph: more is better

Comment

Well, XCache is no more the best opcode cache around. We're looking forward to widespread adoption of PHP 5.5+. Unfortunately, there is currently no GUI for nicely analyzing cache usage, like XCache has. However, Zend exposes a new function, accelerator_get_status(), which could be used to create such a GUI - so certainly somebody will release relevant GUI in the next weeks.

EDIT: We didn't have to wait long - as a reader pointed out, there is a GUI for Zend Optimizer.

As usual, if you need help, we will make your website fast for you.

Pure-PHP Celery connector

Our contribution to the open source community, the Celery-PHP library, had one drawback: it needed a PECL extension that was pretty hard to compile, and not shipped with the most popular distributions. I can only imagine how hard it must have been for some people to set up a working environment.

So during the last month the library has undergone massive refactoring to abstract away the parts responsible for connecting to a queue. This allowed to create pluggable queue backend objects. Right now two are implemented: "pecl", the old code utilizing a PECL extension, and "php-amqplib", using a pure PHP AMQP library.

If you've used Celery-PHP before, you won't notice any difference. The library will just default to using PECL.

Here's how it works: I've added an extra parameter to the constructor of Celery class. If not supplied, it triggers an auto-detection system which first looks for the PECL extension, then checks if PHP-amqplib has been installed by composer, and gives up if neither of these is present. You can force the queue type by passing a string - currently 'pecl' and 'php-amqplib' are accepted.

The new PHP-AMQPLib backend passed most unit tests, but the ones checking how it behaves on long running tasks. The problem is that AMQPLib doesn't allow specifying an arbitrary interval when waiting for a new message, it's always an integer number of seconds. For this reason some tests don't behave as expected, but it shouldn't affect real-life code other than that "asynchronous" checks for result take a whole second. I've prepared a pull request to address this problem, but the developer didn't decide to pull it in yet.

If you want to try it out, pull the code from its branch on github. It will be available in the main branch once it's well tested and the php-amqplib pull request goes through (so vote for it if you want to use PECL-less Celery!).

Celery-PHP updated, works with Celery 3.0

Updated Celery binding for PHP is available from Github.

I've ran unit tests with latest Celery and RabbitMQ from Debian Squeeze. They went well without any change in code logic.

I've came across some problems during setup of testing environment, mainly during installation of php-pecl-amqp. The install_amqp.sh script is now updated with current librabbitmq-c URL and other minor changes to build properly on Squeeze. I've also updated documentation about testing and updated celeryconfig.py in test scenario to use Celery 3.0-style BROKER_URL variable instead of BROKER_HOST etc. used in Celery 2.x.

Software versions used:

• Celery 3.0.19
• RabbitMQ 2.8.4
• PHP 5.4.4
• PHP-AMQP 1.0.10
• librabbitmq 0.3.0

Varnish-Joomla patch version 1.2.2 released

A new version of Varnish-Joomla patch is available. The most interesting news is Joomla! 3.1 support.

Other than that, patches for Joomla! 2.5 and 3.0 have been updated to the latest version of their respective families (2.5.11 and 3.0.4).

A bug pointed by Yves Lavoie has been fixed too. It could make the patched system error out in some rare circumstances.

If you're an existing customer, contact us to get the latest package.

If you're interested in purchasing the patch, please head to Order section of the product page: make your Joomla! faster

MySQL administration tool

Administrating a server set up by someone else can be a real pain. One of the problems I have regularly encountered on clients' systems has been database administration. While you can perform some tasks from command line, I really can't imagine working without a tool like PHPMyAdmin.

PHPMyAdmin, however, has a number of problems. The main one for me is its installation process which is much harder than necessary and involves editing configuration files. Consulting time spent installing PMA vs doing actual work is rather hard to explain to customers. So I started looking for alternatives.

The tool I've found performed beyond my wildest expectations. It consists of a single file (which is, by the way, interesting - they "compile" a nice, readable PHP source tree into a single minified PHP file). Not only it supports MySQL, but works with Postgres, SQLite, MS SQL and Oracle. It has a pretty, minimalistic interface. It's fast - not only it executes fast on the server, but it generates small pages so working on a slow network isn't a problem. And - best of all - the one-step installation process consists of uploading a single PHP file to the server.

Adminer, the tool I'm writing about, supports everything I care for. Some things could be done better or easier, but it's a really viable PHPMyAdmin alternative - I never found myself in a position where I couldn't do something and had to install PMA.

As usual, if you need help making your website faster, be sure to contact us.

Rails + supervisord (named sockets)

A recent upgrade of our project management software, Chili Project, was a good opportunity to make it run like other services - ran from Supervisord, connected via FastCGI. Unfortunately, Rails doesn't make it easy to listen for FCGI connections on a named socket.

After some research, I've found a way to run Chilli Project (and, in fact, any rails app) this way. I've put this little script into scripts/ subdirectory of the rails app:

Also, for completeness: contents of /etc/supervisor/conf.d/redmine.conf

Need help hosting Rails?

Making Yootheme Widgetkit load faster

While optimizing a customer's web site today I noticed that resources related to the (otherwise excellent) Yootheme Widgetkit were not cached by Varnish or by clients' browsers. This had a measurable impact on page loading time - it was about 5 scripts fetched on every page load, over and over again.

This happened because on every page load, a different request has been made. For example the media player URL was /media/widgetkit/widgets/mediaplayer/mediaelement/mediaelement-and-player.js?_=1355230620557, then /media/widgetkit/widgets/mediaplayer/mediaelement/mediaelement-and-player.js?_=1355230620586 and so on. Clearly something has been appending a time stamp as a query parameter.

After some investigation, I've found that jQuery was guilty. Its default behavior is to append the time stamp to prevent caching. Fortunately, this behavior is configurable. After adding the following line of code to the beginning of media/widgetkit/js/jquery.plugins.js, the query parameter was gone and the 5 scripts were cached once and for all:

jQuery.ajaxSetup({cache: true});

Yootheme support has been notified, hopefully they will include it or make it configurable in the next release.

The cost of poor web performance

Infographic (C)Smartbear.

Varnish integration for Joomla! 3.0

A version of our Varnish-Joomla integration for Joomla! 3.0 is ready. It doesn't bring any significant updates to older Joomla! versions. The upgrade is free for customers who ordered an older version of this patch.

Varnish is a website accelerator which can help you handle hundreds of requests per second on your Joomla! site.

Introducing TurnKey templates

Turn Key Linux is an open source project providing tens of virtual server templates. We're proud to introduce the ability to order VPS servers with these templates pre-installed directly from our web site: TurnKey based VPS servers.

Cron log in Debian 6

By default, Debian only sends results of cron jobs by email. In some cases, however (no direct internet connection), you want them logged in a regular log file in /var/log. In Debian Squeeze it's not the default but it's very easy to change it. Just edit /etc/rsyslog.conf and uncomment the following line:

cron.*                          /var/log/cron.log

Then, restart rsyslogd by executing service rsyslog restart. Viola!

Amazon Glacier

Amazon has released a new tool: Glacier. Similarly to S3, it is storage, but Glacier is a different beast. It's for keeping long term backups which you will hardly ever need, but which you need to store for one reason or another. The storage fee is just 1 US cent per gigabyte per month. The catch is, however, getting back your data. Retrieval of storage is free if you're only grabbing 5 percent of your data per month.

vztop in vzprocps: fixed

There has been a frequently encountered problem with vztop on OpenVZ making the tool totally useless - it worked perfectly except for it didn't show virtual machine IDs for processes. Instead of actual numerical ID, it showed "N/A" in the VEID column.

So I spent an hour or so debugging, and it turned out that vztop, when requested to show the VEID column, didn't set a flag which enabled collection of VEIDs and other data. While diving into vzprocps source, I also made vztop show VEID by default. The result is this simple patch against vzprocps 2.0.11-2.

Apply with patch -p1 <vztop_veid.txt

If you need to debug a problem with some piece of software, I'm for hire at Massive Scale. I have never seen vzprocps source code before and still the solution was available in just 1 hour of work. That's $70. Would your $15/h programmer solve the problem as efficiently?

Varnish-Joomla patch upgraded to 1.2.1

Varnish integration for Joomla! CMS is our solution for websites based on Joomla which have many page visits by anonymous users. It allows owners to serve even hundreds times more traffic without investing in hardware. Should I mention that it makes Joomla! fast, too?

One of our customers, Free Speaker Plans, was experiencing some issues with the patch he purchased from us. After a quick debugging session, it looked like forum software was not compatible with the patch. So I've modified the patch and now the forum component may run unmodified. If you've purchased an older version and have trouble running Kunena Forum, upgrade now!

At this point, I'd like to thank the webmaster of Free Speaker Plans for patiently testing my subsequent attempts at fixing the problem and providing rich feedback. Thank you!

Technical list of changes:

• Support for Joomla < 2.5.4 has been abandoned


• Fixed a bug introduced in 1.1, causing that pages calling Jerror::raise() raised a Fatal Error


• JFactory is now creating a guest JUser object if anonymous session detected (fixes Kunena and possibly other social software)


• There is no upgrade patch from 1.2 because of numerous Joomla core changes

New server

There was a disk failure on a server serving our customers' virtual private servers. Its disks were not hot swappable, so in an effort to prevent downtime, we're moving to a new server instead of repairing the current one. This was a good opportunity to upgrade the hardware, too. The gigabit network connection is pretty impressive, it downloaded a Linode test file in 1.5 seconds.

Personal server - how to save on memory?

This blog is usually about scaling systems to hundreds of thousands of users, but let's make an exception this time. I've been setting up a virtual server for my private use last few days, and I'd like to share my experience on saving the most precious resource when it comes to VPSes - RAM. There are many different virtual server providers with many different pricing options, but one thing they have in common is that the pricing is mainly dependent on RAM.

Let's make a quick list of demands for a private server. We'd like it to be as cheap as possible. Home budget is important, after all. We won't serve much traffic or do heavy computation, we also don't need to care about speed, with one user at a time the system will be fast enough regardless of the setup. In my case, I wanted to run OwnCloud 3 for syncing files, calendars and contacts with my phone. I'll also need an FTP server for easy sharing photos from my smartphone (using FTP is a bad idea in general, but I can't find anything like URLy with SFTP support).

A traditional approach used in large systems would be to install a full-featured web server with PHP running via FPM, regular FastCGI or mod_php, and an FTP daemon like proftpd or vsftpd. The processes would run in the background and wait for connections. This is very good for performance, but takes a lot of RAM.

So I took the old-school path. I'm running PHP code via CGI, and FTP daemon is running behind inetd. Let's expand this topic.

inetd is called an internet super-server. It's a tiny daemon which listens for connections on different ports and, when it accepts a connection, starts a program to handle this connection. So, in our case, it listens on port 21 (FTP), starts a FTP daemon when a connection comes in, and kills it immediately afterwards. inetd itself uses less than 2MB of RAM, less than any FTP program I'm fond of, so the extra memory is used only when my phone is actually uploading something.

CGI is a similar idea for processing scripts in a web server. When lighttpd needs to process some PHP, it sets some environment variables reflecting script name, HTTP headers and so on, and starts a php-cgi process. The interpreter processes script and exits. The RAM is free again. Of course, starting a PHP interpreter every time is slow and takes resources, but on an almost idle system it takes fraction of a second. It may not be acceptable on a massive scale, but is good enough for personal use.

What do we gain by using inetd and cgi? It's highly unlikely that I upload a file and use ownCloud from my computer at the same time, so I can estimate that the maximum memory usage is max(sizeof(php), sizeof(ftpd)) as opposed to sizeof(php) + sizeof(ftpd) in the case when they're running all the time. Even if this assumption isn't true and it happens that I'm using FTP while browsing ownCloud, nothing bad will happen - I will see an error 500 page, hit Refresh and live on.

ownCloud also needs a database. A standard database for PHP applications is MySQL. It runs as a regular server, so it takes memory all the time too. Fortunately, ownCloud also supports SQLite databases. SQLite is nowhere as robust or scalable as MySQL, in fact it doesn't even allow concurrent access to database. But it's enough for a single user, and even good enough for several users. SQLite needs no daemon running in the background enforcing ACID and taking up RAM - it just works as a PHP extension, a library which is loaded into PHP interpreter when it starts up.

Let's talk about implementation. Running PHP in CGI mode is easy with lighttpd, you just need to execute lighty-enable-mod cgi, edit /etc/lighttpd/conf-enabled/10-cgi.conf and uncomment the cgi.assign section in the bottom. You may also need to apt-get install php5-cgi.

Setting up inetd and ftpd is even easier. apt-get install openbsd-inetd ftpd leaves you with a running FTP server for all your users defined in /etc/passwd. You will also want to edit /etc/ftpchroot and /etc/ftpusers files. man ftpchroot and man ftpusers should explain everything.

As you see, using the right tool for the right job is beneficial - in enterprise I wouldn't dare using CGI or inetd as they would quickly make the server unresponsive, but for personal use it's good to step back and use an old school solution.

If you need a server set up for your personal needs, order one or talk to us about your needs. We're passionate about computing in general, not just at a massive scale.

Varnish/Joomla patch 1.2 in testing

We're testing the new version of Joomla support for Varnish HTTP accelerator. Version 1.2 is available for Joomla! 1.5 and 2.5. Take a look at the changelog.

1.1

Don't cache a page with error box - when JError::raise() is called or JDocumentRendererMessage is used, a header which forbids caching is being sent

1.2

Prevent displaying cached pages directly after logging in.

After successful login the user is usually redirected to referring page. In many cases the page is cached by browser, and contains a login form, thus causing confusion to the user. The problem is solved by adding a random GET variable called _rnd.

How to disable annoying ipython exit prompt

ipython is an interactive Python shell which is generally great with its history and autocompletion, but has one annoying problem. Whenever you hit Ctrl-D to exit from ipython to bash, it asks you this question:

 
Do you really want to exit ([y]/n)?

Perhaps this prompt would be useful if I were using it to control an atomic power plant in emergency mode, but for my humble needs it's just annoying. It's easy to disable it - create a file called ~/.ipython/profile_default/ipython_config.py with the following content.

c = get_config()
c.InteractiveShell.confirm_exit = False

If you need a hand with Python, see what we can do for you.

Varnish/Joomla patch update

Our patch for making Joomla work with Varnish Cache has been upgraded today. It contained a rarely occuring bug causing pages with error boxes to be cached under certain conditions.

Because of this, regular visitors may see errors like the on below, even though they didn't try to log in:

The upgrade is free to all our customers who have purchased the patch.

Varnish and Joomla - how does it work?

Our most popular product, Varnish-Joomla integration, makes your website fly fast. But how does it achieve its goal?

Most visitors to virtually any website are anonymous visitors. These people don't log in, don't interact, they just view your content. In an e-commerce store they may be visitors who didn't buy anything. On a forum website they may be people who have found some thread on Google, read it and forget about it. On a newspaper site, readers who don't leave comments and don't have an account are considered anonymous visitors.

These visitors generate the most traffic. But they are also the easiest to handle, because all they want to do is read your content - so why serve their pages using full Joomla - PHP - MySQL - Apache stack, if they don't really utilize any of these in full?

We took advantage of that fact and we serve content to such users before they reach Joomla. We put a piece of software in front of Joomla which serves content to anonymous visitors from a very fast cache. If it encounters a visitor who's logged in, it forwards his request to Joomla so he gets a full experience.

If you don't believe - check the numbers. Varnish makes Joomla fast.

HTTP 417 errors in lighttpd

One of the most annoying parts of Lighttpd is that it sometimes fails with error 417 when you upload a form or use an API with libcurl. The proper, HTTP/1.1 compiliant solution will be available in version 1.5, which is in development for years now, but since 1.4.21 they have implemented a solution to behave like a HTTP/1.0 server, which is just what 99.9% of users need.

To fix 417 errors in lighttpd, simply add this to your lighttpd.conf:
server.reject-expect-100-with-417 = "disable"
and restart your lighttpd server. Viola!

Christmas in e-commerce

Christmas can be hard on servers

If you're running an e-commerce website, you already know the pattern: during Christmas and Valentine's Day your revenue increases ten-fold, but so does the server load. Pages load slowly, customers are annoyed and so is your staff. Can this be avoided?

Handling more users doesn't necessarily mean getting more expensive hardware and moving from one server to another. More often than not, you can increase the efficiency of your existing infrastructure. Sometimes you could even lower the total cost of operation just by optimizing resource usage. This is equally true for servers and for other business assets.

We work with websites handling lots of traffic all the time. We know how to optimize them for speed, performance and scalability. We know how to test how much they can handle. If that's needed, we also know how to design a larger infrastructure tailored to your needs.

Google research shows that page load time increased by half a second causes a 20% drop in traffic. For e-commerce, more traffic equals more sales. Do we really need to elaborate?

Read more to learn what we can do to help you.

Joomla website made fast - case study

I've just finished optimizing the infrastructure behind WomensForum.com. They have used our Joomla-Varnish package and a few hours of my website optimization consulting time. The amount of money they spent on optimization is comparable to monthly hosting fee for a big website.

While the website was usually fast before optimization, the server couldn't handle traffic spikes. At one point, there were more than 1500 requests per second - that's a lot for Joomla, but not for Varnish. Page load time was consistently low even during traffic spikes. Additionally, we have used the failover capability of Varnish to be less reliable on backend server failures. Speed up your website too.

Virtual desktop

What would you do with a virtual Ubuntu desktop? We have a few ideas. You could view facebook while you're at work, no one will know. You could work on a document with other people. You could download torrents or rapidshare/megaupload files while your computer is off. Your imagination is the limit, but check out our remote VNC desktop page to see a few ideas.

Gitlab, the private Github clone

We programmers know and love Github, the source code hosting website that's free for open source projects. But we often like to keep our projects private before releasing them publicly, and that's one thing Github is not good at. Sure, you can get the personal plan to have 5 private repositories, but is that really enough for all your personal mini-projects?

Meet Gitlab, the open source Github clone that you can host on your server. The number of repositories and collaborators is limited just by your server's capabilities. But you need to set up the server, and it takes a few hours unless you're proficient with Ruby.

We're proud to introduce the Git Server Appliance, featuring gitlab as web GUI and gitosis as accounts manager. All this starting at the price of Github personal plan!

There's no set up, you're provided with administrator login and password to your newly provisioned git server appliance. All you need to do is log in and start working.

Lamson, the Python SMTP server

Do the words "sendmail", "postfix" and "qmail" sound scary to you? You're not alone, and some developers were so fed up with 1970-s style software traditionally used to handle email routing, that they have written an email server in Python.

The front page of Lamson features some simple examples of how routing rules may be created - it's so logical, it's hard to believe it's handling email.

Don't know where to start? We're here to support developers.

Spanish website

We are proud to announce MassiveScale website in Spanish. We're also introducing live chat with sales in both English and Spanish.

Secure Joomla 1.0 hosting

At MassiveScale, we are starting to offer VPS hosting. While we don't have any generally available offer yet, we have started marketing a niche service. Joomla 1.0 websites hosting.

"It's as if you offered Windows 3.11 support", a friend of mine said. Well, yes and no. The newest version of Joomla CMS is 1.7, and 1.0 has been unsupported since 2008. However, there are many websites made on 1.0 that will not upgrade to 1.7 simply because it's too expensive. If someone ordered a page made with Joomla and a slightly modified off-the-shelf template, it's not because he has a large budget. Now, try explaining that person that he needs to spend the money twice or there will be countless break-ins into his page.

That's why we decided to offer a service to help these people out. Most of these websites are now in maintenance mode, only content is being added/edited and no major redesigns are taking place. We just lock all the files and, optionally, parts of the database, and that's the secret sauce. It makes Joomla 1.0 hosting pretty secure.

Varnish and FLV streaming

When doing usual website speed optimization to a website that has lots of videos, I have encountered a problem. Videos started breaking for no apparent reason, they loaded very slowly and in general were unusable.

It turned out it was because of two things. One was that Apache has been set up to compress everything except images with mod_deflate - this does not play well with streaming. Another problem was that Varnish running in front of the web server wasn't set up to do streaming, so it first fetched whole video to its cache and then sent it to the browser. It was slow.

The solution was to upgrade Varnish to 3.0.1 and add the following code to vcl_fetch:

if(beresp.http.Content-Type && beresp.http.Content-Type ~ "video") {
        set beresp.do_stream = true;
}

Optimizing website performance is not an easy job and has tons of catches like this.Order optimization now!

Celery task queue with PHP

We have released a Celery client for PHP some time ago. What does it do and why is it useful?

Celery is a piece of software that helps you easily run time-consuming tasks. If you have ever used system() or exec() to create a big ZIP file or encode a video, you probably already know what the problem is. If you didn't - well, time-consuming tasks executed from within a PHP script (or any other web application) usually generate many problems, unresponsive user interface and timeouts being two major ones. There are many ways to overcome these issues, but face it - spending many hours and hundreds of lines of code just to create a ZIP file isn't especially effective.

So how does a basic Celery application to create a ZIP file look like? Let's see:

#!/usr/bin/python
from celery.task import task
from subprocess import Popen, PIPE, STDOUT

@task
def create_zip(zip_path, files_path):
    command = ("zip", zip_path, files_path)
    return Popen(command, stdout=PIPE, stderr=STDOUT).communicate()[0]


The above code calls the command-line zip command with two arguments: path to a zip file and path of the files to be archived. "def" is Python for "function". (this is just an example, in real world you will probably want to validate paths and replace zip command with Python's zipfile module)

Now you will probably want to call that code from your PHP application. First schedule the task for execution:

$c = new Celery('localhost', 'myuser', 'mypass', 'myvhost');
$_SESSION['zip_result'] = $c->PostTask('tasks.create_zip', array('/home/user/file.zip', '/home/user/images/'));

Then you will want to somehow asynchronously display the result to user. You can use AJAX or some other technique to display the result of this script every second:

$result = $_SESSION['zip_result'];
if(!$result->isReady())
{
    echo 'Please wait...';
}
else
{
    echo '<a href="/file.zip">Ready!</a>';
}


That's it, as easy as this. Check out the Celery-PHP documentation and try it out.

Also, the examples above are oversimplified - make sure you validate your input data, handle errors and don't unnecessarily poll the server.

RabbitMQ 2.6.0

It looks like new RabbitMQ is just behind the corner.