Sunday, June 23, 2013

PHP 5.5: Zend Optimiser+ OPcache vs XCache

PHP 5.5 has a new feature: built-in opcode cache! It's an alternative for extensions like XCache, APC or eAccelerator. Let's see how it performs compared to established solutions.

Test environment

Server: OpenVZ container running on 1 Xeon W3520 CPU core, high I/O priority (to ensure CPU-bound execution), 1 GB RAM. Requests originating from another VPS on that system.

Software: Debian 7.0 i386, nginx 1.2.1, PHP 5.5.0 from dotdeb, XCache SVN trunk from 2013-06-23 (r1269).

(This test may or may not reflect what is happening on dedicated servers. Unfortunately, we don't have a spare server-grade physical machine at the moment. That being said, the machine which held the VPS is under a small network and IO load.

PHP-FPM configuration

pm = dynamic
pm.max_children = 25
pm.start_servers = 4
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 5000

Zend Opcache has been configured as following:

root@opcode:/var/www# grep ^opcache /etc/php5/fpm/php.ini
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=4
opcache.max_accelerated_files=3907
opcache.fast_shutdown=1

XCache has been configured as following:

xcache.size = 128M
xcache.count = 3
xcache.cacher = On

Joomla

Joomla! 3.1.1. Installation was tricky, installation/application/framework.php had to be edited, this line in particular: ini_set('display_errors', true); - otherwise AJAX calls didn't succeed because JSON has been polluted with error messages.

The "Default English" sample data set has been installed. Joomla cache has been disabled because it's not what we want to test.

Test

The Apache Benchmark has been used to measure average response time and number of requests served per seconds. The command line was:

ab -n 10000 -c 20 

Ten thousands requests have been performed by 20 concurrent threads. This corresponds with the pm.max_children=25 setting in php.ini.

No cache

Performance is clearly CPU-bound. CPU usage is 100%, RAM utilization is about 880 MB out of 1GB. Disk I/O is negligible measured from both within the container (%wa 0) and the hypervisor (%wa 7).

Apache Benchmark output (Unfold)

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 176.31.28.48 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.2.1
Server Hostname:        176.31.28.48
Server Port:            80

Document Path:          /
Document Length:        8626 bytes

Concurrency Level:      20
Time taken for tests:   1743.994 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      89890000 bytes
HTML transferred:       86260000 bytes
Requests per second:    5.73 [#/sec] (mean)
Time per request:       3487.989 [ms] (mean)
Time per request:       174.399 [ms] (mean, across all concurrent requests)
Transfer rate:          50.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   656 3483 1020.6   3166    9587
Waiting:      403 2999 778.5   2812    8757
Total:        656 3483 1020.6   3166    9587

Percentage of the requests served within a certain time (ms)
  50%   3166
  66%   3528
  75%   3849
  80%   4091
  90%   4797
  95%   5540
  98%   6695
  99%   7420
 100%   9587 (longest request)

Zend OPcache enabled

It looks like the CPU hasn't been fully utilized with 20 concurrent requests. The usage fluctuated between 70% and 90%. I/O and memory usage were, as previously, within reasonable limits.

Apache Benchmark output (Unfold)

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 176.31.28.48 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.2.1
Server Hostname:        176.31.28.48
Server Port:            80

Document Path:          /
Document Length:        8626 bytes

Concurrency Level:      20
Time taken for tests:   658.408 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      89890000 bytes
HTML transferred:       86260000 bytes
Requests per second:    15.19 [#/sec] (mean)
Time per request:       1316.817 [ms] (mean)
Time per request:       65.841 [ms] (mean, across all concurrent requests)
Transfer rate:          133.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   421 1316 426.2   1214    5664
Waiting:      275 1053 355.8    986    4329
Total:        421 1316 426.2   1214    5664

Percentage of the requests served within a certain time (ms)
  50%   1214
  66%   1333
  75%   1448
  80%   1537
  90%   1826
  95%   2065
  98%   2484
  99%   2780
 100%   5664 (longest request)

XCache

Just like previously, the load has been CPU-bound even after enabling XCache.

Apache Benchmark output (Unfold)

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 176.31.28.48 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.2.1
Server Hostname:        176.31.28.48
Server Port:            80

Document Path:          /
Document Length:        8626 bytes

Concurrency Level:      20
Time taken for tests:   769.788 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      89890000 bytes
HTML transferred:       86260000 bytes
Requests per second:    12.99 [#/sec] (mean)
Time per request:       1539.576 [ms] (mean)
Time per request:       76.979 [ms] (mean, across all concurrent requests)
Transfer rate:          114.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   601 1538 431.1   1439    4401
Waiting:      393 1232 361.6   1178    4205
Total:        601 1539 431.1   1440    4401

Percentage of the requests served within a certain time (ms)
  50%   1440
  66%   1582
  75%   1696
  80%   1784
  90%   2079
  95%   2333
  98%   2847
  99%   3179
 100%   4401 (longest request)

Graphs

Time graph: less is better

Requests per second graph: more is better

Comment

Well, XCache is no more the best opcode cache around. We're looking forward to widespread adoption of PHP 5.5+. Unfortunately, there is currently no GUI for nicely analyzing cache usage, like XCache has. However, Zend exposes a new function, accelerator_get_status(), which could be used to create such a GUI - so certainly somebody will release relevant GUI in the next weeks.

EDIT: We didn't have to wait long - as a reader pointed out, there is a GUI for Zend Optimizer.

As usual, if you need help, we will make your website fast for you.

13 comments:

  1. "Unfortunately, there is currently no GUI for nicely analyzing cache usage, like XCache has."

    Yes there is, and actually looks quite nice: https://gist.github.com/ck-on/4959032

    ReplyDelete
  2. Hi,

    AFAIK XCache support for PHP 5.5 is still in development and will be fully supported when XCache 3.1 is released.

    So you should redo this test after XCache 3.1 was released. :-)

    ReplyDelete
  3. Well - I did what I could at the moment. I'm sure that a million other websites will publish comparisons once XCache 3.1 is released.

    ReplyDelete
  4. I wonder why did you did not set option "xcache.optimizer = Yes"? This gives significant performance boost...

    ReplyDelete
  5. Well, last time I've read the documentation it stated that this option doesn't do anything. Looks like it has changed now. Thanks for the tip!

    ReplyDelete
  6. FWIW I couldn't get a significant performance benefit using 5.5/zop+ over 5.3/apc - it's possible that 5.5 is just a lot slower than 5.3 - as yet I've not tried comparing the same PHP version on the 2 caches - but your efforts are appreciated!

    http://symcbean.blogspot.co.uk/search/label/Zend%20Optimizer

    ReplyDelete
  7. This is what I feel. It appears that 5.5 is just a lot slower than 5.3 - and that zop+ just compensate.
    More generally, with PHP it is like latter version are slower than previous.
    Have a look at graph presented here:
    http://3v4l.org/am3S3/perf
    worst: http://3v4l.org/Kb2F0/perf#tabs

    ReplyDelete
    Replies
    1. your second link tell the opposite

      memory wise that may be true, except for PHP7 that use less memory then previous version. that is acceptable, since new version give more feature. however the processing speed stay the same. see this: http://3v4l.org/medfO/perf#tabs

      Delete
  8. https://github.com/PeeHaa/OpCacheGUI OPCache GUI

    ReplyDelete
  9. Has anyone done any recent benchmarking? Most of these comparison tests I could find are outdated and on older versions. The latest versions of both have been optimized for PHP 5.5.x ? What about Apache Mod vs fastCGI php implementation? We need a new benchmark test people!!

    ReplyDelete
    Replies
    1. For mod_php vs FastCGI, the benchmarks for newer versions will be the same as for the old versions, I'm not aware of any technological advances in any of these technologies.

      Delete
  10. this analisys are obvously using mysql, i try with postgres and performance problems are very different... there's some problems in mysql and lokc tables when ther many requests per hosts... so for real performance dont use poor scalar dbms...

    ReplyDelete
    Replies
    1. It's 2017 and you're answering to a post from 2013. You could barely run Joomla CMS on Postgres in 2013, and you could forget about extensions. It was not a practical test to run.

      Delete