MySQL administration tool

Administrating a server set up by someone else can be a real pain. One of the problems I have regularly encountered on clients' systems has been database administration. While you can perform some tasks from command line, I really can't imagine working without a tool like PHPMyAdmin.

PHPMyAdmin, however, has a number of problems. The main one for me is its installation process which is much harder than necessary and involves editing configuration files. Consulting time spent installing PMA vs doing actual work is rather hard to explain to customers. So I started looking for alternatives.

The tool I've found performed beyond my wildest expectations. It consists of a single file (which is, by the way, interesting - they "compile" a nice, readable PHP source tree into a single minified PHP file). Not only it supports MySQL, but works with Postgres, SQLite, MS SQL and Oracle. It has a pretty, minimalistic interface. It's fast - not only it executes fast on the server, but it generates small pages so working on a slow network isn't a problem. And - best of all - the one-step installation process consists of uploading a single PHP file to the server.

Adminer, the tool I'm writing about, supports everything I care for. Some things could be done better or easier, but it's a really viable PHPMyAdmin alternative - I never found myself in a position where I couldn't do something and had to install PMA.

As usual, if you need help making your website faster, be sure to contact us.

Rails + supervisord (named sockets)

A recent upgrade of our project management software, Chili Project, was a good opportunity to make it run like other services - ran from Supervisord, connected via FastCGI. Unfortunately, Rails doesn't make it easy to listen for FCGI connections on a named socket.

After some research, I've found a way to run Chilli Project (and, in fact, any rails app) this way. I've put this little script into scripts/ subdirectory of the rails app:

Also, for completeness: contents of /etc/supervisor/conf.d/redmine.conf

Need help hosting Rails?

Making Yootheme Widgetkit load faster

While optimizing a customer's web site today I noticed that resources related to the (otherwise excellent) Yootheme Widgetkit were not cached by Varnish or by clients' browsers. This had a measurable impact on page loading time - it was about 5 scripts fetched on every page load, over and over again.

This happened because on every page load, a different request has been made. For example the media player URL was /media/widgetkit/widgets/mediaplayer/mediaelement/mediaelement-and-player.js?_=1355230620557, then /media/widgetkit/widgets/mediaplayer/mediaelement/mediaelement-and-player.js?_=1355230620586 and so on. Clearly something has been appending a time stamp as a query parameter.

After some investigation, I've found that jQuery was guilty. Its default behavior is to append the time stamp to prevent caching. Fortunately, this behavior is configurable. After adding the following line of code to the beginning of media/widgetkit/js/jquery.plugins.js, the query parameter was gone and the 5 scripts were cached once and for all:

jQuery.ajaxSetup({cache: true});

Yootheme support has been notified, hopefully they will include it or make it configurable in the next release.

The cost of poor web performance

Infographic (C)Smartbear.

Varnish integration for Joomla! 3.0

A version of our Varnish-Joomla integration for Joomla! 3.0 is ready. It doesn't bring any significant updates to older Joomla! versions. The upgrade is free for customers who ordered an older version of this patch.

Varnish is a website accelerator which can help you handle hundreds of requests per second on your Joomla! site.

Introducing TurnKey templates

Turn Key Linux is an open source project providing tens of virtual server templates. We're proud to introduce the ability to order VPS servers with these templates pre-installed directly from our web site: TurnKey based VPS servers.

Cron log in Debian 6

By default, Debian only sends results of cron jobs by email. In some cases, however (no direct internet connection), you want them logged in a regular log file in /var/log. In Debian Squeeze it's not the default but it's very easy to change it. Just edit /etc/rsyslog.conf and uncomment the following line:

cron.*                          /var/log/cron.log

Then, restart rsyslogd by executing service rsyslog restart. Viola!

Amazon Glacier

Amazon has released a new tool: Glacier. Similarly to S3, it is storage, but Glacier is a different beast. It's for keeping long term backups which you will hardly ever need, but which you need to store for one reason or another. The storage fee is just 1 US cent per gigabyte per month. The catch is, however, getting back your data. Retrieval of storage is free if you're only grabbing 5 percent of your data per month.

vztop in vzprocps: fixed

There has been a frequently encountered problem with vztop on OpenVZ making the tool totally useless - it worked perfectly except for it didn't show virtual machine IDs for processes. Instead of actual numerical ID, it showed "N/A" in the VEID column.

So I spent an hour or so debugging, and it turned out that vztop, when requested to show the VEID column, didn't set a flag which enabled collection of VEIDs and other data. While diving into vzprocps source, I also made vztop show VEID by default. The result is this simple patch against vzprocps 2.0.11-2.

Apply with patch -p1 <vztop_veid.txt

If you need to debug a problem with some piece of software, I'm for hire at Massive Scale. I have never seen vzprocps source code before and still the solution was available in just 1 hour of work. That's $70. Would your $15/h programmer solve the problem as efficiently?

Varnish-Joomla patch upgraded to 1.2.1

Varnish integration for Joomla! CMS is our solution for websites based on Joomla which have many page visits by anonymous users. It allows owners to serve even hundreds times more traffic without investing in hardware. Should I mention that it makes Joomla! fast, too?

One of our customers, Free Speaker Plans, was experiencing some issues with the patch he purchased from us. After a quick debugging session, it looked like forum software was not compatible with the patch. So I've modified the patch and now the forum component may run unmodified. If you've purchased an older version and have trouble running Kunena Forum, upgrade now!

At this point, I'd like to thank the webmaster of Free Speaker Plans for patiently testing my subsequent attempts at fixing the problem and providing rich feedback. Thank you!

Technical list of changes:

• Support for Joomla < 2.5.4 has been abandoned


• Fixed a bug introduced in 1.1, causing that pages calling Jerror::raise() raised a Fatal Error


• JFactory is now creating a guest JUser object if anonymous session detected (fixes Kunena and possibly other social software)


• There is no upgrade patch from 1.2 because of numerous Joomla core changes

New server

There was a disk failure on a server serving our customers' virtual private servers. Its disks were not hot swappable, so in an effort to prevent downtime, we're moving to a new server instead of repairing the current one. This was a good opportunity to upgrade the hardware, too. The gigabit network connection is pretty impressive, it downloaded a Linode test file in 1.5 seconds.

Personal server - how to save on memory?

This blog is usually about scaling systems to hundreds of thousands of users, but let's make an exception this time. I've been setting up a virtual server for my private use last few days, and I'd like to share my experience on saving the most precious resource when it comes to VPSes - RAM. There are many different virtual server providers with many different pricing options, but one thing they have in common is that the pricing is mainly dependent on RAM.

Let's make a quick list of demands for a private server. We'd like it to be as cheap as possible. Home budget is important, after all. We won't serve much traffic or do heavy computation, we also don't need to care about speed, with one user at a time the system will be fast enough regardless of the setup. In my case, I wanted to run OwnCloud 3 for syncing files, calendars and contacts with my phone. I'll also need an FTP server for easy sharing photos from my smartphone (using FTP is a bad idea in general, but I can't find anything like URLy with SFTP support).

A traditional approach used in large systems would be to install a full-featured web server with PHP running via FPM, regular FastCGI or mod_php, and an FTP daemon like proftpd or vsftpd. The processes would run in the background and wait for connections. This is very good for performance, but takes a lot of RAM.

So I took the old-school path. I'm running PHP code via CGI, and FTP daemon is running behind inetd. Let's expand this topic.

inetd is called an internet super-server. It's a tiny daemon which listens for connections on different ports and, when it accepts a connection, starts a program to handle this connection. So, in our case, it listens on port 21 (FTP), starts a FTP daemon when a connection comes in, and kills it immediately afterwards. inetd itself uses less than 2MB of RAM, less than any FTP program I'm fond of, so the extra memory is used only when my phone is actually uploading something.

CGI is a similar idea for processing scripts in a web server. When lighttpd needs to process some PHP, it sets some environment variables reflecting script name, HTTP headers and so on, and starts a php-cgi process. The interpreter processes script and exits. The RAM is free again. Of course, starting a PHP interpreter every time is slow and takes resources, but on an almost idle system it takes fraction of a second. It may not be acceptable on a massive scale, but is good enough for personal use.

What do we gain by using inetd and cgi? It's highly unlikely that I upload a file and use ownCloud from my computer at the same time, so I can estimate that the maximum memory usage is max(sizeof(php), sizeof(ftpd)) as opposed to sizeof(php) + sizeof(ftpd) in the case when they're running all the time. Even if this assumption isn't true and it happens that I'm using FTP while browsing ownCloud, nothing bad will happen - I will see an error 500 page, hit Refresh and live on.

ownCloud also needs a database. A standard database for PHP applications is MySQL. It runs as a regular server, so it takes memory all the time too. Fortunately, ownCloud also supports SQLite databases. SQLite is nowhere as robust or scalable as MySQL, in fact it doesn't even allow concurrent access to database. But it's enough for a single user, and even good enough for several users. SQLite needs no daemon running in the background enforcing ACID and taking up RAM - it just works as a PHP extension, a library which is loaded into PHP interpreter when it starts up.

Let's talk about implementation. Running PHP in CGI mode is easy with lighttpd, you just need to execute lighty-enable-mod cgi, edit /etc/lighttpd/conf-enabled/10-cgi.conf and uncomment the cgi.assign section in the bottom. You may also need to apt-get install php5-cgi.

Setting up inetd and ftpd is even easier. apt-get install openbsd-inetd ftpd leaves you with a running FTP server for all your users defined in /etc/passwd. You will also want to edit /etc/ftpchroot and /etc/ftpusers files. man ftpchroot and man ftpusers should explain everything.

As you see, using the right tool for the right job is beneficial - in enterprise I wouldn't dare using CGI or inetd as they would quickly make the server unresponsive, but for personal use it's good to step back and use an old school solution.

If you need a server set up for your personal needs, order one or talk to us about your needs. We're passionate about computing in general, not just at a massive scale.

Varnish/Joomla patch 1.2 in testing

We're testing the new version of Joomla support for Varnish HTTP accelerator. Version 1.2 is available for Joomla! 1.5 and 2.5. Take a look at the changelog.

1.1

Don't cache a page with error box - when JError::raise() is called or JDocumentRendererMessage is used, a header which forbids caching is being sent

1.2

Prevent displaying cached pages directly after logging in.

After successful login the user is usually redirected to referring page. In many cases the page is cached by browser, and contains a login form, thus causing confusion to the user. The problem is solved by adding a random GET variable called _rnd.

How to disable annoying ipython exit prompt

ipython is an interactive Python shell which is generally great with its history and autocompletion, but has one annoying problem. Whenever you hit Ctrl-D to exit from ipython to bash, it asks you this question:

 
Do you really want to exit ([y]/n)?

Perhaps this prompt would be useful if I were using it to control an atomic power plant in emergency mode, but for my humble needs it's just annoying. It's easy to disable it - create a file called ~/.ipython/profile_default/ipython_config.py with the following content.

c = get_config()
c.InteractiveShell.confirm_exit = False

If you need a hand with Python, see what we can do for you.

Varnish/Joomla patch update

Our patch for making Joomla work with Varnish Cache has been upgraded today. It contained a rarely occuring bug causing pages with error boxes to be cached under certain conditions.

Because of this, regular visitors may see errors like the on below, even though they didn't try to log in:

The upgrade is free to all our customers who have purchased the patch.

Varnish and Joomla - how does it work?

Our most popular product, Varnish-Joomla integration, makes your website fly fast. But how does it achieve its goal?

Most visitors to virtually any website are anonymous visitors. These people don't log in, don't interact, they just view your content. In an e-commerce store they may be visitors who didn't buy anything. On a forum website they may be people who have found some thread on Google, read it and forget about it. On a newspaper site, readers who don't leave comments and don't have an account are considered anonymous visitors.

These visitors generate the most traffic. But they are also the easiest to handle, because all they want to do is read your content - so why serve their pages using full Joomla - PHP - MySQL - Apache stack, if they don't really utilize any of these in full?

We took advantage of that fact and we serve content to such users before they reach Joomla. We put a piece of software in front of Joomla which serves content to anonymous visitors from a very fast cache. If it encounters a visitor who's logged in, it forwards his request to Joomla so he gets a full experience.

If you don't believe - check the numbers. Varnish makes Joomla fast.

HTTP 417 errors in lighttpd

One of the most annoying parts of Lighttpd is that it sometimes fails with error 417 when you upload a form or use an API with libcurl. The proper, HTTP/1.1 compiliant solution will be available in version 1.5, which is in development for years now, but since 1.4.21 they have implemented a solution to behave like a HTTP/1.0 server, which is just what 99.9% of users need.

To fix 417 errors in lighttpd, simply add this to your lighttpd.conf:
server.reject-expect-100-with-417 = "disable"
and restart your lighttpd server. Viola!

Christmas in e-commerce

Christmas can be hard on servers

If you're running an e-commerce website, you already know the pattern: during Christmas and Valentine's Day your revenue increases ten-fold, but so does the server load. Pages load slowly, customers are annoyed and so is your staff. Can this be avoided?

Handling more users doesn't necessarily mean getting more expensive hardware and moving from one server to another. More often than not, you can increase the efficiency of your existing infrastructure. Sometimes you could even lower the total cost of operation just by optimizing resource usage. This is equally true for servers and for other business assets.

We work with websites handling lots of traffic all the time. We know how to optimize them for speed, performance and scalability. We know how to test how much they can handle. If that's needed, we also know how to design a larger infrastructure tailored to your needs.

Google research shows that page load time increased by half a second causes a 20% drop in traffic. For e-commerce, more traffic equals more sales. Do we really need to elaborate?

Read more to learn what we can do to help you.

Joomla website made fast - case study

I've just finished optimizing the infrastructure behind WomensForum.com. They have used our Joomla-Varnish package and a few hours of my website optimization consulting time. The amount of money they spent on optimization is comparable to monthly hosting fee for a big website.

While the website was usually fast before optimization, the server couldn't handle traffic spikes. At one point, there were more than 1500 requests per second - that's a lot for Joomla, but not for Varnish. Page load time was consistently low even during traffic spikes. Additionally, we have used the failover capability of Varnish to be less reliable on backend server failures. Speed up your website too.

Virtual desktop

What would you do with a virtual Ubuntu desktop? We have a few ideas. You could view facebook while you're at work, no one will know. You could work on a document with other people. You could download torrents or rapidshare/megaupload files while your computer is off. Your imagination is the limit, but check out our remote VNC desktop page to see a few ideas.

Gitlab, the private Github clone

We programmers know and love Github, the source code hosting website that's free for open source projects. But we often like to keep our projects private before releasing them publicly, and that's one thing Github is not good at. Sure, you can get the personal plan to have 5 private repositories, but is that really enough for all your personal mini-projects?

Meet Gitlab, the open source Github clone that you can host on your server. The number of repositories and collaborators is limited just by your server's capabilities. But you need to set up the server, and it takes a few hours unless you're proficient with Ruby.

We're proud to introduce the Git Server Appliance, featuring gitlab as web GUI and gitosis as accounts manager. All this starting at the price of Github personal plan!

There's no set up, you're provided with administrator login and password to your newly provisioned git server appliance. All you need to do is log in and start working.

Lamson, the Python SMTP server

Do the words "sendmail", "postfix" and "qmail" sound scary to you? You're not alone, and some developers were so fed up with 1970-s style software traditionally used to handle email routing, that they have written an email server in Python.

The front page of Lamson features some simple examples of how routing rules may be created - it's so logical, it's hard to believe it's handling email.

Don't know where to start? We're here to support developers.

Spanish website

We are proud to announce MassiveScale website in Spanish. We're also introducing live chat with sales in both English and Spanish.

Secure Joomla 1.0 hosting

At MassiveScale, we are starting to offer VPS hosting. While we don't have any generally available offer yet, we have started marketing a niche service. Joomla 1.0 websites hosting.

"It's as if you offered Windows 3.11 support", a friend of mine said. Well, yes and no. The newest version of Joomla CMS is 1.7, and 1.0 has been unsupported since 2008. However, there are many websites made on 1.0 that will not upgrade to 1.7 simply because it's too expensive. If someone ordered a page made with Joomla and a slightly modified off-the-shelf template, it's not because he has a large budget. Now, try explaining that person that he needs to spend the money twice or there will be countless break-ins into his page.

That's why we decided to offer a service to help these people out. Most of these websites are now in maintenance mode, only content is being added/edited and no major redesigns are taking place. We just lock all the files and, optionally, parts of the database, and that's the secret sauce. It makes Joomla 1.0 hosting pretty secure.

Varnish and FLV streaming

When doing usual website speed optimization to a website that has lots of videos, I have encountered a problem. Videos started breaking for no apparent reason, they loaded very slowly and in general were unusable.

It turned out it was because of two things. One was that Apache has been set up to compress everything except images with mod_deflate - this does not play well with streaming. Another problem was that Varnish running in front of the web server wasn't set up to do streaming, so it first fetched whole video to its cache and then sent it to the browser. It was slow.

The solution was to upgrade Varnish to 3.0.1 and add the following code to vcl_fetch:

if(beresp.http.Content-Type && beresp.http.Content-Type ~ "video") {
        set beresp.do_stream = true;
}

Optimizing website performance is not an easy job and has tons of catches like this.Order optimization now!

Celery task queue with PHP

We have released a Celery client for PHP some time ago. What does it do and why is it useful?

Celery is a piece of software that helps you easily run time-consuming tasks. If you have ever used system() or exec() to create a big ZIP file or encode a video, you probably already know what the problem is. If you didn't - well, time-consuming tasks executed from within a PHP script (or any other web application) usually generate many problems, unresponsive user interface and timeouts being two major ones. There are many ways to overcome these issues, but face it - spending many hours and hundreds of lines of code just to create a ZIP file isn't especially effective.

So how does a basic Celery application to create a ZIP file look like? Let's see:

#!/usr/bin/python
from celery.task import task
from subprocess import Popen, PIPE, STDOUT

@task
def create_zip(zip_path, files_path):
    command = ("zip", zip_path, files_path)
    return Popen(command, stdout=PIPE, stderr=STDOUT).communicate()[0]


The above code calls the command-line zip command with two arguments: path to a zip file and path of the files to be archived. "def" is Python for "function". (this is just an example, in real world you will probably want to validate paths and replace zip command with Python's zipfile module)

Now you will probably want to call that code from your PHP application. First schedule the task for execution:

$c = new Celery('localhost', 'myuser', 'mypass', 'myvhost');
$_SESSION['zip_result'] = $c->PostTask('tasks.create_zip', array('/home/user/file.zip', '/home/user/images/'));

Then you will want to somehow asynchronously display the result to user. You can use AJAX or some other technique to display the result of this script every second:

$result = $_SESSION['zip_result'];
if(!$result->isReady())
{
    echo 'Please wait...';
}
else
{
    echo '<a href="/file.zip">Ready!</a>';
}


That's it, as easy as this. Check out the Celery-PHP documentation and try it out.

Also, the examples above are oversimplified - make sure you validate your input data, handle errors and don't unnecessarily poll the server.

RabbitMQ 2.6.0

It looks like new RabbitMQ is just behind the corner.