Showing posts with label debugging. Show all posts
Showing posts with label debugging. Show all posts

Monday, October 7, 2013

Thread creation error in Varnish

Today I installed Varnish on a pretty standard system (CentOS + WHM) and the service wouldn't start. The system log (/var/log/messages) contained the following:

Oct  7 04:15:46 server varnishd[20836]: Child (20846) Panic message: Assert error in WRK_BgThread(), cache_pool.c line 582:#012  Condition((pthread_create(thr, ((void *)0), wrk_bgthread, bt)) == 0) not true.#012errno = 11 (Resource temporarily unavailable)#012thread = (cache-main)#012ident = Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit,no_waiter#012

The settings for number of processes and file descriptors in /etc/security/limits.{conf,d} were correct and it still didn't start. The problem turned out to be with stack size - I have no idea why this is happening in just this one system and works with every other centos, but the solution was as following:

1. Patch /etc/init.d/varnish adding a new ulimit call in start()

        # Stack size
        ulimit -s unlimited

2. Increase system limits in /etc/security/limits.conf accordingly

varnish         stack   soft            unlimited
varnish         stack   hard            unlimited

3. Restart Varnish

After successfully running Varnish this way, be sure to change the stack limit back to a reasonable size! Unlimited is not a very wise setting, but helps debugging the problem. Now Varnish and Joomla could happily work together.

Thursday, August 16, 2012

vztop in vzprocps: fixed

There has been a frequently encountered problem with vztop on OpenVZ making the tool totally useless - it worked perfectly except for it didn't show virtual machine IDs for processes. Instead of actual numerical ID, it showed "N/A" in the VEID column.

So I spent an hour or so debugging, and it turned out that vztop, when requested to show the VEID column, didn't set a flag which enabled collection of VEIDs and other data. While diving into vzprocps source, I also made vztop show VEID by default. The result is this simple patch against vzprocps 2.0.11-2.

Apply with patch -p1 <vztop_veid.txt

If you need to debug a problem with some piece of software, I'm for hire at Massive Scale. I have never seen vzprocps source code before and still the solution was available in just 1 hour of work. That's $70. Would your $15/h programmer solve the problem as efficiently?