307 Temporary Redirect, to myself. Lets hope the image is done next time around

Funny FAIL-bug in the drupal imagecache module. Last night we had some serious trouble with our sites and witnessed the requests on our loadbalancers going from 1k to 10k pr second. From our graphs we found out which site was being hammered and then checked varnishtop to see what was going on. 2 missing images were causing a 307 temporary redirect. We fixed it fast by touching the missing files and the traffic went away. Today we did some research into what was going on, and with firebug we found out that the page was trying to redirect to it self and fetch the same missing image over and over. Here’s the rather naughty code in the imagecachemodule :


if (file_exists($lockfile)) {
watchdog('imagecache', 'ImageCache already generating: %dst, Lock file: %tmp.', array('%dst' => $dst, '%tmp' => $lockfile), WATCHDOG_NOTICE);
// 307 Temporary Redirect, to myself. Lets hope the image is done next time around.
header('Location: '. request_uri(), TRUE, 307);
exit;
}

Now imagine if the image “doesnt exist the next time around”.
FAIL!

Tags: , , , ,

Nvidia and invalid checksum for EDID (Xorg-issues)

Having troubles installing a DVI splitter through a HDMI converter on our Jira-dashboard i found that the splitter made the EDID(Extended display identification data) fancy automagically validation shit fucked up and made the screen falling back to 640×480.
This made me shat brix.

After alot of googling i found a faboulous trigger called IgnoreEDIDChecksum that i put under the Screen section in the xorg.conf.

Hurrayh for new fancy automagic-probe-validation-fuckups

Tags: , , ,

Our new dashboard

The old dashboard we used earlier had a couple of issues. It showed all SOFT nagios states and it also listed every service pr host that was down. Since it’s pretty obvious that a service is down on a host that is down, we wanted to change that. Instead of continuing the rather hard work of changing the dirty status.dat parsing, we just dropped that project and checked out Merlin. Once installed and configured correctly, merlin will enable an eventbrokermodule in the nagiosconfig and update merlins mysql database via the eventbroker. The database contains all hosts and statuses state changes and so on, so this is what we ended up with : (Pic of our current dashboard in our office)

dashern

This dashboard lists only hosts that are down and not acknowledged in nagios in its upper left corner. Then there’s a little tactical overview in the upper right corner (this will have more info shortly) and finally all unhandled serviceproblem listed below. Exactly what we want.

The bottom “toolbar” is transparent and has a countdown timer for page refresh and shows the current time.
Thanks again to Jonas, for the design!

You can download the files from git : https://github.com/mortis1337/nagios-dashboard
Just change the login info in merlin.php to match your merlin database and it should run smoothly.

Note: The dashboard needs firefox 3.6.
Enjoy.

http://dingleberry.me/2010/04/our-new-dashboard/

Tags: , ,

Mobile redirects using Varnish

Our loadbalancers TMM has gone up quite a bit lately, so I started looking into how to move some of the workload to Varnish instead. I came across this configexample and pulled out the mobile redirects part. It’s a rather dirty hack, but it works. Varnish does not have support of HTTP redirects, so you have to trigger an error and then pick it up in the vcl_error subroutine later.
This is what the redirect-config looks like on my test-system :

sub vcl_recv {
if ( req.http.user-agent ~ "(.*Blackberry.*|.*BlackBerry.*|.*Blazer.*|.*Ericsson.*|.*htc.*
|.*Huawei.*|.*iPhone.*|.*iPod.*|.*MobilePhone.*|.*Motorola.*|.*nokia.*
|.*Novarra.*|.*O2.*|.*Palm.*|.*Samsung.*|.*Sanyo.*|.*Smartphone.*
|.*SonyEricsson.*|.*Symbian.*|.*Toshiba.*|.*Treo.*|.*vodafone.*
|.*Xda.*|^Alcatel.*|^Amoi.*|^ASUS.*
|^Audiovox.*|^AU-MIC.*|^BenQ.*|^Bird.*|^CDM.*|^DoCoMo.*|^dopod.*
|^Fly.*|^Haier.*|^HP.*iPAQ.*|^imobile.*|^KDDI.*|^KONKA.*|^KWC.*
|^Lenovo.*|^LG.*|^NEWGEN.*|^Panasonic.*|^PANTECH.*|^PG.*|^Philips.*
|^portalmmm.*|^PPC.*|^PT.*|^Qtek.*|^Sagem.*|^SCH.*|^SEC.*|^Sendo.*
|^SGH.*|^Sharp.*|^SIE.*|^SoftBank.*|^SPH.*|^UTS.*|^Vertu.*
|.*Opera.Mobi.*|.*Windows.CE.*|^ZTE.*)"
&& req.http.host ~ "(www.somehost.com)"
&& req.url == "/") {
set req.http.newhost = regsub(req.http.host, "(www)?\.(.*)", "http://m.\2");
error 750 req.http.newhost;
}


sub vcl_error {
if (obj.status == 750) {
set obj.http.Location = obj.response;
set obj.status = 302;
deliver;
}
}

Tags: , ,

Automatically create bugs in Jira with a Nagios eventhandler

The most important part about this is ….dont use it too often, but it CAN make sense on really critical events, like warnings/criticals on partitionspace. For instance, if your mysql server is running out of space on /var/lib/mysql and your operationsteam didnt see the WARNING/CRITICAL notification from Nagios, it might be a good idea to have the bug created in Jira to make it even more visible.

Here’s how you do it.
First of all, be sure to have eventhandlers enabled in Nagios.

Configure your commands.cfg file to have something similar to this :

define command{
command_name jira_eventhandler
command_line $USER1$/jira_eventhandler -a morten -s $SERVICESTATE$ -t $SERVICESTATETYPE$ -A $SERVICEATTEMPT$ -H $HOSTNAME$ -S $SERVICEDESC$
}

Configure your services.cfg to have something similar to this :
define service{
use generic-service
host_name myhost
service_description CHECK_DISK_ROOT
is_volatile 0
max_check_attempts 3
normal_check_interval 10
retry_check_interval 1
contact_groups linux-admins
notification_period 24x7
notification_options c,w,r
check_command check_remote_disk_nagios!10%!5%!/
process_perf_data 1
event_handler jira_eventhandler
flap_detection_enabled 0
}

And be sure to have the jira_eventhandler script in place. You can download mine here : jira_eventhandler

Tags: , , ,

Easy Big-IP LTM backup

Recently we upgraded the sw on our bigips, and then all our custom local scripts were gone.
Thats why i wanna blog this easy bigip recipe in case we are stupid enough to do this again.
This is the backup procedure for BIG-IP 9.4.x

/usr/local/bin/bigipbackup.sh: here

/etc/cron.d/backup:
0 21 * * * root /usr/local/bin/bipipbackup.sh > /dev/null

Tags: ,

XBMC compiled with Crystal HD support is now available from unofficial nightly builds

TheQuestor from xbmc forums has announced that his xbmc for osx nightly builds now includes crystal hd support!

Forum post found here.
And downloads found here.

You will still need to compile, or download a precompiled version from somewhere, crystal hd lib and kext to get it working. Xbmc will detect the presence of said lib/kext and enable crystalhd rendering as an option if found.

Follow this post from the xbmc forum on how to use the nightly build together with the atv-xbmc-launcher. You will have to install the crystal hd lib/kext as well to enable the crystalhd rendering option from within xbmc.

Enjoy Apple TV with 1080p support :)

Tags: , , , , , , , , , , , , , , ,

CrystalHD AppleTV OSX kext lib is ready for download

Yay, Finally we have osx support for broadcoms crystal hd! This means it is now possible to keep the existing Apple TV os AND run xbmc with CrystalHD hd video acceleration :)

Scott Davilla finally made his efforts in porting the broadcom open source linux crystalhd driver to osx lib and kext available to the public :)

More info on Davillas efforts here
Visit broadcom for more info on Crystal HD and check here for windows and linux driver downloads.

Tags: , , , , , , , , , , , , ,

Compile xbmc with broadcom crystalhd support on linux

Broadcom CrystalHD is a Mini-PCIe expansioncard to make your box do hardware decoding of HD video material (h264, vc1++)
Got a old AppleTV which you dont use? Buy a CrystalHD chip on ebay for 30 dollars, and it will play 1080p for you :)

Install Ubuntu 8.04 Hardy on your box, if you are going to use an AppleTV you can follow the guide here:
http://dingleberry.me/2009/11/ubuntu-karmic-koala-on-appletv-with-broadcom-crystal-hd-decoder/

Download the xbmc crystalhd branch and the lib directory from the xdmx branch which has the necessary crystalhd include files:
svn co -r 22347 https://xbmc.svn.sourceforge.net/svnroot/xbmc/branches/xdmx/XBMC/xbmc/lib/
svn co -r 25101 https://xbmc.svn.sourceforge.net/svnroot/xbmc/branches/crystalhd

prepare the system for crystalhd support:
mkdir /usr/include/crystalhd
cp -a lib/crystalhd/include/* /usr/include/crystalhd
cp /usr/include/crystalhd/windows/bc_drv_if.h /usr/include/crystalhd/bc_ldil_if.h

Edit /usr/include/crystalhd/bc_dts_types.h and comment out line 59 so it looks like this:
//typedef void VOID;

Copy the crystalhd include files to the crystalhd branch source:
cp -a /usr/include/crystalhd crystalhd/xbmc/lib/

Download the bcm70012 source and build the module:
wget http://www.norco.net.cn/UpLoadFile/Driver/bcm70012_linux_01142009.rar
unrar e bcm70012_linux_01142009.rar
tar xf bcm70012_linux_01142009
dpkg -i bcm70012kmod-source_0.9.30_all.deb
apt-get -f install
dpkg -i bcm70012kmod-source_0.9.30_all.deb
cd /usr/src/
m-a -t a-i bcm70012kmod
cd -

Install the gstreamer plugin from the HP mini 110 distro (the file came with the driver download in the step above)
dpkg -i --force-all gst-broadcom-70012_0.9.30_lpia.deb

Now follow the steps from the README.ubuntu file inside the crystalhd source:
sudo aptitude install subversion make g++ gcc gawk pmount libtool nasm automake cmake gperf unzip bison libsdl-dev libsdl-image1.2-dev libsdl-gfx1.2-dev libsdl-mixer1.2-dev libfribidi-dev liblzo2-dev libfreetype6-dev libsqlite3-dev libogg-dev libasound-dev python-sqlite libglew-dev libcurl3 libcurl4-openssl-dev x11proto-xinerama-dev libxinerama-dev libxrandr-dev libxrender-dev libmad0-dev libogg-dev libvorbis-dev libsmbclient-dev libmysqlclient-dev libpcre3-dev libdbus-1-dev libhal-dev libhal-storage-dev libjasper-dev libfontconfig-dev libbz2-dev libboost-dev libfaac-dev libenca-dev libxt-dev libxtst-dev libxmu-dev libpng-dev libjpeg-dev libpulse-dev mesa-utils libcdio-dev libsamplerate-dev libmms-dev libmpeg3-dev libfaad-dev libflac-dev libiso9660-dev libass-dev libssl-dev

On 8.04 and older versions, libmms is outdated and thus XBMC will not compile properly.
In this case you will have to manually compile the latest version.
wget "http://launchpad.net/libmms/trunk/0.4/+download/libmms-0.4.tar.gz"
tar zxvf libmms-0.4.tar.gz
cd libmms-0.4
./configure --prefix=/usr
make
sudo make install

After all deps has been installed and libmms has been compiled and installed you are ready to compile xbmc. cd to the crystalhd source and do:
./bootstrap
./configure
make
make install

Finito! :)

POST-install tweaks:
Enable Pixel Buffer Objects
Start xbmc without loading Gnome
And some more AppleTV specific stuff:
Set fixed cpu frequency
Various fixes for Ubuntu on ATV

Tags: , , , , , , , , , , , , , , ,

Ubuntu Karmic Koala on AppleTV with Broadcom Crystal HD decoder

UPDATE! – Karmic Koala uses a newer kernel which wont make the bcm driver work, use Ubuntu 8.04 Hardy in step 4, and your driver will work :) Confirmed 1080p playback on Apple TV with hardy! :D

1. build patchstick with http://code.google.com/p/atvusb-creator/
2. boot atv with patchstick
3. Partition sda as per instructions:

http://code.google.com/p/atv-bootloader/wiki/PartitioningLinux

4. Download netboot http://no.archive.ubuntu.com/ubuntu/dists/karmic/main/installer-i386/current/images/netboot/netboot.tar.gz
4. Run netboot as per instructions under Netboot Install:

http://code.google.com/p/atv-bootloader/wiki/BootingLiveCD

5. Install karmic with sda4 = / and sda5 = swap
5.1 Remember to select sda4 as root filesystem with ext3 NOT ext4
6. Modify boot_linux.sh with something like the following:

mount /dev/sda4 /tmp/
kexec --load /tmp/vmlinuz --initrd=/tmp/initrd.img --command-line="root=/dev/sda4 initrd=initrd.img vga=normal vesa video=vesafb"
umount /tmp/
kexec -e

7. WORKING UBUNTU WOOHOO

8. Download linux bcm70012 from:

http://www.norco.net.cn/DownNews_Show.asp?ID=215

Making system ready:
aptitude install build-essential unrar
unrar e bcm70012_linux_01142009.rar
cd bcm70012_linux_01142009
mv bcm70012_linux_01142009 bcm70012_linux_01142009.tar
tar xf bcm70012_linux_01142009.tar
dpkg -i bcm70012kmod-source_0.9.30_all.deb
apt-get -f install
dpkg -i bcm70012kmod-source_0.9.30_all.deb
cd /usr/src/
m-a -t a-i bcm70012kmod
cd -
dpkg -i --force-all gst-broadcom-70012_0.9.30_lpia.deb

22. WORKING bcm70012 !! WOHOO

Tags: , , , , , , , , , , ,