Server Crashes, Up Coming Trainings

If you spend much time here or if you have a lot of FreePBX servers that you look after then you are probably aware of the troubles we have been haunted with over the last months. We want to let you know more about this and what we are doing to address these issues and assure there is a robust infrastructure going forward. In the mean time, let me remind those of you about our [url=/open-telephony-training-seminar]Fast Approaching FreePBX Training[/url] that we told you about in our [url=/news/2011-11-07/2-10-beta2-and-come-see-us-in-naples-florida]last blog[/url]. There’s still time to take advantage of the [url=/open-telephony-training-seminar-register]Early Bird Discount[/url] that won’t be around much longer so act fast. Also keep in mind we had to turn away multiple attendees in our last training so register before it fills up! For those of you that take the expense off in your taxes, act quick to get a 2011 deduction!

The last time we gave our infrastructure a boost was almost 5 years ago in early 2007 when we were having problems with the old hosting facility and underpowered hardware. We’ve been chugging on that same piece of iron now and it’s starting to show. The site was down for several hours the day after Thanksgiving which was eventually diagnosed to a bad NIC (network interface). It went down again a few days ago for what looked like a disk starting to go bad. Beyond these, we’ve had problems for months, often with the system thrashing on swap space. Despite reasonable efforts trying to locate a root cause we eventually resorted to a nightly restart of apache to help temper the system, a real [i]”ghetto”[/i] bandaid we are not proud of.

The time has come to deal with our infrastructure head on so here’s a little information about what makes up the FreePBX site and the steps we are taking to assure a solid future. We will approach this in two phases, the first and most urgent being to take the current environment and move it to new hardware and facilities along with some additional changes to make it more robust. The second and longer term step being to investigate the modernization of the core underlying applications that make up the FreePBX site.

At a macro level the FreePBX site is made up of two primary components. There is this website you are currently reading which uses a dated version of the [i]Drupal Content Management System (CMS)[/i] in conjunction with a dated version of the [i]Trac[/i] ticketing/project management system paired with the [i]Subversion (SVN)[/i] repository. There are a handful of other components and some custom modules as well that glue everything together and provide the single sign-on ability between Drupal, Trac and the underlying SVN access. The second component is the FreePBX [i]mirror[/i] site which provides the backend infrastructure for the online Module Admin repository and server side publishing tools as well as the general repository for all other FreePBX tarballs and downloads. This software is almost all custom and is used to manage the environment in conjunction with a little MySQL There is actually a third component that we have not touched on, which is all the parts involved in the FreePBX Distro. This was developed from inception on a new infrastructure so does not need to be addressed here. In essence the old FreePBX infrastructure is now catching up with the lead that the Distro has taken.

With a lot of help from, we have taken some initial steps already which include securing new and much beefier hardware at a new hosting facility, and staging the the [i]Site[/i] and [i]Mirror[/i] into two virtual instances that will be placed on the new hardware. Our intention is to replicate the Mirror site to a second hosting facility and enhance the backend environment to keep both of these up-to-date at all times. We are currently considering either roundrobin DNS records for some level of high availability and load balancing on the Mirror sites, or updating FreePBX to do it directly. Both have their advantages and flaws. For the Site configuration, we plan on having a duplicate virtual instance which will be constantly replicated from the main Site using a master-slave MySQL replication in conjunction with [i]rsync[/i] and other custom scripts to assure an up-to-date copy at all times. We will also be employing other levels of traditional backup for both to assure that everything is robust. We want to re-iterate the gratitude to Schmooze, and in particular to Bryan Walters (GameGamer43) and Tony Lewis for their dedication as well as expertise in helping with this transition.

At this point we are already in process of creating these two environments and we hope to do an actual switch over in the next two weeks. If all goes well it should be almost transparent though we will update you when this will be happening and of any planned outages we may need just prior to assure a clean transition with no data loss. Once we are up on the new infrastructure we will be well positioned to start planning the second phase which is the modernization of the Drupal and Trac environments. Since the new Site will now live within a virtual container, it will be very easy to replicate into a development environment that will allow us to work on a migration plan to updated the software, which has not been something we have had available up to now. We will provide more news on this process once we get through the first phase as well as let you know where community help might be desired in doing such.

A quick note on the 2.10 release. We continue to work strong on this with a handful of last minutes additions we have put into it. Juggling between the site issues and the beta release have drawn it out some though there are dozens of production sites that are running solidly on the 2.10 beta with Asterisk 1.8 so we are feeling very strongly about this. We plan on teaching the next [url=/open-telephony-training-seminar]FreePBX Open Telephony Training Seminar[/url] on 2.10 in sunny Naples, Florida next month so rest assured we have strong motivation to get things wrapped up with this release. We hope to see many of you next month so don’t procrastinate and end up having the class filled up on you! Get registered now, get your 2011 tax deduction and know you will have some sun in the middle of the winter next month!

[b]Philippe[/b] – On Behalf of the FreePBX Team