It’s been several years since I’ve blogged here and I’m honored to have the privilege of wishing Andrew the best on his future endeavors! For those that don’t know me, I used to run FreePBX prior to Andrew taking the helm so it only seemed appropriate that I have the opportunity to wish Andrew the best and further reflect on where FreePBX has been and is going!
Andrew has had a wonderful journey through his involvement and influence on FreePBX and the Asterisk world. The marks that he’s made on the project will continue for years to come. He has had his hands on so many different aspects of the project that I couldn’t even begin to enumerate everything he’s touched! Luckily we have ‘git blame’… Since I haven’t been involved in the day to day goings of FreePBX I thought I’d touch on my views and reflections of the attributes of a successful Open Source project like FreePBX.
I’ll start with some minor reminiscing of my own. My involvement started at the end of 2005, before there was even a FreePBX as it was still called AMP (Asterisk Management Portal) and just prior to Rob Thomas taking over the project. The database schema at the time comprised of 8 tables if I recall correctly, sip, iax, zap and extensions, and a few others. In my case I had just left the corporate world looking for something more fun to do and wanting to get back into telecom. I hooked up with a good friend who had been playing around in this space and immersed myself in this new world.
Rob Thomas had been running the project but other life priorities were competing for his time so I ended up taking it over. This was a time during the FreePBX life that I equated to the 60’s. It was a wild and free time where any and every contribution thrown at the project was getting absorbed without any real structure or planning or release management. It was a great period of growth and recognition but at the time, with the popularity of Asterisk@Home FreePBX was taking off and there was a huge boom in the commercialization of and adoption of FreePBX and Asterisk.
Upon taking over, it was my goal to provide stabilization for the project while continuing to keep it a vibrant and creative environment. These are somewhat opposing forces but necessary to make sure it would come out as an innovative project that thousands of resellers could build dependable businesses and millions of installations could result. During my tenure I accepted the help of energetic and significant forces in the community, whether individuals who came and left or significant players such as Schmooze, with whom I eventually joined forces. The goal was a continued and vibrant creative base to make sure that FreePBX excelled and at the same time, a strong supporter who eventually became Sangoma after FreePBX’s path through Bandwidth.com and then Schmooze who merged with Sangoma.
One underlying force behind Open Source is the collective and ever shifting brain trust that the openness nurtures. It’s through this combination of the constant giving back to the project, and the continuous inflow of new ideas as significant contributors enter the project for their tenure before moving on. Just as this cycling of various community members fuels the innovation in a project, so does the life cycle of its various leaders. Rob Thomas had a charisma and openness to invite the huge influx of contributions during the project’s early days at a time when that’s what it needed. Although Rob could never say ‘no’ and it resulted in plenty of headaches that took years to undo, it was also absolutely the right thing to do. It gave momentum, ideas, innovation and fuel to a great project. I was able to take that momentum and steer it in a direction to give stability and confidence while continuing to allow innovation but at the same time, occasionally having to say no. That took it well into the Schmooze days when we began the transition to let a new wave of creative energy take FreePBX to the next level and move Andrew into the role to take it somewhere I never could have.
Andrew’s leadership has taken the project to new levels that would never have happened had I stayed at the helm. The work that Andrew did, as well as Rob who resurfaced after a 5 year sabbatical, and James, Jason, Kapil, Philip, Mohit, Franck and others is simply astounding. Andrew has led the project through total rewrites and architectural enhancements of everything from its underlying core to the server side infrastructure that supports FreePBX to the surrounding ecosystems such as Zulu and beyond.
I’m very excited about what is possible going forward. For the first time in over a decade we’ve brought the two underlying technologies of FreePBX and Asterisk under a common umbrella. This has been an evolution that was once ridden by old school grumpy “real men program in dialplan” … they don’t use GUIs. Fast forward 14 years and we have these two symbiotic technologies under one roof. I know all of us who have been close to the project, whether visibly active or silently watching, won’t be far in keeping tabs on what this recent marriage can do in taking FreePBX into its next leap into the future.
For now, please join me in giving a huge thanks to the contributions that Andrew has achieved in his leadership and wishing him well in his next endeavors. Thank you Andrew!
When asked in high school back in 2003 what I wanted to do when I ‘grew up’ I had always said I wanted to work in both technology and technical theater. Over the last 16 years I’ve watched as the world of technical theatre expanded as a result of implementing new technologies. As a result of this I often found myself at the forefront of learning how to manage these new technologies in the theatrical world which one day in 2008 lead me to discovering Asterisk and FreePBX. I had just graduated from California State Fullerton University in June and I was working part time maintaining the network at an event production company and part time as a freelance theatrical sound engineer.
As the story goes, one day I was sitting at the desk at the event production company using a Grandstream GXP 2000 when I noticed the phone wasn’t connected with a standard RJ11 connector but instead was connected with an RJ45 connector. Fascinated, I then went through the menus of the GXP and realized this wasn’t the normal copper telephone I grew up with (and yes I am ‘old’ enough to have used copper telephones). Digging deeper I discovered the IP address that the phone was registered to and attempted a connection to it.
This looks pretty similar to what I saw the first time I logged into the machine
When I connected to that IP address I discovered the world of… Trixbox, funnily enough. Unfortunately I didn’t have the username and password for the interface and since it was a Friday I knew I wouldn’t get a response from the IT company that was hired before me to get the password from them. I was, however, curious as to what a ‘trixbox’ was. I also noticed normal copper RJ11 jacks on the back of the machine and I was fascinated that you could connect your home or business telephone system directly into a computer for telephone questions.
So I did what any reasonable 23 year old does, I found the download for trixbox and attempted to reformat the entire machine and so began the weekend where I dove head first into the world of Asterisk/FreePBX/Trixbox. At some point while installing the trixbox media I noticed that Trixbox was a clone of a product called FreePBX. I went to the FreePBX website downloaded FreePBX (there was no easy FreePBX distro back then!) and tried to make it work to no avail. Since I formatted the machine with previous trixbox media, the company I worked for no longer had a working phone system. Luckily it was a Saturday afternoon.
The FreePBX website circa the time I discovered FreePBX
Over the course of the weekend I learned more about Asterisk, TDM, and FreePBX cards than I ever thought I would. At some point over the weekend I got Asterisk to recognize the TDM card and the telephone system was back online and I thought all was going to be right with the world. Yet I ended up being very intrigued by this “Free” telephone system and all of it’s features and the power it contained.
The original Polycom 501. Still in operation today at the event production company
The only problem I had at this juncture was that the phones in the office didn’t know how to connect to Asterisk anymore. The phones I decided to work on first were in the boss’s office. A Polycom 501. I thought to myself “how hard could it be to configure this thing”. Boy was I wrong.
After dealing with polycom provisioning for what seemed like hours, I next attempted to configure the grandstream gxps. Although these weren’t as hard as the polycoms I continued to ask myself why there wasn’t an easier way to configure these phones remotely. They all seemed to support dynamic DHCP and TFTP options to retrieve configuration files but all attempts at scouring the internet returned only results of people configuring these devices manually.
Around this time I discovered the PBX-in-a-Flash distribution and the community that surrounded it. At the time it seemed like a far more vibrant and active community that the FreePBX forums. There I found a post announcing a GUI for configuring phones for FreePBX written by a forum user named John Mullinix. I sent him a quick private message (that I still have!)
My name is Andrew Nagy and I was a former trixbox user for about a year. I loved how easy it was to use their endpoint manager but their implementation of FreePBX and Asterisk was rather weak I thought, which is why I came over to PBX in a Flash. When I saw your project for an endpoint manager I thought it was pretty cool. And so far it’s worked wonders but I wanted to know if I would be able to help out with coding on the project. I have a ton of PHP (4–5) and MySQL(4–5) experience. My information onprojects.colsolgrp.net is tm1000. Same as here. I’ve worked with Linksys PAP2, Polycom and Grandstream phones. Specifically I’ve programed functions on the Grandstreams such as XML screens and such and I’ve worked with gd on PHP which is especially interesting as it’s compiled with PHP on PBX in a flash which perhaps could be used with endpoint manager for uploading personal images to phones. etc.
What other information do you need specifically?
I received a reply almost instantaneously
Thanks for volunteering Andrew. Tony [Shiffer] has anticipated my request and you have been added to the project.
If you need to call, you can reach me at
My email is <redacted>
I then began to work feverishly on rewriting Endpoint Manager for FreePBX for an open source initiative group named Col Sol Projects. They mainly had open source projects for FreePBX such as CID Superfecta and of course Endpoint Manager. This is where I ended up meeting Lorne Gaetz through his contributions to CID Superfecta. However their Endpoint manager ran separately from FreePBX itself. So I took the opportunity to learn about FreePBX internals to be able to re-write Endpoint Manager into FreePBX itself.
Open Source Endpoint Manager in it’s early days
I continued to work on Endpoint Manager in my spare time throughout 2009.
It’s around this time that I also ran into the blogger named “Michigan Telephone” and started reading his blogs about FreePBX and I started actively following the blogs from Philippe Lindheimer of FreePBX. I also worked on a few other modules you might not have ever heard of, including swiss army knife, which included some things I felt the community wanted but weren’t in FreePBX (like the ability to add old school dial patterns in inbound and outbound routes). I also wrote the motif module, which allowed users to use Google Voice and I based that work off of Marcus Brown’s original Google Voice module. Marcus wrote code for the original Jabber implementation and I did a module based on the Motif implementation.
Then one day in June of 2010 I got an email from a “Tony Lewis” of Schmoozecom asking me if I would give him and Philippe a call. The next thing I knew I was a ‘contributing’ developer to the FreePBX codebase and Endpoint Manager was distributed to FreePBX systems all over the world.
Around this time I also met Darren Schreiber and Patrick Sullivan of 2600hz. Two stellar guys. I even went up to work on provisioning services for them in San Francisco from time to time. At one point a few months later I almost ended up giving up on Endpoint Manager after an altercation with a FreePBX developer who is no longer around. That is the first time I met Bryan Walters, also of Schmoozecom. He encouraged me to not listen to said developer and to keep participating in the community and in Endpoint Manager.
Sometime in the early summer of 2012 Tony Lewis asked me if I wanted to go to Astricon. I gladly accepted and flew out to Atlanta in the fall of 2012. There I meet in person; Tony Lewis, Luke Duquaine, Philippe Lindheimer, Bryan Walters and Preston McNair of Schmoozecom and Jason Parker and Josh Colp of Asterisk.
By the summer of 2013 I was officially working for Schmoozecom on FreePBX under the direction of Philippe Lindheimer and over the years I gained more responsibility within the FreePBX project. The Schmoozecom years were a period of massive growth for the FreePBX project, the company hired James Finstrom (formerly of Rhino), Jason Parker (from Digium) and Rob Thomas (who pretty much created ‘FreePBX’ as you know it today). Rob, Jason, Bryan, Tony, James, Luk, Philippe and I created fwconsole, we fixed the installer, we fixed autoloading so it’d work and Rob Thomas added what’s now known as BMO, which changed the game for FreePBX turning it into a more object oriented code base. You’d also probably be surprised to know that I never worked on the commercial FreePBX Endpoint Manager. Luke Duquaine ran that originally then passed it off to Jason Parker who passed it off to Kapil Gupta, who still works on it today.
Sangoma later acquired Schmoozecom on January 1, 2015 which gave FreePBX more global recognition than it ever had before. At some point in the last few years Philippe moved on to other aspects of Sangoma’s business operations (he actually still works for Sangoma!) while I became project lead for FreePBX.
In the years that followed the FreePBX team accomplished several goals support for PHP 5.6 and we brought in and implemented more object oriented concepts. We had 3 releases: 12,13,14 and soon we will have 15. I’ve gone to every Astricon since 2012 (So 7 years!) and I’ve met so many wonderful members of the community here and on the forums. I’ve really treasured my time running this project. If you’ve read my posts on the forums or followed me for any length of time on here you’d probably think I was pretty blunt/rude in my replies. However I did care about what each and every one of you posted and often brought it up to Rob, Tony, Bryan, Jason and Philippe when we all worked together on FreePBX to find solutions for people’s issues. I am the type of person that tries to give facts as they are, without “fluffing” it up. If I’ve offended you in the past I am sorry for that, but I hope my contributions to the project make up for that. Looking back I remember a previous developer conference we had in Neenah Wisconsin (the former HQ of Sangoma US) where the community said Yealink phones were having issues with FreePBX 14 and Rob Thomas and I walked to the “Schmoozecom wall of phones” pulled down a Yealink and worked on trying to figure out the issues together for a whole day. Those are times I truly treasure with the FreePBX project and the community, including YOU!
Over this last summer Sangoma and Digium merged into one company and we gained a huge array of wonderful Asterisk developers (not only developers but the entire Digium staff). For once I was finally able to work directly with Matthew Fredrickson, Joshua Colp and George Joseph. All Asterisk developers that I used to communicate with only over IRC. I also had the pleasure of spending time working with the Switchvox team in San Diego and I can’t express how welcoming the entire Digium team has been. You might have noticed a few of them participating in the forums recently.
One of the last major projects I was able to work on was something I’ve been ‘hotly’ involved in over the last year and that is FreePBX CPU utilization, you can read more about that in my performance improvements post here: https://community.freepbx.org/t/performance-improvements-in-freepbx/57016. But also our recent Framework upgrades that consolidate all of our crons in the Asterisk user’s crontab into a single cron in PHP, this allows all FreePBX cron code to run in “sequence” and only have a single startup cost, we think you as the community will really enjoy these changes.
After much thought and deliberation I’ve decided that it’s time for me to move on to new opportunities. I’ve worked with FreePBX and Asterisk for a little over 10 years (since the fall of 2008). I’ve worked professionally in open source technologies during my entire time at Sangoma and Schmoozecom. Over the last 10 years I’ve learned so much about the open source community and I really do love it. I’ve met so many wonderful people at Astricon and online through various mediums (the forums, dslreports, reddit, IRC). I’ve seen how much FreePBX has affected other people’s lives. I’ve seen and heard about the major places it’s used and even the minor places and the impact it’s had on businesses. Hopefully you can see how much it’s affected mine. However, everyone has an expiration date and I’ve finally reached that time. I’m excited about what the future holds for me and hopefully I’ll run into you all around the community in the future. My last day with Sangoma will be this Friday (The 17th of May).
Taking my place will be Matt Fredrickson of the Asterisk project. Matt will now be the manager for both the Asterisk and FreePBX projects. He’s been working with Digium since the early days and has an employee number almost as low as Malcolm Davenport. Matt’s been in charge of the Asterisk Open Source project for the last few years and he’s done a stellar job at it. I have no doubt that his leadership will help continue to grow FreePBX in this new era.
I want to encourage all of you that FreePBX itself is in a very strong place. The team is working hard to release a few wonderful additions in the next year and I wouldn’t be surprised if FreePBX fully supports PHP 7.2 in the next year!
Thanks for the great years FreePBX community. You are all great in your own individual ways!
If you’ve been around the forums long enough you’ve probably seen posts and comments from our staff and even our users referring to the “Sangoma 7 Distro” or “SNG7”. The Sangoma Distro is a derivative of CentOS which itself is a Linux distribution that provides a free, enterprise-class, community-supported computing platform functionally compatible with its upstream source, Red Hat Enterprise Linux (RHEL). This means that the Sangoma Distro itself is a variant of Red Hat. The “7” references the “7” in the CentOS Distro and RHEL 7.
The Sangoma Distro started it’s life close to 10 years ago as the Schmooze Distro (way back with CentOS 5) and slowly morphed/grew/upgraded into the FreePBX Distro and now the Sangoma Distro. It’s seen many iterations over the last 10 years but it is still primarily based on the CentOS and RHEL distro releases. Which gives FreePBX stability from upstream Red Hat.
On top of that the Sangoma 7 Distro also includes access to the EPEL repository, also known as “Extra Packages for Enterprise Linux”. Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high quality set of additional packages for Enterprise Linux, including, but not limited to,Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including buildsystem, bugzilla instance, updates manager, mirror manager and more.
What this means for you, the end user, is access to hundreds of other RPMs for your FreePBX system that you can use to enhance your product. Additionally the Sangoma team also builds PHP 5.6 (which is used by FreePBX 14) and NodeJS 8.16.0 (used by some of our commercial products).
A few weeks ago we announced (https://community.freepbx.org/t/sng-7-6-testing-release/58142) that the CentOS 7.6 updates, released by CentOS back in October 2018, were now available to FreePBX users in the Sangoma Distro through our testing repository. This is, in large part, thanks to the collaborative efforts of engineers at Digium and Sangoma working together. Matteo Bignotti, who joined Sangoma as a part of the Digium acquisition, was one of those engineers and I want to briefly highlight him in this post.
Matteo is a true blue Italian, originating from the sunny city of Milano over 30 …plus a few more… years ago. With his seductive accent and a talent for BSD/Linux applications, Matteo pursued a career as a web programmer back in the late 90s working together with Telecom Italia at their MMS/sms portal delivering content to their users. His technological thirst was not quenched though, and he knew he could not stop his quest for knowledge. It was one spring day in 2005 that he discovered the intricacies of phone systems with Asterisk and Switchvox and set upon what he thought would surely be a new career path.
After working at several personal Asterisk projects, Matteo had a revelation. While he truly appreciated his new skill, he realized his efforts would be better spent writing code as a result of his unique ability to see instinctively how systems should interact; and how to make them work. It was then Matteo recognized his true calling had been right all along as a system developer he joined the Switchvox team in 2012 and quickly owned the distribution and has been working tightly with CentOS ever since.
As a result of his versatility, extensive experience and savvy familiarity with all things web, mobile, client and server, he constantly rises to any challenge thrown at him and loves every minute of it.
Favorite quote: “I had a broken watch once and even it was right twice a day”
So now you know a little bit more about Matteo and his experience working with CentOS and how that has already benefited the FreePBX Distro. Moving forward we hope to have Matteo more involved in helping with the FreePBX “SNG” 7 distro.
About a week ago our QA team notified us that the SNG 7.6 upgrades were ready to go into “GA” (General Audience) release. Therefore, we are proud to announce that the QA cycle has completed and as of last week we pushed the release of 7.6 into our stable branches. Which means if you ran a yum upgrade on your system you were presented with well over 200 packages to upgrade. Many including performance and security fixes and at no cost to you.
We have also updated our downloadable distro, in which we removed the ability to install Asterisk 15 from a base install (it’s now End of Life) and upgraded our base Asterisk versions to 13.22.0 and 16.3.0 which are both selectable at installation time. To download the distro and try out the updated installer checkout our downloads page: https://www.freepbx.org/downloads/freepbx-distro/
One more thing: Have you always longed for the day where you could install Asterisk Certified on Sangoma 7 and utilize it with FreePBX? Well, today’s your day! If you’d like to test out Asterisk Certified on your Sangoma 7 FreePBX system you’ll just need to run the following commands (note this is only in the testing repo):
When we started the development of FreePBX 15 around September 2017 we had high hopes for a release within six months. We announced that this release would include a redesigned backup module that would no longer require FreePBX restores to be from the same version you are restoring to (meaning in the past you could only restore 13 into 13, or 14 onto 14). However, software development never goes the way one plans for it to go.
There’s been a lot of work going on behind the scenes in getting FreePBX 15 to a stabilized release that we will explain below. If you haven’t seen our most recent blog be sure to check out: Performance Improvements in FreePBX. Additionally, during this time we were also able to release our highly anticipated Zulu 3.0 redesign and Queue Wallboard commercial modules.
Backup & Restore
Since backup’s inception on Nov 7, 2005 the main purpose of the module was to be able to backup FreePBX and restore it either on the same system or a brand new system. The way this was achieved for the past 14 years was to do a complete database dump of the FreePBX system and store that in a compressed backup file.
While doing a database dump is by far the easiest way to be able to backup a system, in the case of FreePBX it because increasingly more difficult to support restores from older FreePBX versions and sometimes even restores from the same FreePBX version. The reason being that FreePBX is a modular system and modules are constantly changing database table structures. If you load the database structure for a module from version 12 onto a module in version 15 the module won’t know what to do or how to migrate and you’ll be faced with errors that sometimes you can’t correct.
What FreePBX needed was the ability to utilize the modular system already provided to it and give the power of backing up and restoring to each module. In this model an individual module would assume responsibility for backing up and restoring all of its own data. FreePBX would no longer make global assumptions about modules.
There is one drawback to this idea. Namely that there are over 128 modules in FreePBX that need to be individually updated to support the new functionality the backup and restore module provided. Early on a decision was made that there would be no default fallback for unsupported modules because doing so would land you in the same boat as before the redesigned backup and restore.(Speaking of which, if you are a developer, supporting Backup and Restore is an easy process! Checkout the developer documentation for implementing Backup & Restore in your own modules)
Additionally, it was requested that each module should also be able to deal with restoring from legacy versions of FreePBX. This creates other technical issues in that we never wanted to load the FreePBX database dump back into our database because it would blow everything away. Our team came up with a unique solution to load the FreePBX MySQL database dump back into an in-memory SQLite database to use temporarily. Each individual module is then provided a handler to talk to the in memory database to query against it and then add the data it needs to the permanent database.
After many months of work and refinement our development team successfully updated 95% of the 128 modules and we are excited to be able to release the new Backup & Restore module for FreePBX 15 to the general public. We’ve also tested a few restores from FreePBX 14, 13 and yes, even 12 backup files and they all worked, however we know that not every FreePBX system is alike so we encourage you to run restores from older versions of FreePBX and report them to our bug tracker at http://issues.freepbx.org.
Additionally, because most of the data stored from each module are text strings, backup file sizes have been reduced significantly. The files can be so small that technically you could have FreePBX email you your nightly backup. (And we provide you with that ability in Filestore)
Backup & Restore has always had support for Local and FTP file storage locations. About four years ago a community member submitted an Amazon S3 patch and our own Bryan Walters committed that patch to Backup & Restore. The patch allowed users to store backup packages on Amazon S3. While this worked well for a while, the original code the patch was based on unfortunately hasn’t seen very many updates and thus was hard for the FreePBX development team to maintain.
When we started the planning for FreePBX 15 we wanted to make sure that the new backup module would be able to officially support Amazon S3 through a regularly maintained library. Then we found Flysystem (Spelled correctly!).
Flysystem is a filesystem abstraction library for PHP. By providing a unified interface for many different filesystems you’re able to swap out filesystems without application wide rewrites. Using Flysystem can eliminate vendor-lock in, reduce technical debt, and improve the testability of your code.
Enter the Filestore module. A centralized place in FreePBX to manage remote or local file storage locations. Right out of the gate FreePBX File Store supports Dropbox and Amazon S3 and with demand from the community we could potentially support any other service that is a supported adaptor from Flysystem.
The best part about Flysystem is that it gives FreePBX the flexibility in the future to be able to store items like recordings, faxes and much more away from your system and into the cloud.
Somewhere along the road of FreePBX 15 we also decided to be ambitious in adding an official API based on GraphQL with the ability to use a normal RESTFul API as well. In doing so we added the ability for FreePBX to become an OAUTH 2.0 server for API authentication.
Adding GraphQL or RESTFul API calls to 128 modules is not an easy task. At this moment we have about 10 modules completed. Since the FreePBX API is mainly a developer resource we decided over time that we would add features and functionality as developers contribute or request them.
If you’d like to upgrade your 14 system to BETA 15, simply go to Module Administration and check online for PBX Upgrader.
If you are the type of person that enjoys using Ansible then you’ll love that we added the ability to perform upgrades using PBX Upgrader without having to go to the GUI at all.
With the release of the Sangoma 7 Distro almost 3 years ago we are finally able to provide a modern PHP version. When we started the process of bringing FreePBX to a modern supported version of PHP we had to first go to the last release of the PHP 5.x series. Both FreePBX 14 and 15 only support PHP 5.6.x. However with the release of “FreePBX 16”, FreePBX will finally have a minimum supported PHP version of 7.3. Commercial modules will also work with FreePBX 16 without the need or use of Zend. Our team is starting on this work immediately and we hope to have something to show you in the next six months.
FreePBX 13 Support
Once we have our releases of High Availability for FreePBX 14 & 15 into stable we will be sunsetting our support for FreePBX 13, so if you haven’t already start making plans to upgrade to 14 or make a backup from your 13 system and restore that onto a new 15 installation.
In the next few months you’ll be hearing from me again as we release High Availability 14 and announce our stable release of FreePBX 15.
It’s been about 6 months since the engineering team last posted a blog on FreePBX and let me tell you, it’s been a busy 6 months. Last time we posted was shortly before the Digium/Sangoma merger. You’ll be happy to know that we still have a great team working on FreePBX as we did before the merger. Best of all, now that Digium and Sangoma are one team we are able to work on complex issues together that end up benefiting the open source community as a whole. Performance in FreePBX has always been a complex, multifaceted issue.
Performance problems can stem from many issues such as individual hardware, bad hard disks, corrupt ram, loading too many virtual machines on a single host. Our support team deals with so many different types of issues that it can be hard to track down the root cause at times. However there are also performance improvements that can come directly from FreePBX’s internal open source code. We will document four of these changes in this post.
Usage of AGI
The Asterisk Gateway Interface, also known as AGI, is a language-independent API for call processing. AGIs allow external scripts to manipulate Asterisk which lets Asterisk perform tasks that would otherwise be difficult or impossible.
In the world of AGIs, there are two types of AGIs actively in use today in Asterisk: AGI() and FastAGI(). Both have been in existence in Asterisk since around version 1.0. In the beginning days of FreePBX all it required was a LAMP stack; Linux, Asterisk, MySQL and PHP. To utilize FastAGI at the time meant you’d have to run a daemon that continually runs in the background listening on TCP port 4573. Additionally you’d need to effectively load all AGIs into this server at run time. Back in 2007 this was a huge mountain to climb so instead the developers of FreePBX (then known as Asterisk Management Portal) decided to use regular AGI and so was born dialparties.agi, one of the first agi php scripts to be used in FreePBX. Standard AGI is the simplest, and most widely used form of AGI. Standard AGI scripts run on the local PBX and communicate with Asterisk through socket descriptors (namely STDIN and STDOUT). The standard AGI allows for usage of all AGI commands, and is what this article will be discussing.
Fast AGI is the AGI over TCP sockets protocol. It allows for all AGI functionality except EAGI, and is provided as a solution to developers who need to run resource intensive AGI programs. By running the bulk of the AGI logic on another server, the Asterisk server itself can process calls and not worry about handling complex computation for other services. This is the recommended protocol for large applications.
Over the course of the next several years of FreePBX development, more agi scripts were added to FreePBX to do tasks that developers either didn’t want to do in Asterisk directly, or couldn’t get Asterisk to do with straight dialplan. This never caused serious issues in FreePBX until the last three years when Sangoma Support started to notice a significant impact to systems that were running for long periods of time without restarting Asterisk and had a high volume of calls. The main issue users experienced was dropped audio frames in conference rooms. This lead one of our lead engineers to open up a ticket on the Asterisk bug tracker.
With the help of the Asterisk team and our through own investigations we came to a few conclusions. First we setup a test system, against which we generated 250,000 AGI calls. This load would get Asterisk into a broken state. In this broken state the MoH will be extremely choppy, and conferences will also have dropped audio frames. After a restart of Asterisk, the MoH is no longer interrupted and conference bridges are fine.
Josh Colp of Sangoma provided more information to us:
If ConfBridge didn’t send a frame for 100 milliseconds then either every channel was blocked for that period of time (in which case you would see a flood of frames as it catches up) or the timerfd file descriptor didn’t wake up within 20ms as it was told to and instead woke up 100ms later and then resumed at a 20ms interval, or the transcoding process in the mixing loop took 100ms to execute.
Which allowed our engineer to narrow down the problem
Awesome! So, thanks to your input, it seems like we’ve narrowed down where the problem is. For some reason, that timerfd isn’t being woken up correctly when an AGI is launched.
Our engineer Rob was then able to create a few test cases by using timing test inside of Asterisk while generating the 250,000 AGI calls.
So timerfd, for whatever reason, took 5ms longer [than normal]. — Josh Colp
You’ll see that the ‘Active’ machine slipped twice. This, actually, confirms what I posted earlier — running a large number of AGI’s causes timing to slip — Rob Thomas
Rob then determined there was an issue with timerfd and Asterisk on CentOS machines.
It is an issue with the timerfd timing source freezing when asterisk uses the fork() system call to spawn an AGI. The workaround is to use dahdi timing, or, switch to using FastAGI, which does not require a fork(). — Rob Thomas
It was then determined internally that we needed to find a way to convert FreePBX modules into FastAGI. This was original no easy feat. However the code base for FreePBX references the same function call in php to create AGIs: ext_agi() what if we could use that centralized call to call fastAGIs instead:
Next we needed to create an AGI server. What we originally thought we should do was go and update all old AGIs to utilize the fastAGI server but in doing so we might break any non FreePBX Distro instances out there. Instead we came up with a way for FreePBX to be able to do a hybrid way of launching AGIs on the system and communicating with Asterisk.
The idea we had was to create a NodeJS daemon that merely does thin proxying back to Asterisk and launches AGIs just like Asterisk did. Then if this setting is enabled in FreePBX and the daemon is running to change all calls to ext_agi() to use fastagi() instead (with a call to the locally listening server).
The performance gains were nothing short of outstanding. Here is a comment from the customer who has on average 25 to 30 simultaneous calls.
The CPU still climbs from 200% to 400% depending of the lines in use (we average 25 to 30 simultaneous calls). The more lines are in use, the higher the CPU climbs and the sound gets worse.
Pre FreePBX FastAGI Asterisk at 25–30 simultaneous calls, notice the 212% CPU
Packet loss in FreePBX 14 before the implementation of the FastAGI Proxy
After our support team enabled the FastAGI server the customer sent back the following response:
I have monitored the system since last week’s update and everything seems to be working fine. We had a peak of 29 simultaneous calls this morning and the CPU did not go above 60%. At that time, I did a tcpdump and checked the stats in wireshark and I observed only 0.1% packet loss which is the best stat we had so far. At the present time, we have 24 active calls and the CPU is stable between 15% and 20% (some occasional peaks at around 60%). I think it is safe to say that the issue is resolved.
Post FreePBX FastAGI Proxy Implementation in FreePBX 14
You can enable the FastAGI server in open source FreePBX today. In FreePBX 14 it is disabled by default (but you can enable it). Once you enable it in Advanced Settings you will need to do an fwconsole reload & fwconsole restart. While in FreePBX 15 the FastAGI server is enabled by default. FreePBX 14 & 15 versions of the FastAGI server both require the PM2 module to maintain stability and uptime of the FastAGI server.
The advanced setting to enable the FastAGI Server
In addition to the work we performed with FastAGI another sticking point in our cloud environments was the dashboard scheduler cron that runs once a minute. The dashboard scheduler cron is used to display statistics from your system such as the one below.
PBXact is the commercial platform for FreePBX
You can also disable dashboard scheduler in Advanced settings
However for those systems that do not want to disable the collection of system statistics the resulting queries against MySQL can be staggering. As you can see below every minute there is a huge uptick in the amount of queries to hit the MySQL server every minute.
Insert operations per second
Closer view of queries per second of different clients
After doing some initial investigations it was determined that dashboard was keeping data for over 90 days and continually reprocessing that data. So if there was data from a month ago, dashboard would take the data from a month ago and reprocess it every minute. This means you’d be continually averaging the last three months, the last 12 weeks, and the last 90 days. As you can imagine there is no point in reprocessing data from the past because it never changes. Working through the code base we were able to tell the dashboard scheduler to only update and average our most current data groupings. This means that instead of processing data from months ago you are only processing the most recent hour, day, week and month. This results in a huge performance increase to the system and a huge decrease in the amount of SQL queries per second.
Can you tell from this screenshot below when we installed the updated dashboard module?
The new dashboard module is installed at the end of the graph around 13:10
Another look at the data pre and post upgrade of the dashboard module
Data over a wider range
The most impressive stats are on the hosts as a whole. The red lines below mark when our cloud staff did a massive grouping of dashboard upgrades. You can see the drastic results below.
We immediately go from about 1.2k Queries per second to about 0.8k
Another graph on a different host shows about 1.4 kps down to 0.9 kps
These changes are in the latest versions of the Dashboard module for both FreePBX 13, 14 and 15. Respectively 18.104.22.168 and 22.214.171.124.
Key Value Store (KVStore)
The next big issue we noticed in regards to performance was how FreePBX’s internal Key Value Store works. The Key Value Store was introduced in FreePBX 12 as a way to store data without maintaining tables or making direct database calls.
The way Key Value Store was designed originally was that every key was stored in the same database table. The use of this feature outgrew its design as we moved from 12 and into versions 13 and 14.. In FreePBX 14 we broke out of the “1 table to rule them all” design, and made it so that the Key Value Store engine would create a separate table for each module and a separate table for values greater than 4096 bytes to optimize performance.
But there were more steps we knew we could make. Starting in Framework 126.96.36.199 and Framework 188.8.131.52 the Key Value Store now implements in-memory caching. This means that multiple calls to the same ‘key’ data will not make multiple calls to the database. This will create huge performance gains in code that calls the same key more than once but does not cache it locally, decreasing the amount of SQL Select calls that need to be made. Additionally, this also has the result of speeding up Key Value methods that attempt to get data in other ways, such as getting all the keys for a single module. Since the values are loaded from MySQL at first query then the keys are already in memory.
The last major benefit is that you don’t need to worry about running out of RAM with this new caching system, as anything over 4096 bytes is not stored in RAM at startup. Anything over 4096 bytes is only loaded in RAM on request.
DialParties in Dialplan
Finally, the biggest change of them all. If you know anything about the inner workings of FreePBX you might have heard of an AGI file called “dialparties.agi”. This file is launched by Asterisk for any ringgroup call or any direct extension call that has Find me/Follow me enabled. It is not considered a resource intensive file, but you can imagine the issues this file causes in terms of FreePBX’s usage of AGIs, especially when you consider there are audio cut outs due to the previously discussed AGI forking issue.
Also, developers have long wanted a way to “splice” (add) dialplan directly into dialparties.agi. Unfortunately, there has never been a way to do this unless one completely forked out the Core module.
All of this changed in FreePBX 15. In FreePBX 15 we made the effort to turn dialparties.agi into straight Asterisk dial plan. This means that every FreePBX 15 system is using the new Dial Plan Dialparties. The dialplan for dialparties can be easily hooked into by any third party module and best of all Asterisk is no longer launching an AGI for about 80% of calls. This is enabled by an advanced setting, however, it is enabled by default so you should never need to disable it.
Dialparties.agi Dialplan setting
As we creep closer to FreePBX 15’s release (we’ll have a blog on the status of that soon) we are proud to be able to provide these open source performance improvements. We discovered, diagnosed, and fixed these issues in our own cloud infrastructure, and are pleased the efforts will be of benefit to the community as a whole. Sangoma is proud to be supporting the FreePBX project and we look forward to providing you with more updates as the year progresses.