When we started the development of FreePBX 15 around September 2017 we had high hopes for a release within six months. We announced that this release would include a redesigned backup module that would no longer require FreePBX restores to be from the same version you are restoring to (meaning in the past you could only restore 13 into 13, or 14 onto 14). However, software development never goes the way one plans for it to go.
There’s been a lot of work going on behind the scenes in getting FreePBX 15 to a stabilized release that we will explain below. If you haven’t seen our most recent blog be sure to check out: Performance Improvements in FreePBX. Additionally, during this time we were also able to release our highly anticipated Zulu 3.0 redesign and Queue Wallboard commercial modules.
Backup & Restore
Since backup’s inception on Nov 7, 2005 the main purpose of the module was to be able to backup FreePBX and restore it either on the same system or a brand new system. The way this was achieved for the past 14 years was to do a complete database dump of the FreePBX system and store that in a compressed backup file.
While doing a database dump is by far the easiest way to be able to backup a system, in the case of FreePBX it because increasingly more difficult to support restores from older FreePBX versions and sometimes even restores from the same FreePBX version. The reason being that FreePBX is a modular system and modules are constantly changing database table structures. If you load the database structure for a module from version 12 onto a module in version 15 the module won’t know what to do or how to migrate and you’ll be faced with errors that sometimes you can’t correct.
What FreePBX needed was the ability to utilize the modular system already provided to it and give the power of backing up and restoring to each module. In this model an individual module would assume responsibility for backing up and restoring all of its own data. FreePBX would no longer make global assumptions about modules.
There is one drawback to this idea. Namely that there are over 128 modules in FreePBX that need to be individually updated to support the new functionality the backup and restore module provided. Early on a decision was made that there would be no default fallback for unsupported modules because doing so would land you in the same boat as before the redesigned backup and restore.(Speaking of which, if you are a developer, supporting Backup and Restore is an easy process! Checkout the developer documentation for implementing Backup & Restore in your own modules)
Additionally, it was requested that each module should also be able to deal with restoring from legacy versions of FreePBX. This creates other technical issues in that we never wanted to load the FreePBX database dump back into our database because it would blow everything away. Our team came up with a unique solution to load the FreePBX MySQL database dump back into an in-memory SQLite database to use temporarily. Each individual module is then provided a handler to talk to the in memory database to query against it and then add the data it needs to the permanent database.
After many months of work and refinement our development team successfully updated 95% of the 128 modules and we are excited to be able to release the new Backup & Restore module for FreePBX 15 to the general public. We’ve also tested a few restores from FreePBX 14, 13 and yes, even 12 backup files and they all worked, however we know that not every FreePBX system is alike so we encourage you to run restores from older versions of FreePBX and report them to our bug tracker at http://issues.freepbx.org.
Additionally, because most of the data stored from each module are text strings, backup file sizes have been reduced significantly. The files can be so small that technically you could have FreePBX email you your nightly backup. (And we provide you with that ability in Filestore)
Backup & Restore has always had support for Local and FTP file storage locations. About four years ago a community member submitted an Amazon S3 patch and our own Bryan Walters committed that patch to Backup & Restore. The patch allowed users to store backup packages on Amazon S3. While this worked well for a while, the original code the patch was based on unfortunately hasn’t seen very many updates and thus was hard for the FreePBX development team to maintain.
When we started the planning for FreePBX 15 we wanted to make sure that the new backup module would be able to officially support Amazon S3 through a regularly maintained library. Then we found Flysystem (Spelled correctly!).
Flysystem is a filesystem abstraction library for PHP. By providing a unified interface for many different filesystems you’re able to swap out filesystems without application wide rewrites. Using Flysystem can eliminate vendor-lock in, reduce technical debt, and improve the testability of your code.
Enter the Filestore module. A centralized place in FreePBX to manage remote or local file storage locations. Right out of the gate FreePBX File Store supports Dropbox and Amazon S3 and with demand from the community we could potentially support any other service that is a supported adaptor from Flysystem.
The best part about Flysystem is that it gives FreePBX the flexibility in the future to be able to store items like recordings, faxes and much more away from your system and into the cloud.
Somewhere along the road of FreePBX 15 we also decided to be ambitious in adding an official API based on GraphQL with the ability to use a normal RESTFul API as well. In doing so we added the ability for FreePBX to become an OAUTH 2.0 server for API authentication.
Adding GraphQL or RESTFul API calls to 128 modules is not an easy task. At this moment we have about 10 modules completed. Since the FreePBX API is mainly a developer resource we decided over time that we would add features and functionality as developers contribute or request them.
If you’d like to upgrade your 14 system to BETA 15, simply go to Module Administration and check online for PBX Upgrader.
If you are the type of person that enjoys using Ansible then you’ll love that we added the ability to perform upgrades using PBX Upgrader without having to go to the GUI at all.
With the release of the Sangoma 7 Distro almost 3 years ago we are finally able to provide a modern PHP version. When we started the process of bringing FreePBX to a modern supported version of PHP we had to first go to the last release of the PHP 5.x series. Both FreePBX 14 and 15 only support PHP 5.6.x. However with the release of “FreePBX 16”, FreePBX will finally have a minimum supported PHP version of 7.3. Commercial modules will also work with FreePBX 16 without the need or use of Zend. Our team is starting on this work immediately and we hope to have something to show you in the next six months.
FreePBX 13 Support
Once we have our releases of High Availability for FreePBX 14 & 15 into stable we will be sunsetting our support for FreePBX 13, so if you haven’t already start making plans to upgrade to 14 or make a backup from your 13 system and restore that onto a new 15 installation.
In the next few months you’ll be hearing from me again as we release High Availability 14 and announce our stable release of FreePBX 15.
It’s been about 6 months since the engineering team last posted a blog on FreePBX and let me tell you, it’s been a busy 6 months. Last time we posted was shortly before the Digium/Sangoma merger. You’ll be happy to know that we still have a great team working on FreePBX as we did before the merger. Best of all, now that Digium and Sangoma are one team we are able to work on complex issues together that end up benefiting the open source community as a whole. Performance in FreePBX has always been a complex, multifaceted issue.
Performance problems can stem from many issues such as individual hardware, bad hard disks, corrupt ram, loading too many virtual machines on a single host. Our support team deals with so many different types of issues that it can be hard to track down the root cause at times. However there are also performance improvements that can come directly from FreePBX’s internal open source code. We will document four of these changes in this post.
Usage of AGI
The Asterisk Gateway Interface, also known as AGI, is a language-independent API for call processing. AGIs allow external scripts to manipulate Asterisk which lets Asterisk perform tasks that would otherwise be difficult or impossible.
In the world of AGIs, there are two types of AGIs actively in use today in Asterisk: AGI() and FastAGI(). Both have been in existence in Asterisk since around version 1.0. In the beginning days of FreePBX all it required was a LAMP stack; Linux, Asterisk, MySQL and PHP. To utilize FastAGI at the time meant you’d have to run a daemon that continually runs in the background listening on TCP port 4573. Additionally you’d need to effectively load all AGIs into this server at run time. Back in 2007 this was a huge mountain to climb so instead the developers of FreePBX (then known as Asterisk Management Portal) decided to use regular AGI and so was born dialparties.agi, one of the first agi php scripts to be used in FreePBX. Standard AGI is the simplest, and most widely used form of AGI. Standard AGI scripts run on the local PBX and communicate with Asterisk through socket descriptors (namely STDIN and STDOUT). The standard AGI allows for usage of all AGI commands, and is what this article will be discussing.
Fast AGI is the AGI over TCP sockets protocol. It allows for all AGI functionality except EAGI, and is provided as a solution to developers who need to run resource intensive AGI programs. By running the bulk of the AGI logic on another server, the Asterisk server itself can process calls and not worry about handling complex computation for other services. This is the recommended protocol for large applications.
Over the course of the next several years of FreePBX development, more agi scripts were added to FreePBX to do tasks that developers either didn’t want to do in Asterisk directly, or couldn’t get Asterisk to do with straight dialplan. This never caused serious issues in FreePBX until the last three years when Sangoma Support started to notice a significant impact to systems that were running for long periods of time without restarting Asterisk and had a high volume of calls. The main issue users experienced was dropped audio frames in conference rooms. This lead one of our lead engineers to open up a ticket on the Asterisk bug tracker.
With the help of the Asterisk team and our through own investigations we came to a few conclusions. First we setup a test system, against which we generated 250,000 AGI calls. This load would get Asterisk into a broken state. In this broken state the MoH will be extremely choppy, and conferences will also have dropped audio frames. After a restart of Asterisk, the MoH is no longer interrupted and conference bridges are fine.
Josh Colp of Sangoma provided more information to us:
If ConfBridge didn’t send a frame for 100 milliseconds then either every channel was blocked for that period of time (in which case you would see a flood of frames as it catches up) or the timerfd file descriptor didn’t wake up within 20ms as it was told to and instead woke up 100ms later and then resumed at a 20ms interval, or the transcoding process in the mixing loop took 100ms to execute.
Which allowed our engineer to narrow down the problem
Awesome! So, thanks to your input, it seems like we’ve narrowed down where the problem is. For some reason, that timerfd isn’t being woken up correctly when an AGI is launched.
Our engineer Rob was then able to create a few test cases by using timing test inside of Asterisk while generating the 250,000 AGI calls.
So timerfd, for whatever reason, took 5ms longer [than normal]. — Josh Colp
You’ll see that the ‘Active’ machine slipped twice. This, actually, confirms what I posted earlier — running a large number of AGI’s causes timing to slip — Rob Thomas
Rob then determined there was an issue with timerfd and Asterisk on CentOS machines.
It is an issue with the timerfd timing source freezing when asterisk uses the fork() system call to spawn an AGI. The workaround is to use dahdi timing, or, switch to using FastAGI, which does not require a fork(). — Rob Thomas
It was then determined internally that we needed to find a way to convert FreePBX modules into FastAGI. This was original no easy feat. However the code base for FreePBX references the same function call in php to create AGIs: ext_agi() what if we could use that centralized call to call fastAGIs instead:
Next we needed to create an AGI server. What we originally thought we should do was go and update all old AGIs to utilize the fastAGI server but in doing so we might break any non FreePBX Distro instances out there. Instead we came up with a way for FreePBX to be able to do a hybrid way of launching AGIs on the system and communicating with Asterisk.
The idea we had was to create a NodeJS daemon that merely does thin proxying back to Asterisk and launches AGIs just like Asterisk did. Then if this setting is enabled in FreePBX and the daemon is running to change all calls to ext_agi() to use fastagi() instead (with a call to the locally listening server).
The performance gains were nothing short of outstanding. Here is a comment from the customer who has on average 25 to 30 simultaneous calls.
The CPU still climbs from 200% to 400% depending of the lines in use (we average 25 to 30 simultaneous calls). The more lines are in use, the higher the CPU climbs and the sound gets worse.
Pre FreePBX FastAGI Asterisk at 25–30 simultaneous calls, notice the 212% CPU
Packet loss in FreePBX 14 before the implementation of the FastAGI Proxy
After our support team enabled the FastAGI server the customer sent back the following response:
I have monitored the system since last week’s update and everything seems to be working fine. We had a peak of 29 simultaneous calls this morning and the CPU did not go above 60%. At that time, I did a tcpdump and checked the stats in wireshark and I observed only 0.1% packet loss which is the best stat we had so far. At the present time, we have 24 active calls and the CPU is stable between 15% and 20% (some occasional peaks at around 60%). I think it is safe to say that the issue is resolved.
Post FreePBX FastAGI Proxy Implementation in FreePBX 14
You can enable the FastAGI server in open source FreePBX today. In FreePBX 14 it is disabled by default (but you can enable it). Once you enable it in Advanced Settings you will need to do an fwconsole reload & fwconsole restart. While in FreePBX 15 the FastAGI server is enabled by default. FreePBX 14 & 15 versions of the FastAGI server both require the PM2 module to maintain stability and uptime of the FastAGI server.
The advanced setting to enable the FastAGI Server
In addition to the work we performed with FastAGI another sticking point in our cloud environments was the dashboard scheduler cron that runs once a minute. The dashboard scheduler cron is used to display statistics from your system such as the one below.
PBXact is the commercial platform for FreePBX
You can also disable dashboard scheduler in Advanced settings
However for those systems that do not want to disable the collection of system statistics the resulting queries against MySQL can be staggering. As you can see below every minute there is a huge uptick in the amount of queries to hit the MySQL server every minute.
Insert operations per second
Closer view of queries per second of different clients
After doing some initial investigations it was determined that dashboard was keeping data for over 90 days and continually reprocessing that data. So if there was data from a month ago, dashboard would take the data from a month ago and reprocess it every minute. This means you’d be continually averaging the last three months, the last 12 weeks, and the last 90 days. As you can imagine there is no point in reprocessing data from the past because it never changes. Working through the code base we were able to tell the dashboard scheduler to only update and average our most current data groupings. This means that instead of processing data from months ago you are only processing the most recent hour, day, week and month. This results in a huge performance increase to the system and a huge decrease in the amount of SQL queries per second.
Can you tell from this screenshot below when we installed the updated dashboard module?
The new dashboard module is installed at the end of the graph around 13:10
Another look at the data pre and post upgrade of the dashboard module
Data over a wider range
The most impressive stats are on the hosts as a whole. The red lines below mark when our cloud staff did a massive grouping of dashboard upgrades. You can see the drastic results below.
We immediately go from about 1.2k Queries per second to about 0.8k
Another graph on a different host shows about 1.4 kps down to 0.9 kps
These changes are in the latest versions of the Dashboard module for both FreePBX 13, 14 and 15. Respectively 18.104.22.168 and 22.214.171.124.
Key Value Store (KVStore)
The next big issue we noticed in regards to performance was how FreePBX’s internal Key Value Store works. The Key Value Store was introduced in FreePBX 12 as a way to store data without maintaining tables or making direct database calls.
The way Key Value Store was designed originally was that every key was stored in the same database table. The use of this feature outgrew its design as we moved from 12 and into versions 13 and 14.. In FreePBX 14 we broke out of the “1 table to rule them all” design, and made it so that the Key Value Store engine would create a separate table for each module and a separate table for values greater than 4096 bytes to optimize performance.
But there were more steps we knew we could make. Starting in Framework 126.96.36.199 and Framework 188.8.131.52 the Key Value Store now implements in-memory caching. This means that multiple calls to the same ‘key’ data will not make multiple calls to the database. This will create huge performance gains in code that calls the same key more than once but does not cache it locally, decreasing the amount of SQL Select calls that need to be made. Additionally, this also has the result of speeding up Key Value methods that attempt to get data in other ways, such as getting all the keys for a single module. Since the values are loaded from MySQL at first query then the keys are already in memory.
The last major benefit is that you don’t need to worry about running out of RAM with this new caching system, as anything over 4096 bytes is not stored in RAM at startup. Anything over 4096 bytes is only loaded in RAM on request.
DialParties in Dialplan
Finally, the biggest change of them all. If you know anything about the inner workings of FreePBX you might have heard of an AGI file called “dialparties.agi”. This file is launched by Asterisk for any ringgroup call or any direct extension call that has Find me/Follow me enabled. It is not considered a resource intensive file, but you can imagine the issues this file causes in terms of FreePBX’s usage of AGIs, especially when you consider there are audio cut outs due to the previously discussed AGI forking issue.
Also, developers have long wanted a way to “splice” (add) dialplan directly into dialparties.agi. Unfortunately, there has never been a way to do this unless one completely forked out the Core module.
All of this changed in FreePBX 15. In FreePBX 15 we made the effort to turn dialparties.agi into straight Asterisk dial plan. This means that every FreePBX 15 system is using the new Dial Plan Dialparties. The dialplan for dialparties can be easily hooked into by any third party module and best of all Asterisk is no longer launching an AGI for about 80% of calls. This is enabled by an advanced setting, however, it is enabled by default so you should never need to disable it.
Dialparties.agi Dialplan setting
As we creep closer to FreePBX 15’s release (we’ll have a blog on the status of that soon) we are proud to be able to provide these open source performance improvements. We discovered, diagnosed, and fixed these issues in our own cloud infrastructure, and are pleased the efforts will be of benefit to the community as a whole. Sangoma is proud to be supporting the FreePBX project and we look forward to providing you with more updates as the year progresses.
Typically, when vendors release the latest version of their product, it costs the customer slightly more than the previous year’s model because of the improvements made to make that product better. As customers, we may be inclined to give in because we want those extra features of the new model. This is evident in the automotive industry and with mobile phone vendors, for example.
On the other hand, some vendors will reduce the feature set of their new model to keep costs low. But what good is that if the customer gets less features in return for the same price? Especially when the price of technology goes down with time, vendors should be able to put that extra savings into delivering more value for the customer.
At Sangoma, we thrive on delivering value-based products, and we are doing it again with the new PBXact 1200 and FreePBX 1200 phone system appliances. The new appliances can support up to 1200 users and can handle up to 350 concurrent calls. They are replacements for our PBXact 1000 and FreePBX 1000 systems, which handle 1000 users and up to 300 concurrent calls. With the new appliances, all the end user features are the same as previous models, allowing you to get more value with the additional capacity. And we also include dual PSU with every appliance at no extra cost!
The new models are being introduced with the same price as the existing PBXact 1000 and FreePBX 1000 models of US$7295 and US$1595, respectively. So, while vendors may be upping their price tags or reducing their features to keep costs down, Sangoma is delivering more value to its customers, without taking more out of your pocket. You now get 200 more users per system, 50 more concurrent calls, dual PSU, and the latest hardware. Now, that’s true value.
The new PBXact 1200 and FreePBX 1200 appliances are available for purchase today, February 1, from our worldwide network of authorized partners. For more information on the new appliances, check out our website.
It’s not just that Fort Lauderdale, FL, is a rather nice place to go at this time of year… No, we have some amazing sessions for you at Asterisk World 2019 too! And it’s right around the corner: January 30 – February 1.
WE HAVE ADDED A WORKSHOP!
Preston McNair from the FreePBX project will be hosting a new workshop on Tuesday, January 29, called Go To Market Strategies for Asterisk and FreePBX. So, if you would like to improve your Asterisk/FreePBX-based business, explore further revenue opportunities, optimize your margins, etc., be sure to register NOW.
In addition to this exciting news, we have a good number of relevant sessions from a great collection of speakers over the main 3 days of the conference (Wednesday-Friday).
Subjects like AI, chatbots, contact center, scaling and securing Asterisk, looking at the HaaS model, IVR optimization, and winning government contracts with your offerings will all be addressed in 9 outstanding sessions.
If you are new to Asterisk, I will personally be running a collection of sessions just for you on Wednesday, January 30.
You really don’t want to miss this excellent opportunity to soak up some goodness from Sangoma/Digium and a very nice collection of great people all wanting to share their expert stories and experience!
And if that was not enough, we will be co-located with IT EXPO, so there will be plenty of other sessions to attend and exhibitors to visit so that you can make the most of your business in this exciting new year.
New Releases of Both Asterisk and FreePBX Software Announced
AstriCon, ORLANDO, Fla., October 9, 2018 – Sangoma Technologies Corporation (TSX VENTURE: STC), a trusted leader in value-based Unified Communications (UC) and UC as a Service (UCaaS) solutions and the world’s largest provider of open source communications solutions, today at the annual AstriCon users and developers conference, announced Asterisk 16 and FreePBX 15, the next major releases of the world’s two most popular open source communications projects.
“On behalf of all the employees of Sangoma and Digium, I would like to welcome the global community of open source communication developers to AstriCon,” said Bill Wignall, President and CEO of Sangoma. “It is clearly the premier event in the world today for this part of our industry, with attendance approaching 1,000 people, dozens of speaking sessions from amongst hundreds of submitted abstracts, and a full exhibitor floor. I’m very excited about this year’s event, and as we work hard at bringing our two companies together, I’d like to take this opportunity to assure the community of our commitment to these critical open source projects that mean so much to the communications world. I hope that today’s major releases of both Asterisk 16 and FreePBX 15 demonstrate that we are already actively delivering even more innovation that will further enhance the missions of the two projects and their value to the communities.”