Online Module Management: The Challenges of Serving Millions of Diverse Installations

Online Module Management

The Challenges of Serving Millions of Diverse Installations

FreePBX has grown in many ways over the years and that growth has been both vertically and horizontally. Vertically, there are millions of installations, many of which check back with our servers daily to get up-to-date modules and security notices. Horizontally there are multiple distributions, multiple PHP versions, and multiple Asterisk versions to account for. Some of these factors affect which modules and which module versions will be presented to a system as potentials for upgrades.

The online Module Admin system has been around since 2006, and for adventurous developers, all the tools have been and still are available to generate and create your own repository. In fact, several other projects over the years have done exactly this. In the past, the process was straight forward. The packaging tools we created would mark a module with a version number, check source code into SVN, create the tarball, calculate an MD5 hash of the tarball, and then check that back into SVN. To create an online repository, you could simply pull all the latest module.xml files, concatenate them together into a single XML file, put that up on your server and put the corresponding tarballs in the appropriate location and you were done.

Growing Demands

As the project grew, the demands from our customer base also grew and put strains on this process to the breaking point. We needed to find ways to:

  • Dynamically generate an XML manifest that could vary depending on the version of FreePBX, Asterisk, PHP and even the Distro you are running. This is needed to allow us to choose the proper version of a module, or not include a module that might break a given system.

  • We were asked to provide the ability to go backwards, offering up previous versions of modules so people could revert to older versions.

  • More security, a way to not only authenticate the tarball and its contents at the time of delivery, but to retain that manifest and regularly check the integrity of the installation to detect a hacked system that has had portions of modules modified or entirely new modules installed on the system that were not intended and possibly malicious(1).

  • An ability to transmit security notifications about known vulnerabilities and the required modules versions to patch them.

Migrating to Git

SUPThe online system was already strained at the time we migrated from SVN to Git two years ago. Furthermore, the migration implied that all of the tools would have to be re-written since much of the process was specific to SVN. As such, we started to attack this problem at the same time we made our move to Git.

The first step was to change how we would represent module branches in a Git repository. In the SVN era, each major release of FreePBX required the entire set of modules to be branched, even if the exact same module was used on multiple versions of FreePBX. This was largely done to support the online system and tools which required this structure and made it difficult to maintain multiple versions of the same module. With Git, we defined a new branching standard that would allow each module to contain only as many branches as were necessary for different versions of the module. The <supported> tag in the module.xml of a branch indicates the major versions of FreePBX that branch works with. Using this, the tools that interact with the Git repository can determine which branch to use on the target FreePBX version. The new structure also fits well with Git and has made it much easier for the community to contribute and collaborate on module development.

All of this is managed by the development tools, also available in Git, for developers to create and tag modules. By using the supplied management script, package.php, “minor branch tags” are inserted into the module’s repo delineating the minor branch points of a module. The script also statically checks for and tries to block errors such as PHP syntax violations or other integrity issues. Once properly packaged back into the Git repo, the tags can be used to find and pull out any specific version of a module for subsequent preparation into a consumable tarball for FreePBX. There is also a module.sig “packaging slip” that can be created by the Git devtools using sign.php. This is discussed in more detail below.  Furthermore, we created the ability for FreePBX to accept any name combination file during upload. This means not only does the old format of “module-1.1.1.tgz” work but even Github’s format of “master.zip” works since FreePBX now reads the module.xml to figure out which module it is, and if a URL is provided, will fetch it from there first.

Dynamic Manifest Generation

Along with moving to Git, we needed to find a new way to address the dynamic nature of module delivery and the differing requirements that were listed above. A single XML manifest per FreePBX version would not suffice. As such, we ended up rebuilding the entire backend of our delivery system and moving it from statically generated XML files to be completely database driven. When a system checks for module updates now, what they get back looks very similar to what they always got back, an XML manifest with information about the latest versions of the different modules that will work on their system. The creation of that manifest is now generated on the fly, at the time the request is made, upon receiving various information such as their versions of FreePBX, PHP and Asterisk as well as the Distro type being used. This mechanism allows us, for instance, to not offer up a module that would require PHP 5.4, to a system running PHP 5.3, which would otherwise result in their system crashing with a “white screen” because of the incompatibility. This same mechanism also allows us to determine previously available module versions to offer up, compile up a list of security vulnerabilities that we may want to inform the system about, and more.

Originally this system consisted of a single XML file, modules-<version>.xml containing a concatenated list of all the module.xml files. It was later enhanced to deliver a second file, security-<version>.xml which provided security CVE announcements that would allow FreePBX to inventory itself and determine if there were any known vulnerable modules that urgently needed to be updated. In version 12, with the new rollback and beta features we added two more XML files, namely old-<version>.xml and beta-<version>.xml. The proliferation of XML manifests began to make slower systems run very sluggishly simply to check for updates. We thus combined all 4 manifests into a single one, called all-<version>.xml resulting in significant performance gains on both the FreePBX client side as well as the server side distribution(2).

Module Security

SUPOur dynamic online system has been in place for over two years now and solved some aspects of the original problem. However, it didn’t solve all the evolving security concerns. There are many aspects of security that have been, and continue to be addressed within FreePBX as we take a 10 year old application and continue to “rebuild” it from the inside out with security, stability, usability and features in mind.

However, the very modular nature of FreePBX in conjunction with it being written in PHP, which is a scripted language, provides many opportunities for malicious attackers to compromise a system. This meant we had to come up with a solution that would not only allow you to authenticate a module being downloaded to your system, but also a way to continuously monitor that same system against malicious changes once loaded or a malicious module being installed undetected.

In the past, FreePBX did nothing more then check that the MD5 hash of the tarball matched what the online XML manifest said it should. If you loaded a module manually there was no way to authenticate it. We thus took the path that many other open source projects have followed and implemented GPG signing and verification into the process.

Module security starts when a developer wants to release a module. As part of the development tools (licensed under the GPL, and available via Git) we run sign.php which creates a simple manifest, module.sig, that is nothing more then a catalog of every file that is included in that module, along with a SHA1 hash for each of those files. This is effectively a “packaging list” that, once prepared, is signed by the developer’s trusted private key so its authenticity can be verified. All of the files from Git, for this tagged version, are then included and packaged into a compressed tarball that is generated for distribution so that FreePBX can ultimately authenticate the integrity of each included file.

Next, we take that tarball, which is effectively a “binary file” and add another GPG binary signature to it so that it can be verified before ever opening it up once delivered to a system for installation. At this stage there are two tarballs a TGZ and a GPG that are available for download.

To make an analogy, each tarball of source code (and other files) contains a packaging list in it. That packaging list, which is not code, is “sealed” so that if tampered with, a consumer can detect such. The entire package is then wrapped up for shipping, and on the outside of the package, we put a type of “tracking label,” just like FedEx might do. The recipient can then examine that label to determine if the package is authentic before even opening it. Once you open it, you can check the seal on the package list, just imagine that wax seal with the king’s stamp on it…, and then you can use the package list to confirm that the contents of the package are intact.

What does all of this do? After checking for updates and receiving the dynamically generated XML manifest of available modules, the user may now choose to update or install some or all modules presented to it. The manifest tells FreePBX where to get those updates, which are just the above mentioned signed TGZ or GPG tarballs. FreePBX then proceeds to download the GPG signed tarballs one at a time. The first part of the GPG validation involves obtaining the most recent public keys available from the online GPG Web of Trust. For systems that don’t have internet access and are loading modules locally, they will attempt to fall back to a manually distributed list of keys. Once the tarball is on the system, the key is used to authenticate the package. Only after the package is authenticated is the tarball extracted. This assures that we don’t extract the content of a tarball if it is signed by a key that can’t be authenticated.

Once authenticated and the package has been opened, we are ready to run the module’s install script to upgrade (or install) the module. The files are extracted and FreePBX is able to use the signed “packaging list” to verify each file against its SHA1 hash to make sure it’s exactly what was checked out from the publically available Git repository used to originally prepare it(3). This same “packaging list” is used to verify the integrity of every module on the system on a regular basis, to detect if files have been maliciously tampered with and alert the system if it detects such. If there is no packaging list or the signature can’t be verified, the system will also alert you which is how Trojan horses(1) can be detected.

Summary

The code required to generate the key components of this process is part of our “devtools” repository, and the documentation to use it is explained in the corresponding Wiki. We want, and encourage, people to extend and develop our code, which is why we license it under the GPL (and the AGPL). We make sure that all of it is mirrored to Github (with a ‘Fork This!’ button on the top of every page), and we try our hardest to make sure that everything is thoroughly documented.

When you look at the underlying workings of the FreePBX Module Admin code, or the XML manifest that gets returned, most of it has minimal change since 2006 when the first Online System was introduced. The addition of some security monitoring components, GPG signing, rollbacks and module beta release tracking are the more evident changes. That is quite a testament to the original designers and their very simple but elegant solution that has been able to stay largely intact.

The simplicity and elegance of the original XML format has made it easy for us to ensure that there aren’t any incompatibilities between distributions using the old way of generating an online module list and the new way, which also links in security alerts and the ability to roll back to previous versions along with beta releases.

However, the huge changes required to scale vertically with the volume of users and horizontally with the diverse range of versions and Distros currently putting demands on the system has been an immense undertaking and a tribute to the FreePBX developers who have continued to engineer a world class system while maintaining the legacy of compatibility and virtually no bumps to the installed base. The project has become more scalable, stable and secure as a result of these changes and innovations and the success of the project and diversity of the community who uses it is a result of this hard work.

Questions or comments? We’d love to hear from you: http://community.freepbx.org/t/online-module-management-the-challenges-of-serving-millions-of-diverse-installations/29832


 

    (1) When we first introduced module signing we were immediately alerted by several customers to a rapidly spreading virus on hacked systems. The particular exploit was not a FreePBX exploit in this case. These hacked systems had a new, unauthorized module installed on them that was named “Admin Dashboard.” Since FreePBX is a modular system, it allows new modules, user modules, etc., to be installed. This module was actually a “Trojan horse” which would allow unauthorized remote access into a FreePBX system. Prior to the signature checking, and without it, this module would not have been detected. With the new security measures put in place the customers were quickly notified of the unauthentic and suspect module. A few of them notified the project and we were able to take other measures in addition to the signature signing to inform the community as well as active measures to “search and destroy” in order to squash this exploit as quickly as reasonably possible.

    (2) Prior to combining the 4 XML manifests into a single all-<version>.xml, we were seeing some slower systems, such as Raspberry PIs, take 40 seconds to download and process the set of manifests. This was simply to get to the point of presenting what modules are available for update. The inefficiencies were accounted for at all stages of the process. Four separate requests to the server which means setting up and tearing down TCP/IP connections four times along with the round trip latency. Four instances of the server side parsing the requests, querying the database and generating the various manifests dynamically on the fly. But most notably, four sets of XML manifests that need to be parsed by the local machine for internal representation of which the XML parser is a fairly significant computational task. With these changes, this has been reduced down to 2 seconds or less in some instances.

    (3) The module.sig “packaging list” can and is used not only on open source modules but any FreePBX module, whether obfuscated, Zended, or any other format that FreePBX is able to consume.

 

FreePBX 13 BETA is Ready For Testing

FreePBX 13 BETA is Ready For Testing

The development team is excited to announce the FreePBX 13 BETA release is now OPEN and available for install via the FreePBX Distro, or upgrade your current Alpha system through Module Admin.

When you open up the new version the first impression will be a complete change in the look and feel. You’ll discover features such as scrollable dropdowns, search bars, wizards and re-designed navigation menus that make your day-to-day interactions easier and faster. What won’t be immediately obvious is the vast “internal plumbing” changes that result in benefits such as significantly faster reload times especially for large installs and the replacement of the deprecated PearDB library with the modern PDO for faster and more secure database access.

The team has been working hard for 9 solid months; with close to a dozen developers involved, the results are impressive. We’ll be introducing a “Version Upgrade” module in the coming weeks once we get the initial feedback that this Beta looks solid, which means we need your help now to provide us that feedback!

“Town Meeting” via Google Hangout

We’ll brief you in just a moment, but first, please mark your calendars for our premier in a series of interactive Hangout Meetings where we will tell you more about Version 13 and give you an chance to interact with us directly to get your questions answered!

When: Thursday, July 9th 4PM EST

Where: bit.ly/1GNCg4c

Understanding What’s New in FreePBX 13

This new version takes off on many of the technologies and experiences that were introduced in FreePBX 12 where an all new mobile friendly User Control Panel (UCP) was introduced based off of Twitter’s bootstrap framework along with a myriad of other enhancements spanning from Asterisk’s PJSIP support to HTML5 voicemail playback and recording to secure module signing.

The visual enhancements of 13 stick out right away. From the new grid layout of each module to the ease of use wizards:

FreePBX 13 Grid Layout

 FreePBX 13 Quick Create Extension

FreePBX 13 Wizard

The team has made changes based on your user feedback and improved UI design research. Past problems such as menu selections too long for your browser window have been addressed with scrollable menu bars and advanced users can navigate quickly with search box abilities:

FreePBX 13 ScrollFreePBX Search Boxes           

The list of navigation and usability improvements goes on; the best way to discover them is to get it up in front of you and start experiencing all the great things the team has been working on!

We also put in effort addressing some bigger “sub-systems” to further enhance FreePBX based on user demands and changing technologies with the evolution of Asterisk.

Bulk User Management has been a big request and has seen minimal support in the past. This version introduces the new Bulk Manager which is a pluggable module that allows you to do mass “spreadsheet” management of Users, Extensions, Contacts and DIDs, and designed to grow and scale with the evolution of FreePBX and the introduction of features from other modules that affect these.

 FreePBX 13 Bulk

Call Event Log (CEL) integration and reporting, the Asterisk evolution of CDRs, has been added to be able to view the detailed activity of events from calls on your system.

 FreePBX 13 CEL

 FreePBX 13 CEL

With FreePBX prevalent in over 220 countries, language localization has been a priority and version 12 already saw huge efforts in the improved localization abilities of FreePBX for different languages. This new version adds the next dimension of easy sound file management with simple online access to other languages. The last leg of this effort will be the System Recording module which will be re-written to enable the concept of multiple language versions of the same call recording and will be following right on the heals of this release. This means, for instance, you will be able to have a single IVR that can play in different languages with the exact same recording based on language selection decisions in the call flow.

 FreePBX 13 Localization

FreePBX has always had a very limited ability to work with it at the CLI level via various ‘amportal’ commands. This latest version has adopted the Symfony framework and began to offer much more extensive CLI access through the “fwconsole” command.

 

 FWConsole

There’s so much more to talk about that we can barely scratch the surface. The above are only highlights and the real efforts that have gone in can only be experienced by trying it out yourself. So we look forward to you helping us drive this forward to a release candidate and final release soon!

Downloads and BETA Availability

FreePBX 13 BETA is immediately available on the FreePBX 13 download page. The plan is once we move further in the BETA process we will release the .tgz file for users wishing to do localized builds on their own standalone systems. For the time being we are releasing only for the FreePBX Distro so as to control factors introduced by the OS or other unknown variables but any user can always checkout 13 from FreePBX git servers.

In order to help us release FreePBX 13 on schedule, users of the BETA release are encouraged to report any bugs, issues or errors at http://issues.freepbx.org.

For now we look forward to your feedback testing and helping us get 13 finalized! Without your contributions and support FreePBX wouldn’t be the most prevalent and open source PBX platform on the planet and for that we are deeply grateful!

Holistic FreePBX Security: Engineering Protection From the Big Bad Wolf

In the cloud-enabled, highly networked world of VoIP, security is one of, if not THE, most important facets of proper software engineering. The FreePBX Development team takes security seriously, and when vulnerabilities are discovered or reported, we have a fantastic history of responding and mitigating issues quickly. We aim not to just resolve issues as they are discovered but have a goal of continually developing with security in mind – this is paramount in today’s world with many Big Bad Wolves constantly trying to get into your PBX. 

Security is not just another bullet point item. You cannot just bolt it on at the end of the development process, or at the time of installation. You must consciously design security into your application or service from the very beginning, and make it a continuous part of the entire process from design through implementation to testing and release. To keep on top of this — and so that we implement the best security practices and methodologies as they evolve — we are continually revising and revisiting FreePBX code, from something that was written yesterday, to code that goes back more than 10 years! 

Sadly, in today’s world of Internet telephony you are just as likely (if not more so!) to have a system attacked and compromised from within your local LAN as you are from an external connection, so implementing a firewall and other perimeter protections, while vital in themselves, are no longer enough. 

There are some integrators that recommend you just do the equivalent of building your security out of a bundle of various different security packages – just like building your house out of sticks. Sure it might be quick and easy to do that, and yes – It’s true that the more obstacles you put in the way of a drive-by attacker,  the more likely it is they will move on to other lower hanging fruit. But this often has the unexpected and unwanted consequence (we have seen in far too many instances) that when this ‘House of Sticks’ – because it hasn’t been planned and designed from the ground up – makes their platforms too restrictive to the end users, or even removes functionality inadvertently! 

If user panels and other Unified Communications aspects of a system are too laborious to utilize then one of two things will happen – either people will not use them, or system administrators will start pulling sticks out! This has the potential to significantly weaken the security of the overall system in the process, as quite often there’s no obvious and documented reason WHY a random security policy was put in place. 

So, instead of building your security policies with a bundle of sticks methodology, we focus on building and improving the core of the FreePBX platform, brick by brick. We ensure that everything from the OS, Asterisk, and FreePBX, is maintained and monitored as one. In several cases we have released patches for the FreePBX Distro before the upstream maintainers have! This is why we believe that a strong foundation is central to a strong and secure house, or phone system. 

Security and system updates are built holistically, and are much more likely to survive attacks from the Big Bad Wolf, than systems built with straw and bundles of sticks. Even if the Big Bad Wolf huffs and puffs and blows away your bundle of sticks, we do our best to ensure that the core, made of bricks, is still going to be there, standing strong. 

One recent example of our engineering in security is the Signature Checking, or Module Signing, that we implemented in FreePBX 12 and onward.   

If you purchase a proprietary PBX platform, it’s shipped from the manufacturer pre-loaded with its own software and firmware. Not that you can typically interact with the code on those appliances, however since the return address on the package is from the manufacturer, you can reasonably trust that the software installed is authentic and unmodified. Even then, there have been well publicized cases of this not being true!  We wanted a way to allow users to independently confirm that the FreePBX software on their server hadn’t been changed by an additional, unexpected third party. 

Module Signing takes a two-pronged approach. The first avoids the most common issues on an insecure Internet – it validates that the packages you’re downloading and installing are the ones that you’re EXPECTING to download and install, by cryptographically signing the module packages (exactly the same, in fact, as Debian and Redhat do with DPKG and RPMS!) 

The second is a constant monitoring of the FreePBX files on your system. If a file is changed, the FreePBX machine sends you an email, and displays a warning on every page, as soon as it detects that something wrong is happening. 

Module Signing uses the standard, proven and well known, Web of Trust, as implemented by GPG. This is a decentralized trust model where there are many independent webs of trust, which any user (through their identity certificate or key) can be a part of. 

Module Signing also assists in ensuring integrity and authenticity, by anyone being able to answer the two important questions “Who published this module?” and “Was this module changed since it was published?” Both of these questions are able to be easily answered, by even an unskilled user, by simply looking at the publisher of the module and making sure that no security alerts are being displayed. 

We looked at what other Open Source projects were doing, and implemented Module Signing using the same methods that Redhat and Debian/Ubuntu are using, as these methods have been successfully proven to be secure. Coincidentally, almost immediately after we implemented and published Module Signing, we were alerted by the community to an active and in the wild vulnerability that was then quickly patched. 

Read more about Module Signing in the FreePBX Wiki

Next week we will discuss utilizing the FreePBX Distro, which not only gives you a quick method of installation of a FreePBX System, but also the OS and other dependencies while at the same time allowing you to keep the entire PBX platform updated. 

Preston McNair – on behalf of the Sangoma FreePBX Team

http://community.freepbx.org/t/holistic-freepbx-security-engineering-protection-from-the-big-bad-wolf