Farewell Andrew

Great Job Andrew!

It’s been several years since I’ve blogged here and I’m honored to have the privilege of wishing Andrew the best on his future endeavors! For those that don’t know me, I used to run FreePBX prior to Andrew taking the helm so it only seemed appropriate that I have the opportunity to wish Andrew the best and further reflect on where FreePBX has been and is going!

Andrew has had a wonderful journey through his involvement and influence on FreePBX and the Asterisk world. The marks that he’s made on the project will continue for years to come. He has had his hands on so many different aspects of the project that I couldn’t even begin to enumerate everything he’s touched! Luckily we have ‘git blame’… Since I haven’t been involved in the day to day goings of FreePBX I thought I’d touch on my views and reflections of the attributes of a successful Open Source project like FreePBX.

I’ll start with some minor reminiscing of my own. My involvement started at the end of 2005, before there was even a FreePBX as it was still called AMP (Asterisk Management Portal) and just prior to Rob Thomas taking over the project. The database schema at the time comprised of 8 tables if I recall correctly, sip, iax, zap and extensions, and a few others. In my case I had just left the corporate world looking for something more fun to do and wanting to get back into telecom. I hooked up with a good friend who had been playing around in this space and immersed myself in this new world.

Rob Thomas had been running the project but other life priorities were competing for his time so I ended up taking it over. This was a time during the FreePBX life that I equated to the 60’s. It was a wild and free time where any and every contribution thrown at the project was getting absorbed without any real structure or planning or release management. It was a great period of growth and recognition but at the time, with the popularity of Asterisk@Home FreePBX was taking off and there was a huge boom in the commercialization of and adoption of FreePBX and Asterisk.

Upon taking over, it was my goal to provide stabilization for the project while continuing to keep it a vibrant and creative environment. These are somewhat opposing forces but necessary to make sure it would come out as an innovative project that thousands of resellers could build dependable businesses and millions of installations could result. During my tenure I accepted the help of energetic and significant forces in the community, whether individuals who came and left or significant players such as Schmooze, with whom I eventually joined forces. The goal was a continued and vibrant creative base to make sure that FreePBX excelled and at the same time, a strong supporter who eventually became Sangoma after FreePBX’s path through Bandwidth.com and then Schmooze who merged with Sangoma.

One underlying force behind Open Source is the collective and ever shifting brain trust that the openness nurtures. It’s through this combination of the constant giving back to the project, and the continuous inflow of new ideas as significant contributors enter the project for their tenure before moving on. Just as this cycling of various community members fuels the innovation in a project, so does the life cycle of its various leaders. Rob Thomas had a charisma and openness to invite the huge influx of contributions during the project’s early days at a time when that’s what it needed. Although Rob could never say ‘no’ and it resulted in plenty of headaches that took years to undo, it was also absolutely the right thing to do. It gave momentum, ideas, innovation and fuel to a great project. I was able to take that momentum and steer it in a direction to give stability and confidence while continuing to allow innovation but at the same time, occasionally having to say no. That took it well into the Schmooze days when we began the transition to let a new wave of creative energy take FreePBX to the next level and move Andrew into the role to take it somewhere I never could have.

Andrew’s leadership has taken the project to new levels that would never have happened had I stayed at the helm. The work that Andrew did, as well as Rob who resurfaced after a 5 year sabbatical, and James, Jason, Kapil, Philip, Mohit, Franck and others is simply astounding. Andrew has led the project through total rewrites and architectural enhancements of everything from its underlying core to the server side infrastructure that supports FreePBX to the surrounding ecosystems such as Zulu and beyond.

I’m very excited about what is possible going forward. For the first time in over a decade we’ve brought the two underlying technologies of FreePBX and Asterisk under a common umbrella. This has been an evolution that was once ridden by old school grumpy “real men program in dialplan” … they don’t use GUIs. Fast forward 14 years and we have these two symbiotic technologies under one roof. I know all of us who have been close to the project, whether visibly active or silently watching, won’t be far in keeping tabs on what this recent marriage can do in taking FreePBX into its next leap into the future.

For now, please join me in giving a huge thanks to the contributions that Andrew has achieved in his leadership and wishing him well in his next endeavors. Thank you Andrew!

To read Andrew’s post check it out here: https://www.freepbx.org/an-open-source-journey/

-Philippe Lindheimer

To participate in the conversation about this post go here: https://community.freepbx.org/t/farewell-andrew/58812

Sangoma Reinforces its Commitment to Open Source and Addresses Recent Elastix News

This week, significant changes at Elastix were announced, including the involvement of 3CX and the removal of key Elastix versions for download. Since those announcements, many things have been written by many people, and this has left some folks wondering what happened. Sangoma would like to reinforce its commitment to open source, this open letter from Sangoma, will provide our own clarity about how these events affect or involve Sangoma. We are a professional, global, growing, profitable, engineering-focused, publicly traded company, and this is the only reliable source of information to understand how those recent events affect or involve our organization. Other commentary released by other third parties about Sangoma, is not to be relied upon.

There are three (3) parts to this message today, that we invite you read:

  1. Sangoma’s unwavering commitment to Open Source
  2. Sangoma’s history and involvement with recent Elastix news and events
  3. The opportunity that these recent events present to Elastix users to join the Sangoma family
1. Sangoma’s Unwavering Commitment to Open Source

Everyone comes to open source software for their own reasons: software developers to do what they love; some to earn a livelihood; manufacturers to augment the project and sell their wares; and most importantly community members to find flexible/cost effective/well-supported solutions to their ‘business problem’ (in our case, for UC/Telecom/PBX needs). In the end, the good projects build something bigger than themselves… a community, a solution, and an opportunity for end users to utilize the project to build their own businesses. Over the course of a project many people will enter and exit those communities as their needs change.

As the primary investor in and developers of FreePBX, Sangoma actively works with many different members of the Open Source Telephony (OST) community, including Asterisk Developers, other FreePBX-based distros (including Elastix!), and many third-party hardware/software developers and manufacturers. As just one example, we have a great relationship with Digium and talk with them on an almost weekly basis, even though many consider us competitors. This may seem surprising to some, as many folks would think we might be bitter enemies. In fact, the opposite is true…we encourage and help those products to compete in the marketplace on their own merits. And this is entirely consistent with the commitment Sangoma has demonstrated to open source for many, many years over the time when we worked hard to also make Asterisk better. When Sangoma took over stewardship of FreePBX, we reiterated this statement clearly and unequivocally.

So Sangoma continues to work very hard every day, and invests many millions of dollars each year, in order to build strong relationships and to benefit to the entire open source telephony community. There is a saying that ‘a rising tide lifts all boats.’ Thus, it is usually counter-productive for open source contributors to battle with each other. In other words, there is no reason for us to fight over the same slice of pie, when there is an entire cake that no one is touching.

2. Sangoma’s History with Elastix and Involvement in Recent Events There

Our approach was no different with Elastix. For over a decade, Sangoma has been a direct supporter of Elastix, in many, many different ways, visiting them in Ecuador many times. We supported the project financially, we attended/exhibited/supported/spoke at multiple ElastixWorld events over many years, we cooperated with our distribution partners who also supported Elastix, we invested in R&D to ensure our products (software and hardware) were compatible with Elastix, etc. The list goes on and on. We had (and we hope, still have), excellent relationships between our companies, in all parts of our organizations right up to the CEO level of both companies.

With recent changes at Elastix, some people/blogs/websites have made comments which claim that the removal of Elastix downloads of version 4 or MT, was in some way caused by Sangoma/FreePBX, due to concerns about compliance with GPL conditions. That is not true and we wish to set the story straight. We at Sangoma hold ourselves to high ethical standards, and as a publicly traded company as well, setting the record straight with facts and not rumors, is both important and required.

While it is indeed true that Sangoma pointed out to Elastix some time ago, that there was a copyright issue, we did so in a very friendly manner, with words carefully chosen to be respectful of the long term relationship between the companies, and critically, to ensure that this important relationship continued. It was a 2015 letter from CEO to CEO, and certainly did not suggest any legal action, since it was not that kind of letter at all…it was a positive, complementary letter seeking to deepen the relationship, not harm it. That letter was sent shortly after Sangoma acquired FreePBX, when we made it a priority to reach out to PaloSanto to reinforce that the Elastix Project was a valuable strategic partner to Sangoma. It was in no way threatening, did not ask for, was not intended to, and given it was 2015, did not cause any versions of Elastix to be withdrawn. Elastix decision this week to shutdown these versions is a business decision not a response to Sangoma. While it seems that these days, the number of open source projects that remain truly open source is definitely on the decline, Sangoma’s commitment to open source remains as true today, as always.

And while it is admittedly a little unusual for companies to do so, in this case, for full transparency to the open source communities that we respect so very much (and to dispel any untrue rumors or claims), the entire letter is available here. We share it for those who need confirmation of the above statements, and to reassure the Elastix community that Sangoma continues to be committed to you as well as to the entire Latin America region (and would be honored to have you consider joining our family)

Link to Full Letter

For the majority of you who may be less inclined to read the entire letter, we have pasted the text of the opening paragraph between our CEOs, so you can see for yourselves that it was truly a positive, relationship building letter:

“Hello Edgar, I haven’t seen you since my last visit to Ecuador. I hope you’re keeping well! When Sangoma closed our acquisition of FreePBX earlier this year I made it a priority for our staff to reach out immediately to inform you and reinforce that the Elastix Project is a valuable strategic partner of Sangoma. I reach out again today to renew that sentiment…”

The final word on this topic is to point out that Elastix could have resolved this modest copyright issue with a couple of hours of work, and could of course, still do so today. In fact, Sangoma offered our assistance to PaloSanto to perform that work, and we remain most prepared to honor that commitment today, if Elastix wished to make version 4 or MT available, with no problem whatsoever.

3. The Opportunity that These Recent Events Present to Elastix Users, to Join the Sangoma Family

The net result of recent changes at Elastix, is that the key versions of those open source projects, and the years of community involvement, have apparently been rebranded to the proprietary/closed 3CX platform. This sounds similar to the events surrounding another open source project last month (PBX in a Flash).

We at Sangoma have, unsurprisingly, heard from many in the Elastix community, saying that they feel confused, or abandoned, or are scrambling to figure out a next step.

And so we have an opportunity to extend to the valued members of that Elastix community, a chance to explore whether Sangoma and FreePBX may suit your needs. We are very proud of our proven commitment to open source, of our demonstrated investments in Latin America for many years, and of our FreePBX family of products. We invite you to discover what we have to offer by visiting our website or by giving us a call at +1 905 474 1990.

While you might be considering this option, we would like to share a few reasons that we believe you might find FreePBX to be a compelling alternative, including an elegant/one-step “import utility” that we have custom created in the last few days specifically to help Elastix users move their systems to FreePBX in a single step!!

  • Since many Elastix users are already familiar with the FreePBX components of Elastix, many of you will feel right at home when configuring a ‘pure’ FreePBX system.
  • You may not realize that FreePBX is a truly global phenomenon, with full translation of the GUI and ALL sound files in over 8 languages, INCLUDING SPANISH of course!
  • We have made it almost entirely painless for you to import your existing Elastix deployments to FreePBX, in one simple step with our new tool: “Come Home to FreePBX”. This will be available in the next couple of days. Please join us on Wednesday, December 14th for a webinar outlining how to utilize the #Home2FreePBX Conversion Tool.
  • We will also be offering special training, ‘get started as a reseller’ kits, and discounts for existing Elastix Resellers to help them make the transition to FreePBX. See form at bottom of blog.
  • With all of the hard work we’ve been doing the last couple of years, you will see some major functionality that was not available in the Elastix version. Over the past twelve months, 53 developers contributed new code to FreePBX, and FreePBX contributor activity is in the top 2% of all projects monitored by Open Source Monitor OpenHub.com. Since Jan of 2015 when Sangoma became the project steward, FreePBX has added over 480,000 lines of code into the platform based on OpenHub.com stats. FreePBX has been more actively developed the past 2 years than at any time in its 12-year history.
  • This year Sangoma introduced its own line of advanced IP-phones, designed specifically for FreePBX. They are available in a range of models from entry-level to executive, and we think you might be surprised how cost effective they are in Latin America. We have sold tens of thousands in just the first few months, so they are receiving a warm reception around the globe. These phones complete a ‘full solution’ available from Sangoma, so you can get your PBX, your phones, and your ‘connectivity products’ (gateways, cards, SBCs, etc) all from a single vendor and be guaranteed it will all work together. And take advantage of Zero-Touch Provisioning!
  • Many other companies such as Digium, Grandstream, Yealink, and Snom have also joined us in building a better Phone system full solution, by having their products certified for the FreePBX platform. By continuing to add FreePBX EcoSystem partners that contribute engineering resources, hardware, and expertise we continue to expand the capabilities and Freedom provided by FreePBX.
  • FreePBX offers a truly unlimited, free distro that provides the business and calling features you demand in a modern Unified Communications platform, including functionality such as:
    • Support for video calling
    • Announcements, Text to Speech
    • Flexible time based call routing
    • Fax to Email
    • IVR- Interactive Voice Response Menu
    • Basic Call Center Functionality – ACD and Call Queues, Hunt / Ring Groups
    • Three Way Calling, Caller ID, Call Transfer Call Recording, Do Not Disturb, Call Waiting, System wide Speed Dials, Call Screening
    • Secure communications
    • Built in Conference Bridge
    • Customizable Music on Hold
    • Voicemail – Voicemail to email
    • Follow Me / Find Me Calling
    • Call History – Call Detail Records and Call Event Logging
    • Paging / Intercom
    • Call Parking
    • Integrated Call Recording
  • Expanding ease-of-deployment, system administration and ongoing maintenance.
    • Choose your default language at time of installation, and your system GUI and sound prompts will be set with no additional configuration needed, including of course full translation into Spanish.
    • Upgrade system (OS, FreePBX, Asterisk and dependencies) with granular control, including the ability to roll back upgrades, and switch to supported versions of Asterisk on the fly.
    • New system dashboards and bulk import utilities (trunks, extensions, users, DIDs)
    • New User Management controls, and a responsive web based User Control Panel for End Users.
  • The current stable release of FreePBX 13, has support for Asterisk LTS releases of 11 & 13 and Asterisk 14, including support for the PJSIP channel driver. Our beta release of FreePBX 14 is compatible with Asterisk 11, 13 & 14, and completely supports the Opus Codec, for high quality, low bandwidth audio.

  • We will also be doing some special offers soon for existing Elastix Resellers to help them make the transition to a supported platform. Fill in the form below to be contacted about the offers.

    Commercial Modules, Support Provided, Upcoming Changes

    Commercial Modules, Support Provided, Upcoming Changes

    FreePBX has grown from dozens of modules a few years ago to hundreds of modules today (it’s over 100) the bulk of them part of the open source foundation that makes up this great project. We’re constantly adding new functionality, whether new features to existing modules, brand new open source modules, or the occasional introduction of a new commercial module within the mix.

    What determines Commercial vs. Open Source for New Module Development?

    Although most of our development resource are spent maintaining and building upon the core open source project, we always evaluate and think hard if a new module should be part of the open source foundation or introduced as commercial. The open source usually wins :). However, there are a number of specific or vertical market needs that usually come with much higher maintenance costs or the need for a higher then normal commercially acceptable response time in addressing issues. Examples of these are the High Availability Module and End Point Manager. In order to keep the bulk of our development efforts focused on the open source foundation, we evaluate these ‘special needs’ areas when deciding if a module should be commercial to ensure there is a proper revenue stream to create, maintain, and support these specialty components. The alternative would otherwise result in all of our time servicing these, used by a much smaller population of users, at the expense of the mainstream code base used by everyone.

    Commercial Module Licensing and Continued Support

    Most commercial modules are sold with a 25-year license and one year of updates for bug fixes and new features. This model is very standard in the software industry. It means your module will continue to run beyond the first year, but if you want to reap the benefits of the ongoing development and resulting enhancements you will have to pay a small fee after that to keep getting those updates. Otherwise, you can continue using what you have, it WILL continue to function fine.

    Up until now, we have never provided a mechanism to buy that continued support after the first year, despite many customers asking us how they can purchase it. Instead, we’ve simply continued to give you those updates for free. With commercial modules entering into their fifth year, the population has exploded with well over 100,000 such modules out there. The support requests related to this growth has increased to the point where we must start providing you the ability to purchase a renewal contract if you want to continue receiving updates and support for them. The alternative of continuing to give this away free would result in our resources being taken away from the core project, which simply hurts everyone and the project as a whole!

    What does this change mean and how will it impact you?

    For those of you with modules that are more then one year old meaning the year of updates has already expired, we will give you another two weeks free, through November 16th, where you can update any of those modules as has been the case since you purchased them. We’ll attempt to email you with this information proactively based on your portal contact information. (What … doesn’t everyone just follow our blog that we have to email? :)) Furthermore, we’ll provide you with a 15% discount incentive to purchase renewals in the next two week if you renew from this portal link and use discount code RENEWALS through November 15th. For any such renewals, your new anniversary date will be based on a November 16th expiration. This means, if you purchase a one-year renewal whether before or after November 16th, that renewal will be good through November 16th, 2016.

    If you’ve purchased a module in the last 12 months you’ll continue to get updates until the anniversary date of your purchase. You’ll be notified in advance inside of Module Admin when you have modules that soon need renewals where you can make the purchases directly. You can also navigate to the portal to see a comprehensive view of all your module licenses on all your deployments, when they will need renewals, and have the ability to renew any of them there whether they have already expired or you just recent purchased them. Furthermore, the RENEWALS incentive is not only good for already expired renewals, you can be proactive and take advantage of it now even if you just bought a module yesterday! (You must use the portal to take advantage of the 15% promotion, from within FreePBX it will not be available.)

    After November 16th, if you choose not to renew and there are updates available for those modules installed on your system, you will still be notified of the updates. You will have the option of purchasing a renewal in order to obtain them, which you can do right from the GUI.

    Keeping FreePBX thriving for everyone

    The FreePBX team is intensely dedicated to the ongoing success of this project and works tirelessly to make sure you continue to have a world class, feature rich and future proof system to bank your business and your customers’ businesses on. We are grateful to the community members who participate in the project – feeding ideas, code and project help to keep us moving forward. This change being announced will go a long way in assuring we can continue to do that. Whether you’re a consumer of these specialized modules or not, all of you depend on the continued investment in the core project’s long term viability. Enabling our ability to fund the ongoing support of these modules through renewals that have always been part of the licensing terms in our EULA will assure that we can continue providing the resources needed for all sides of the project and benefit everyone!

    Renewing from FreePBX

    When in FreePBX, if Renewals are coming up within 3 months, you can purchase them directly from the GUI. You will see Renewals that are available as seen in the following image.



    Upon clicking the Renewal button it will be added to your cart which you can view and go through the checkout process as shown:



    When purchasing from portal, you have the option of renewing modules from multiple deployments in once checkout process if needed, AND you can take advantage of the RENEWALS discount code:


    You are now set to continue receiving all the new enhancements (and bug fixes) that we continue to provide to you in these important modules that you depend on!

    Online Module Management: The Challenges of Serving Millions of Diverse Installations

    Online Module Management

    The Challenges of Serving Millions of Diverse Installations

    FreePBX has grown in many ways over the years and that growth has been both vertically and horizontally. Vertically, there are millions of installations, many of which check back with our servers daily to get up-to-date modules and security notices. Horizontally there are multiple distributions, multiple PHP versions, and multiple Asterisk versions to account for. Some of these factors affect which modules and which module versions will be presented to a system as potentials for upgrades.

    The online Module Admin system has been around since 2006, and for adventurous developers, all the tools have been and still are available to generate and create your own repository. In fact, several other projects over the years have done exactly this. In the past, the process was straight forward. The packaging tools we created would mark a module with a version number, check source code into SVN, create the tarball, calculate an MD5 hash of the tarball, and then check that back into SVN. To create an online repository, you could simply pull all the latest module.xml files, concatenate them together into a single XML file, put that up on your server and put the corresponding tarballs in the appropriate location and you were done.

    Growing Demands

    As the project grew, the demands from our customer base also grew and put strains on this process to the breaking point. We needed to find ways to:

    • Dynamically generate an XML manifest that could vary depending on the version of FreePBX, Asterisk, PHP and even the Distro you are running. This is needed to allow us to choose the proper version of a module, or not include a module that might break a given system.

    • We were asked to provide the ability to go backwards, offering up previous versions of modules so people could revert to older versions.

    • More security, a way to not only authenticate the tarball and its contents at the time of delivery, but to retain that manifest and regularly check the integrity of the installation to detect a hacked system that has had portions of modules modified or entirely new modules installed on the system that were not intended and possibly malicious(1).

    • An ability to transmit security notifications about known vulnerabilities and the required modules versions to patch them.

    Migrating to Git

    SUPThe online system was already strained at the time we migrated from SVN to Git two years ago. Furthermore, the migration implied that all of the tools would have to be re-written since much of the process was specific to SVN. As such, we started to attack this problem at the same time we made our move to Git.

    The first step was to change how we would represent module branches in a Git repository. In the SVN era, each major release of FreePBX required the entire set of modules to be branched, even if the exact same module was used on multiple versions of FreePBX. This was largely done to support the online system and tools which required this structure and made it difficult to maintain multiple versions of the same module. With Git, we defined a new branching standard that would allow each module to contain only as many branches as were necessary for different versions of the module. The <supported> tag in the module.xml of a branch indicates the major versions of FreePBX that branch works with. Using this, the tools that interact with the Git repository can determine which branch to use on the target FreePBX version. The new structure also fits well with Git and has made it much easier for the community to contribute and collaborate on module development.

    All of this is managed by the development tools, also available in Git, for developers to create and tag modules. By using the supplied management script, package.php, “minor branch tags” are inserted into the module’s repo delineating the minor branch points of a module. The script also statically checks for and tries to block errors such as PHP syntax violations or other integrity issues. Once properly packaged back into the Git repo, the tags can be used to find and pull out any specific version of a module for subsequent preparation into a consumable tarball for FreePBX. There is also a module.sig “packaging slip” that can be created by the Git devtools using sign.php. This is discussed in more detail below.  Furthermore, we created the ability for FreePBX to accept any name combination file during upload. This means not only does the old format of “module-1.1.1.tgz” work but even Github’s format of “master.zip” works since FreePBX now reads the module.xml to figure out which module it is, and if a URL is provided, will fetch it from there first.

    Dynamic Manifest Generation

    Along with moving to Git, we needed to find a new way to address the dynamic nature of module delivery and the differing requirements that were listed above. A single XML manifest per FreePBX version would not suffice. As such, we ended up rebuilding the entire backend of our delivery system and moving it from statically generated XML files to be completely database driven. When a system checks for module updates now, what they get back looks very similar to what they always got back, an XML manifest with information about the latest versions of the different modules that will work on their system. The creation of that manifest is now generated on the fly, at the time the request is made, upon receiving various information such as their versions of FreePBX, PHP and Asterisk as well as the Distro type being used. This mechanism allows us, for instance, to not offer up a module that would require PHP 5.4, to a system running PHP 5.3, which would otherwise result in their system crashing with a “white screen” because of the incompatibility. This same mechanism also allows us to determine previously available module versions to offer up, compile up a list of security vulnerabilities that we may want to inform the system about, and more.

    Originally this system consisted of a single XML file, modules-<version>.xml containing a concatenated list of all the module.xml files. It was later enhanced to deliver a second file, security-<version>.xml which provided security CVE announcements that would allow FreePBX to inventory itself and determine if there were any known vulnerable modules that urgently needed to be updated. In version 12, with the new rollback and beta features we added two more XML files, namely old-<version>.xml and beta-<version>.xml. The proliferation of XML manifests began to make slower systems run very sluggishly simply to check for updates. We thus combined all 4 manifests into a single one, called all-<version>.xml resulting in significant performance gains on both the FreePBX client side as well as the server side distribution(2).

    Module Security

    SUPOur dynamic online system has been in place for over two years now and solved some aspects of the original problem. However, it didn’t solve all the evolving security concerns. There are many aspects of security that have been, and continue to be addressed within FreePBX as we take a 10 year old application and continue to “rebuild” it from the inside out with security, stability, usability and features in mind.

    However, the very modular nature of FreePBX in conjunction with it being written in PHP, which is a scripted language, provides many opportunities for malicious attackers to compromise a system. This meant we had to come up with a solution that would not only allow you to authenticate a module being downloaded to your system, but also a way to continuously monitor that same system against malicious changes once loaded or a malicious module being installed undetected.

    In the past, FreePBX did nothing more then check that the MD5 hash of the tarball matched what the online XML manifest said it should. If you loaded a module manually there was no way to authenticate it. We thus took the path that many other open source projects have followed and implemented GPG signing and verification into the process.

    Module security starts when a developer wants to release a module. As part of the development tools (licensed under the GPL, and available via Git) we run sign.php which creates a simple manifest, module.sig, that is nothing more then a catalog of every file that is included in that module, along with a SHA1 hash for each of those files. This is effectively a “packaging list” that, once prepared, is signed by the developer’s trusted private key so its authenticity can be verified. All of the files from Git, for this tagged version, are then included and packaged into a compressed tarball that is generated for distribution so that FreePBX can ultimately authenticate the integrity of each included file.

    Next, we take that tarball, which is effectively a “binary file” and add another GPG binary signature to it so that it can be verified before ever opening it up once delivered to a system for installation. At this stage there are two tarballs a TGZ and a GPG that are available for download.

    To make an analogy, each tarball of source code (and other files) contains a packaging list in it. That packaging list, which is not code, is “sealed” so that if tampered with, a consumer can detect such. The entire package is then wrapped up for shipping, and on the outside of the package, we put a type of “tracking label,” just like FedEx might do. The recipient can then examine that label to determine if the package is authentic before even opening it. Once you open it, you can check the seal on the package list, just imagine that wax seal with the king’s stamp on it…, and then you can use the package list to confirm that the contents of the package are intact.

    What does all of this do? After checking for updates and receiving the dynamically generated XML manifest of available modules, the user may now choose to update or install some or all modules presented to it. The manifest tells FreePBX where to get those updates, which are just the above mentioned signed TGZ or GPG tarballs. FreePBX then proceeds to download the GPG signed tarballs one at a time. The first part of the GPG validation involves obtaining the most recent public keys available from the online GPG Web of Trust. For systems that don’t have internet access and are loading modules locally, they will attempt to fall back to a manually distributed list of keys. Once the tarball is on the system, the key is used to authenticate the package. Only after the package is authenticated is the tarball extracted. This assures that we don’t extract the content of a tarball if it is signed by a key that can’t be authenticated.

    Once authenticated and the package has been opened, we are ready to run the module’s install script to upgrade (or install) the module. The files are extracted and FreePBX is able to use the signed “packaging list” to verify each file against its SHA1 hash to make sure it’s exactly what was checked out from the publically available Git repository used to originally prepare it(3). This same “packaging list” is used to verify the integrity of every module on the system on a regular basis, to detect if files have been maliciously tampered with and alert the system if it detects such. If there is no packaging list or the signature can’t be verified, the system will also alert you which is how Trojan horses(1) can be detected.


    The code required to generate the key components of this process is part of our “devtools” repository, and the documentation to use it is explained in the corresponding Wiki. We want, and encourage, people to extend and develop our code, which is why we license it under the GPL (and the AGPL). We make sure that all of it is mirrored to Github (with a ‘Fork This!’ button on the top of every page), and we try our hardest to make sure that everything is thoroughly documented.

    When you look at the underlying workings of the FreePBX Module Admin code, or the XML manifest that gets returned, most of it has minimal change since 2006 when the first Online System was introduced. The addition of some security monitoring components, GPG signing, rollbacks and module beta release tracking are the more evident changes. That is quite a testament to the original designers and their very simple but elegant solution that has been able to stay largely intact.

    The simplicity and elegance of the original XML format has made it easy for us to ensure that there aren’t any incompatibilities between distributions using the old way of generating an online module list and the new way, which also links in security alerts and the ability to roll back to previous versions along with beta releases.

    However, the huge changes required to scale vertically with the volume of users and horizontally with the diverse range of versions and Distros currently putting demands on the system has been an immense undertaking and a tribute to the FreePBX developers who have continued to engineer a world class system while maintaining the legacy of compatibility and virtually no bumps to the installed base. The project has become more scalable, stable and secure as a result of these changes and innovations and the success of the project and diversity of the community who uses it is a result of this hard work.

    Questions or comments? We’d love to hear from you: http://community.freepbx.org/t/online-module-management-the-challenges-of-serving-millions-of-diverse-installations/29832


        (1) When we first introduced module signing we were immediately alerted by several customers to a rapidly spreading virus on hacked systems. The particular exploit was not a FreePBX exploit in this case. These hacked systems had a new, unauthorized module installed on them that was named “Admin Dashboard.” Since FreePBX is a modular system, it allows new modules, user modules, etc., to be installed. This module was actually a “Trojan horse” which would allow unauthorized remote access into a FreePBX system. Prior to the signature checking, and without it, this module would not have been detected. With the new security measures put in place the customers were quickly notified of the unauthentic and suspect module. A few of them notified the project and we were able to take other measures in addition to the signature signing to inform the community as well as active measures to “search and destroy” in order to squash this exploit as quickly as reasonably possible.

        (2) Prior to combining the 4 XML manifests into a single all-<version>.xml, we were seeing some slower systems, such as Raspberry PIs, take 40 seconds to download and process the set of manifests. This was simply to get to the point of presenting what modules are available for update. The inefficiencies were accounted for at all stages of the process. Four separate requests to the server which means setting up and tearing down TCP/IP connections four times along with the round trip latency. Four instances of the server side parsing the requests, querying the database and generating the various manifests dynamically on the fly. But most notably, four sets of XML manifests that need to be parsed by the local machine for internal representation of which the XML parser is a fairly significant computational task. With these changes, this has been reduced down to 2 seconds or less in some instances.

        (3) The module.sig “packaging list” can and is used not only on open source modules but any FreePBX module, whether obfuscated, Zended, or any other format that FreePBX is able to consume.


    Security Vulnerability Notice

    FreePBX UpdatesWe are blogging to inform you of a recently discovered security vulnerability reported yesterday in FreePBX Ticket 7123 (originally reported in ticket 7117 which is locked because of sensitive information). All FreePBX versions FreePBX 2.9 before, FreePBX 2.10 before, FreePBX 2.11 before, and FreePBX 12 before 12.0.1alpha22 are affected. You should immediately update your FreePBX Framework Module to secure your system from a potential attack.

    The vulnerability is a Remote Command Execution (RCE) attack which means a compromised PBX can have any system command executed on it. This may result in serious damage to the system or extraction of passwords and other credentials accessible to the apache user (which is asterisk on most systems).

    There were no known exploits at the time this issue was reported but upon going public it is common for hackers to develop scripts to scan the internet in an attempt to find vulnerable systems. The primary attack mode would be blocked on a system that uses HTTP authentication as some Distros set by default. Unfortunately, it is trivial to develop an XSS (Cross Site Scripting) attack that could indirectly access this vulnerability and allow an attacker to get access to any logged in system using all FreePBX authentication options. As such, all systems are vulnerable and should be updated. The fix addresses all attack modes.


    Please make sure to update your systems immediately. We informed project teams of the other popular FreePBX Distro’s earlier today as well as Certified Hosting Partners.

    The FreePBX and Schmooze Team!