Building a more secure communications platform

Network security is expected to be an almost $200 Billion dollar industry by the year 2020. In a world where everything is connected, securing everything can be big business. There are thousands of security researchers working daily to find the next big exploit. We have seen some huge exploits in the last few years such as “Heartbleed”, “Shellshock” and “Poodle” from exploited code that has been around for years.  

A blessing and sometimes a curse in open source software is that no matter how vigilant you are if you slip once someone will find it.  We’ve always taken security very seriously and have employed many approaches to ensuring FreePBX is secure.

FreePBX goes through continuous human and automated scanning looking for various attack vectors. From a human standpoint, we utilize internal developers who are passionate about security in both our software and the software they use. They do code reviews and code audits to ensure new code is up to par. We also work with independent security researchers who review our entire code base looking for things that may have been in the code for years.  We complement the human audits with automated tools including the RIPS scanner from ripstech.com.

RIPS, a static code analysis tool, does what would be impossible for a human to do. It looks at all 400,000+ lines of FreePBX code and does automated checks for Cross-Site Scripting, Code Execution, Command Execution and many other exploitable vectors. From that, it generates a report detailing potential vulnerabilities that may lie in our codebase. That seems like quite a lot, but it’s really only the start with RIPS which then details how to patch the vulnerability to minimize the risk moving forward. The reason we bring this up is because the RIPS utility has found many code issues that we may not have found in a manual review of the FreePBX code base and has helped us to strengthen the security of FreePBX.  

With these approaches, we aim to make your PBX secure so it’s one less issue you have to worry about.

“If you’re the smartest one in the room, you’re in the wrong room.” – Richard Tirendi

It is ultimately a battle of knowledge and someone out there is always smarter than you. This is why some vulnerabilities sit dormant for a decade (Such as Heartbleed). It took that long for someone to come along and see the code in a different way. When they ultimately release the exploit it often seems obvious.

We always welcome fresh eyes to review our code. Whether human or through machine automation we are happy to work with anyone who wants to make the world a more secure place.

Our policy on responsible reporting can be seen at http://wiki.freepbx.org/display/FOP/Security+Reporting and we appreciate all the security researchers that use their time to make the world more secure.

A special thanks to the https://www.ripstech.com team for analyzing our code and helping make FreePBX a more secure project.

Subscribe a BLF button to Monitor a Voicemail Box

I am going to try and start a weekly Saturday morning technical blog about some cool Tip or Trick you can do with FreePBX that most people are not aware of. As anyone who knows me over the years spelling and grammar are not by strong suite so please do not take offense if you find mistakes. I am here to try and provide valuable technical information.

If you have any request for things you would like me to go more indepth on please feel free to comment here and I will see if I can cover it for you in a future blog.

In this first series I am going to show you how you can setup a BLF button to monitor a voicemail box of any user on your FreePBX system. In the past this feature was not possible as Asterisk has no native ability to let a BLF button subscibe to a voicemail box and display when their are 1 or more waiting new voicemails. If you are using FreePBX 2.11 and the FreePBX Distro this feature is now possible.

A asterisk custom dev state hint for each voicemail box will now be auto created. If you log into the Asterisk CLI and issue the following command you will see the hint for each voicemail box called xxx@dialvm

core show hints

In our example above voicemail box 101 has a hint created that you can program any BLF button to as *98101. Now anytime voicemail box 101 has one or more new messages waiting the BLF light will be activated on your phone. Pressing the BLF button will prompt you to login with the password for voicemail box 101.

 

Now you can finally setup that general voicemail box and add a button to any phone that needs to be notified when a new voicemail is left. Thats all thats to it. Hope you find this feature useful.

Also remember to review our new wiki for more indepth information on FreePBX and all things related to the FreePBX Product.

The related wiki post on this feature can be found here.

Version 2.9 – amportal.conf – Advanced Settings – I’m Confused

Some of us have been using FreePBX for a very long time and are very used to going to [i]/etc/amportal.conf[/i] to make certain changes. Others rely on the powers of google to quickly find help on how to do things which means they benefit from our collective wisdom, which all points to instructions that say [i]”Go to /etc/amportal.conf and change XYZ to abc.”[/i] So why is it when you do that on version 2.9 it does not work and how does all that relate to these new [i]Advanced Settings[/i]? Or if you picked up on Advanced Settings, [i]”why won’t it let me change them?”[/i]

Haven’t gotten to 2.9 yet and wondering what I’m talking about? Here’s [url=/news/2011-02-23/2-9-version-upgrade-module-beta-program-status-more][b]FreePBX Version Upgrade Module Information[/b][/url] from last week.

Since I have made at least a dozen or more comments in the forums and tickets on amportal.conf vs. Advanced Settings often related to this confusion, it’s time to try and clarify what is going on. Historically FreePBX had many many Advanced Settings that had to be configured in amportal.conf. For many users this meant going out of their comfort zone requiring command line access and editors like [i]nano[/i] or [i]vim[/i] or sometimes worse, it would involve blindly following some instructions that turned up on google telling them to run some cryptic [i]sed[/i] command that did some magic resulting in their desired behavior. The problem with all of this was two fold. It’s very error prone and it’s hard and potentially dangerous to do for most users.

Some of the more common settings that you may have heard of include :

[list] [*][b]AMPMGRUSER[/b] & [b]AMPMGRPASS[/b] – sets the default AMI user/password that FreePBX requires and must also be changed in manager.conf
[*][b]AMPEXTENSIONS[/b] – set to either [i]extensions[/i] or [i]deviceanduser[/i] to control whether in Extension or Device and User mode
[*][b]USEDEVSTATE[/b] – whether or not to take advantage of DEVICE_STATE() and generate lots of hints for several feature codes to be used with BLF buttons on phones
[/list] and many many many more… So, what changed in 2.9? Why did it change? Why is there still an amportal.conf file? And other questions…

[b]Why did it change?[/b]

The biggest reason for the change was to make things easier for you, the end user, to be aware of the available settings and help control and police the settings so as to try and catch errors, avoid typos and in general present them inside of a GUI consistent with the rest of your configuration experience. The new Advanced Settings page and framework also forces us to document the settings so that you can get some idea what they do and whether or not you should use or change them.

[b]Why is there still an amportal.conf file?[/b]

The amportal.conf file had three main purposes prior to version 2.9:

[list] [*] It provided the “bootstrap” information that FreePBX used to get started, e.g. AMPDBNAME, AMPDBUSER, AMPDBPASS, AMPDBHOST and AMPDBENGINE (and optionally datasource).
[*] it provided a place to define many advanced settings that ultimately are made available inside of the FreePBX code base as $amp_conf[‘VARNAME’] [*] As a place for additional settings, it is used by other applications in the FreePBX eco system (example, Flash Operator Panel) and probably others that we have no awareness of.
[/list]

When migrating to version 2.9, we have defined a new and very minimalistic configuration file which is typically located in /etc/freepbx.conf and which provides the bare essentials that are needed to [i]bootstrap[/i] FreePBX now. We’ll talk about all the bootstrap changes at some other time but for now, if you are wondering why the above mentioned configuration variable can not be found on the Advanced Settings page, it’s because they are in [i]freepbx.conf[/i].

The remaining settings have been migrated to the internal database and should all be available from the GUI page.

So if they’ve all been migrated, how come there’s still an /etc/amortal.conf file on your system? Well the third bullet point, those third part applications, is the main reason we still have an amportal.conf. It works as follows, once you have [i]fully migrated[/i], we will start auto-generating an amportal.conf file every time you press the [i]Apply Configuration Changes[/i] bar and all the other configuration files are generated. This makes sure that those third party applications, AGI scripts, etc. continue to have the file they are used to having with all of the updated configuration settings. We expect those applications to migrate but understand that it will not happen overnight. This is simply our continued attempt to do everything reasonable within our power to help maintain compatibility while at the same time improving and evolving.

We do use [i]amportal.conf[/i] in one other place. On initial install, with the use of [i]install_amp[/i] that you may be doing or your distro may be doing for you. The install_amp script will use the specified settings when you first install, to get you started and even to make [i]freepbx.conf[/i] for you if not present.

[b]What does [i]fully migrated[/i] mean and/or Why can’t you edit your settings?[/b]

The migration process for systems being upgrade is a bit tricky this time around. The amortal.conf file has never belonged to the [i]asterisk[/i] user which is usually the same as the apache user; it has typically belonged to root. As such, we don’t always have the control necessary to make changes to amportal.conf when upgrading through the GUI since the permissions are limited to the web user permissions and thus they can’t usually write to amportal.conf. This presents a dilemma because if we can’t write to amportal.conf then it means we can’t update it with new settings if you go and change those settings. So what do we do and how does it effect you?

The simple answer is, until we can obtain write access to the amportal.conf file, we will not let you change anything on the Advanced Settings page. We’ll also try hard to let you know of the issue including Notifications sent to the panel and log messages. Inside of FreePBX we continue to operate much like we always did. This means we will read amportal.conf and update the internal database with any changes that may have been made to that file so we are always consistent. This means you are running in a [i]crippled mode[/i] and thus you are not [i]fully migrated.[/i]

To finish off the migration process, we need to get the permissions changed on the /etc/amportal.conf file. For some of you, it may be no big deal, you’ll just go and make the file permission or ownership change in Linux and be on your way. For the most though, this is again outside of your comfort zone. That’s where another command comes in that many may be familiar with, the [i]amportal[/i] command. (Which is actually the [i]freepbx_engine[/i] command but most people don’t realize that). This is the command that is used to start and stop Asterisk amongst many other things, and it is almost always used (and should be used) by most distros to start Asterisk when you first boot up. As luck has it, this command must be run by root and when you boot your system it is run by root. This means that eventually, and maybe mysteriously, that file permission issue will go away and your migration will be finished and suddenly you will have access to Advanced Settings again.

Of course one challenge here is three fold. FreePBX easily upgrades without requiring either Asterisk or Linux to be rebooted. Linux is rock solid (not like some other operating systems many of us know about) and thus often has up times in months if not years, and Asterisk has come a long way, meaning it often has uptimes in months at least. Furthermore, if you type [i]core restart now[/i] from within Asterisk, it will not run [i]amportal[/i]. Therefore, if you want to get a jump on things, from the Linux Command line, you can type [i]amportal chown[/i] or any other valid amportal comand because each time run, it will check on this permission issue and should rectify the problem for you.

[b]There are settings I want to change but there’s no option to change them?[/b]

There are a lot of very volatile settings available and as a further step to try to protect user’s against fatal and costly errors we have made many of the more volatile settings [i]read only[/i] which means you need to take extra measures to change them. I’ll cover this by describing the 4 available options at the top of the Advanced Settings page, the [i]Advanced Settings Details[/i] category.

[b][i]Display Friendly Name[/i][/b]

We have [i]tried[/i] to use less cryptic names for the settings vs. the internal variable names that many of us may have gotten used to over the years. You can still see what these map to along with a lot of other useful information by hovering over the label and having a look at the tool tip. If you prefer to simply have all the cryptic names listed (which can be very useful for a developer) you can set this value to false.

[b][i]Display Hidden Settings[/i][/b]

This will not be of interest to most people. FreePBX has a number of settings that it uses internally and are never meant to be set or changed by the user. If you want to see what these are, you can change this value to true. You will not be given an opportunity to edit these regardless of the other settings. Only a FreePBX module can do this.

[b][i]Display Readonly Settings[/i][/b]

As mentioned, there are many settings that are quite volatile and by default we don’t even show these settings. If you want them displayed you can set this to true. You will not be able to edit them without changing the value of the last option.

[b][i]Override Readonly Settings[/i][/b]

This is the [i]dangerous[/i] one. When set to true, it ignores the status of all the read only settings and allows you to change them. You should be [b]VERY[/b] careful when doing this because many of these settings could cripple or crash your system requiring some pretty advanced knowledge to get it back working.

[b]Where’s the Submit button, I only have a Refresh Page button?[/b]

Unlike most FreePBX pages, there is no Submit button. Each setting is changed by clicking on the little green checkbox to the right of the setting that will appear when you make a change. Each setting must be individually changed if you change more than a single setting. If these are not coming up for you, assuming you are [i]fully migrated[/i] then check your bowser. We are currently aware of an IE8 issue which is not fixed at the time of this writing. If there are other issues, seek help in the forums and/or once you are pretty sure it’s a bug, file a report in the tracker and please provide lots of details such as browsers and other so we can reproduce the problem.

So there you have it! In addition to the roughly 100 or settings that existed prior to version 2.9, you can now see the MANY settings available since then (at last check, somewhere close to 200 in all).

[b]Philippe[/b] – On behalf of the FreePBX Team!

High Availability Backup and Restore

In our never ending quest to make FreePBX the best phone system that doesn’t require money to buy (and even better than most that do cost money…), allow us to introduce you to the latest features in the Backup & Restore module. [url=/news/2010-05-09/preview-the-all-new-directory]Last week[/url] we alluded to a critical server error, but left you guessing as to how we address that. This is a big step in that direction.

Along with the kind assistance provided by some customers of [url=/support-and-professional-services]FreePBX Professional Support[/url] (in the form of time donated for a feature they needed for their business), the boys over at Schmoozecom (disclaimer: including yours truly) have once again surpassed all expectations. As of FreePBX 2.8, (currently in beta – more on that later) the Backup module includes built in support for High Availability clustering!

[img]/files/images/HA-backup.png[/img]

The following is an interview I had with Me (aka myself), elaborating on the new features:

[b]I: So, what is this new HA (High Availability) feature, and why are we so excited with it?[/b] [b]Me:[/b] To ensure that their critical infrastructure maintains a high level of reliability, many business seek out HA solutions of one type of another. To that end, we have introduced an automated capability for a designated server (herein: backup server) to periodically backup a production server (herein: primary server) and to restore said backup on the backup server. This ensures that the backup server always has a fresh copy of the primary server’s settings, and is ready to take over should the primary server fail.

[b]I: Billions of bilious blistering barnacles – that’s not true HA! True HA means the backups in real time and instant hand off between the two servers?![/b] [b]Me:[/b] Actually, according to ( wait for it – you’ll never guess!) Wikipedia, HA is [i]“is a system design … to ensures a certain degree of operational continuity…“.[/i] While there may be some benefit to having both the backups and hand-offs in real-time, such solutions can to be overly complex and way above the requirements of many organizations. Additionally, such solution tend to require a unique and complex setup with additional components not usually installed on FreePBX based distributions. Being OS agnostic, FreePBX is best not left to deal with the configuration and maintenance of such solutions. Instead, FreePBX takes a simplistic approach, removing much obscurity and confusion from the picture. Keep in mind, however, that FreePBX’s HA implementation can be used as part of a greater HA solution, perhaps including real-time fail-over support.

[b]I: So, how does FreePBX backup a server?[/b] [b]Me:[/b] The backup server can perform backups at any requested interval – ranging from every minute (not recommended for most scenarios) to multiple times per hour/day/week/month, etc. It then restores the backup to itself, wiping away any previous configurations, and replacing them with the latest setting from the production server.

Optionally, the backup server can be configured NOT to restore the setting locally, and act as a storage area for backing up MANY primary servers. In this configuration, the backup server keeps copies of backups handy, ready to be restored locally in the event of a failure to any of the primary servers. Please note that in this scenario, the backup server will continue to function as a backup server, even in the capacity of a surrogate to the failed production server.

[b]I: How does production move to the backup server?[/b] [b]Me:[/b] As mentioned previously, on its own – it doesn’t. The sysadmin is required to manually change the ip address of the server so that all peers can find it. This also allows the sysadmin more control of the switching over process, more notification to error’s and an incentive to ensure that the primary server doesn’t fail in the first place!

Additionally, there are services that can do the fail over automatically, but those are beyond the scope of this discussion. Apparently, these can be enabled in certain routers as well.

[b]I: Great! This is the perfect way to backup my CDR’s![/b] [b]Me:[/b] Actually, if you require real-time backups of cdr’s, your probably better off with a master-slave setup for your database (for the cdrs that is).

[b]I: After a backup is restored, what happens to my trunks – will they automatically try to register to my provider and ‘steal’ the incoming calls from the primary server?[/b] [b]Me:[/b] No. FreePBX requires that you click the orange bar at the top of the screen for any changed settings to go in to affect. This can be done manually or, if you’re using other elements as part of you HA solution, programmaticly by calling [code]/var/lib/asterisk/bin/module_admin reload[/code]. If you have traditional PSTN trunks (not VoIP), additional measures will have to be taken to connect those lines to your PBX and assure they are up and running.

[b]I: Bougainvillea! How do I backup the backup server?![/b] [b]Me:[/b] Exposed now in the GUI are many different options that can be run after a backup is complete. For example, copy the backup to an ftp server, ssh the backup to another host, or even have it emailed to you.

[b]I: Doesn’t the ability to pull a backup off a primary server pose a security risk?[/b] [b]Me:[/b] We have gone to great lengths to ensure that your phone system remains safe and secure throughout this process. The backups are all executed over ssh and encrypted with the public key.

[b]I: Public keys rock! (Uh, What are public keys?)[/b] [b]Me:[/b] Public/private keys are a method to encrypt data sent over insecure connections – easily considered one of the securest method in the world. See here for more info.

[b]I: Right, I knew that! But just to make sure, can you ‘remind’ me how to set up ssh keys?[/b] [b]Me:[/b] That’s probably something better left for the wiki/forums, but here is a very, very, very quick primer (designed for Cent OS systems, assuming asterisk is running as Linux user asterisk):
[code] sudo -u asterisk ssh-keygen
hit [enter] hit [enter] hit [enter] [/code] Then copy the public key to the primary server:
[code]sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub root@[/code] Your output should look something like this:
[code] [root@localhost /]# sudo -u asterisk ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/asterisk/.ssh/id_rsa):
Created directory ‘/var/lib/asterisk/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/asterisk/.ssh/id_rsa.
Your public key has been saved in /var/lib/asterisk/.ssh/id_rsa.pub.
The key fingerprint is:
ae:2d:fd:b7:19:d3:e8:34:8a:a9:7d:76:1c:71:c4:a8 asterisk@localhost.localdomain

[root@localhost /]# sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub root@myserver.example.com

The authenticity of host ‘myserver.example.com (21.158.66.3)’ can’t be established.
RSA key fingerprint is 8e:ae:6a:49:bb:1b:1b:91:3f:02:4f:65:ab:e7:5e:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘myserver.example.com,21.158.66.3’ (RSA) to the list of known hosts.
root@myserver.example.com’s password:
Now try logging into the machine, with “ssh ‘root@myserver.example.com'”, and check in:

.ssh/authorized_keys

to make sure we haven’t added extra keys that you weren’t expecting.
[root@localhost /]#

[/code] [b]I: Duh! Now, what do I need to do in the backup module?[/b] [b]Me:[/b] Under Remote Backup Options, put in the hostname or IP address of the primary server, the user name (probably root), and the private ssh key (if you followed the steps above, that should be: [code]/var/lib/asterisk/.ssh/id_rsa[/code]). To restore the backup immediately, check “Restore to this server”.
[url=/files/images/ha-backup-opts.png][img=235×75]/files/images/ha-backup-opts.png[/img][/url] [b]I: Great, I’m all set. One last thing – why don’t I see these options in my backup module?[/b] [b]Me:[/b] These options were included starting with FreePBX 2.8, currently in beta. Feel free to start beta testing 2.8! Have a look [url=/news/2010-05-24/freepbx-2-8-beta2]here[/url] for more info.

[b]I: Thank you so much for sharing your thoughts with me. Is there anything else you would like to add?[/b] [b]Me:[/b] Yes. As of [url=/news/2010-05-09/preview-the-all-new-directory]last week[/url] there were some issues threatening the continuity of Custom Contexts as of FreePBX 2.8. Congratulation and a tip of the hat to all those that step up to [url=/bounties/custom-context]contribute and help resolve[/url] the issue!

[b]Moshe Brevda[/b]

Heavy Queue Usage in FreePBX

When it comes to busy queues and FreePBX, it often results in requests for help in the forums and to our support department because of overloaded systems or certain functions that do not seem work right. FreePBX was designed to be an SMB PBX concentrating on feature rich capabilities at the expense of complex dialplan and AGI scripts, all of which take their toll when you setup a reasonable size call center and start throwing lots of simultaneous calls at a group of queue members. Continue reading