High Availability Backup and Restore
In our never ending quest to make FreePBX the best phone system that doesn't require money to buy (and even better than most that do cost money...), allow us to introduce you to the latest features in the Backup & Restore module. Last week we alluded to a critical server error, but left you guessing as to how we address that. This is a big step in that direction.
Along with the kind assistance provided by some customers of FreePBX Professional Support (in the form of time donated for a feature they needed for their business), the boys over at Schmoozecom (disclaimer: including yours truly) have once again surpassed all expectations. As of FreePBX 2.8, (currently in beta - more on that later) the Backup module includes built in support for High Availability clustering!

The following is an interview I had with Me (aka myself), elaborating on the new features:
I: So, what is this new HA (High Availability) feature, and why are we so excited with it?
Me: To ensure that their critical infrastructure maintains a high level of reliability, many business seek out HA solutions of one type of another. To that end, we have introduced an automated capability for a designated server (herein: backup server) to periodically backup a production server (herein: primary server) and to restore said backup on the backup server. This ensures that the backup server always has a fresh copy of the primary server’s settings, and is ready to take over should the primary server fail.
I: Billions of bilious blistering barnacles - that's not true HA! True HA means the backups in real time and instant hand off between the two servers?!
Me: Actually, according to ( wait for it - you’ll never guess!) Wikipedia, HA is “is a system design ... to ensures a certain degree of operational continuity...“. While there may be some benefit to having both the backups and hand-offs in real-time, such solutions can to be overly complex and way above the requirements of many organizations. Additionally, such solution tend to require a unique and complex setup with additional components not usually installed on FreePBX based distributions. Being OS agnostic, FreePBX is best not left to deal with the configuration and maintenance of such solutions. Instead, FreePBX takes a simplistic approach, removing much obscurity and confusion from the picture. Keep in mind, however, that FreePBX’s HA implementation can be used as part of a greater HA solution, perhaps including real-time fail-over support.
I: So, how does FreePBX backup a server?
Me: The backup server can perform backups at any requested interval - ranging from every minute (not recommended for most scenarios) to multiple times per hour/day/week/month, etc. It then restores the backup to itself, wiping away any previous configurations, and replacing them with the latest setting from the production server.
Optionally, the backup server can be configured NOT to restore the setting locally, and act as a storage area for backing up MANY primary servers. In this configuration, the backup server keeps copies of backups handy, ready to be restored locally in the event of a failure to any of the primary servers. Please note that in this scenario, the backup server will continue to function as a backup server, even in the capacity of a surrogate to the failed production server.
I: How does production move to the backup server?
Me: As mentioned previously, on its own - it doesn't. The sysadmin is required to manually change the ip address of the server so that all peers can find it. This also allows the sysadmin more control of the switching over process, more notification to error’s and an incentive to ensure that the primary server doesn't fail in the first place!
Additionally, there are services that can do the fail over automatically, but those are beyond the scope of this discussion. Apparently, these can be enabled in certain routers as well.
I: Great! This is the perfect way to backup my CDR’s!
Me: Actually, if you require real-time backups of cdr’s, your probably better off with a master-slave setup for your database (for the cdrs that is).
I: After a backup is restored, what happens to my trunks - will they automatically try to register to my provider and ‘steal’ the incoming calls from the primary server?
Me: No. FreePBX requires that you click the orange bar at the top of the screen for any changed settings to go in to affect. This can be done manually or, if you’re using other elements as part of you HA solution, programmaticly by calling /var/lib/asterisk/bin/module_admin reload. If you have traditional PSTN trunks (not VoIP), additional measures will have to be taken to connect those lines to your PBX and assure they are up and running.
I: Bougainvillea! How do I backup the backup server?!
Me: Exposed now in the GUI are many different options that can be run after a backup is complete. For example, copy the backup to an ftp server, ssh the backup to another host, or even have it emailed to you.
I: Doesn't the ability to pull a backup off a primary server pose a security risk?
Me: We have gone to great lengths to ensure that your phone system remains safe and secure throughout this process. The backups are all executed over ssh and encrypted with the public key.
I: Public keys rock! (Uh, What are public keys?)
Me: Public/private keys are a method to encrypt data sent over insecure connections - easily considered one of the securest method in the world. See here for more info.
I: Right, I knew that! But just to make sure, can you ‘remind’ me how to set up ssh keys?
Me: That's probably something better left for the wiki/forums, but here is a very, very, very quick primer (designed for Cent OS systems, assuming asterisk is running as Linux user asterisk):
sudo -u asterisk ssh-keygen hit [enter] hit [enter] hit [enter]
Then copy the public key to the primary server:
sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub root@
Your output should look something like this:
[root@localhost /]# sudo -u asterisk ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/var/lib/asterisk/.ssh/id_rsa): Created directory '/var/lib/asterisk/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /var/lib/asterisk/.ssh/id_rsa. Your public key has been saved in /var/lib/asterisk/.ssh/id_rsa.pub. The key fingerprint is: ae:2d:fd:b7:19:d3:e8:34:8a:a9:7d:76:1c:71:c4:a8 [root@localhost /]# sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub The authenticity of host 'myserver.example.com (21.158.66.3)' can't be established. RSA key fingerprint is 8e:ae:6a:49:bb:1b:1b:91:3f:02:4f:65:ab:e7:5e:6b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'myserver.example.com,21.158.66.3' (RSA) to the list of known hosts. 's password: Now try logging into the machine, with "ssh 'root@myserver.example.com'", and check in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. [root@localhost /]#
I: Duh! Now, what do I need to do in the backup module?
Me: Under Remote Backup Options, put in the hostname or IP address of the primary server, the user name (probably root), and the private ssh key (if you followed the steps above, that should be: /var/lib/asterisk/.ssh/id_rsa). To restore the backup immediately, check “Restore to this server”.

I: Great, I’m all set. One last thing - why don't I see these options in my backup module?
Me: These options were included starting with FreePBX 2.8, currently in beta. Feel free to start beta testing 2.8! Have a look here for more info.
I: Thank you so much for sharing your thoughts with me. Is there anything else you would like to add?
Me: Yes. As of last week there were some issues threatening the continuity of Custom Contexts as of FreePBX 2.8. Congratulation and a tip of the hat to all those that step up to contribute and help resolve the issue!
Moshe Brevda
Comments
robfantini
Thu, 07/01/2010 - 11:55
Permalink
thank you
works great!
mbrevda
Thu, 07/01/2010 - 11:56
Permalink
Your welcome!
Your welcome!
Moshe Brevda
robfantini
Sat, 07/10/2010 - 08:06
Permalink
suggestion
I have a standby FreePBX server . The only difference between the primary and secondary is the i/p address .
Is it possible to have some kind of exclude i/p address at 'Remote Backup Options' ?
That way there would be less to change in the event of a switch over.
The primary would never get a backup from the remote,
And when the roles switch [which we do every month to ensure things are working ok] the backup/restore would continue to work.
Do you know what I mean?
robfantini
Mon, 07/12/2010 - 10:53
Permalink
on new install I had to
Hello
the improvements to the backup module are great! However on a fresh install a remote backup/restore failled due to missing /var/lib/asterisk/backups
so I added this to our Debian FreePBX install script:
# this is needed in case you want to o a remote backup/restore
# if it is missing the remote backup/restore will fail as ot 2010-07-12
mkdir -p /var/lib/asterisk/backups
chown asterisk:asterisk /var/lib/asterisk/backups
PS: is there a better place to put suggestions like this?
p_lindheimer
Mon, 07/12/2010 - 11:07
Permalink
in a ticket with a few more
in a ticket with a few more details.
I assume what you are saying is that the backup module is installed on the remote system, but was never run.
Assuming that is the case, it sounds like the backup module does not create the backup directory until it is run locally for the first time, and you are saying that resulted in it failing.
Assuming my above understanding is true, you should file a bug indicating that the backup module needs to create the needed directory in its install script.
But just one more question to check for clarification. Was that the only directory you needed to create on the remote system, or did you need to have a subdirectory with the backup set name as well?
Please provide any of those relevant details in the bug report in the ticket system. Thanks.
Philippe Lindheimer - FreePBX Project Leader
FreePBX Training Opportunities - Click Here
Get Official Paid Support - Click Here
ciscostudent561
Fri, 07/16/2010 - 13:16
Permalink
is there any chance at all
is there any chance at all to automate the fail over process? Thanks for this documentation! ITS GREEAAAT, saved my life
ou812
Wed, 07/21/2010 - 14:19
Permalink
when i try to < sudo -u
when i try to < sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub
> I get Permission denied (publickey,gssapi-with-mic).
But it works on other box's .151 & .152 so there must be wrong permissions on box .150 ? not sure were to look or how to fix.
Gary
mbrevda
Wed, 07/21/2010 - 14:24
Permalink
Seems like you're box is
Seems like you're box is configured not to use passwords. You will need to change that momentarily or copy the with some other method. There are a ton of guide out there, this really isn't the place
Moshe Brevda
ou812
Wed, 07/21/2010 - 15:09
Permalink
Thanks for pointing me in
Thanks for pointing me in the right place.
Gary.
pdc.corbett
Thu, 08/19/2010 - 21:18
Permalink
corbett
type=friend
username=yournumber
fromuser=yournumber
secret=yourpassword
host=2talk.co.nz
context=default ; or your own selected context if desired
dtmfmode=rfc2833
disallow=all
allow=ilbc
allow=gsm
allow=alaw
allow=ulaw
;allow=g729 ; only if you have licenses to use it
nat=yes
canreinvite=no
insecure=very ; use insecure=invite,port in later versions of Asterisk such as v1.4
brendanh
Mon, 08/30/2010 - 12:58
Permalink
This is an awesome feature
This is way easier than me having to setup a MySQL cluster...
Just to note: I believe this feature was added in 2.8
GWalmsley
Tue, 09/28/2010 - 08:21
Permalink
Very neat
Hi,
Just implemented this on a live/standby pair of servers. They are pure VoIP platforms and my SIP trunk provider can cope with a primary/secondary IP for the trunks. This means I can just turn off the live server, press the red bar on the standby server and within 30 seconds or so I have the calls going via the standby server.
I am using Aastra phones so they also have the primary/secondary IPs defined and find the standby server quickly.
A big thank you!
Graham
tededwards
Sun, 10/24/2010 - 11:40
Permalink
Help with SSH host port
The remote backup option works great as long as your SSH is on port 22. However, I am running multiple instances of Asterisk / FreePBX on a sever as virtual machines under Proxmox (single public IP). I am trying to pull a backup to another server on a different IP. I have to port forward to each instance (ie. host SSH port is 65003 -> 22).
I was able to move the RSA key using ssh-copy-id by specifying the host target as '-p 65003
'. But I can't get the remote backup to work.
Is there any way to force a port number in the "Remote SSH Hostname" field?
Thanks,
- Ted
mbrevda
Sun, 10/24/2010 - 11:42
Permalink
Hmm, interesting oversite.
Hmm, interesting oversite. You may want to open a feature request for that.
Moshe Brevda
digitalb
Thu, 11/04/2010 - 07:25
Permalink
Great Addon
Love this add on. I have been using this on Trixbox and it works great. The only issues i am having is how do i keep the sip_nat.conf file from restoring to the backup box. I have tried adding it to the Exclude files list, but it keeps restoring the fie. Any thoughts?
mbrevda
Thu, 11/04/2010 - 07:30
Permalink
Probably no way to do this
Probably no way to do this currently
Moshe Brevda
digitalb
Thu, 11/04/2010 - 10:54
Permalink
Thank you
Ok Thank you for your reply. I will keep playing with it and will let you know if i ever figure it out.
jonstar
Mon, 12/06/2010 - 11:49
Permalink
where are failures logged to
where are failures logged to ? I am attempting to get this running with no luck and no feedback from application.
Thanks,
Jon
jonstar
Mon, 03/07/2011 - 23:10
Permalink
just in case someone else
just in case someone else comes looking for error logging info, it can be found in the apache error.log on your system as it is generated by the FreePBX code running in php.... this is true for backups that you run yourself in realtime via the web interface... scheduled jobs are run as a cron not sure where they are logged to.
The reason i was trying to look for errors was that i was attempting to backup multiple files under Additional Files -> Additional files and folders... I tried multiple methods of separating multiple files (one per line, comma separated, space separated w/ no quotes) and found that each of those methods only backs up the 1st file listed.
So, it might be good to notate in the context help that the proper way to add "additional files" to be backed up; is on a single line, separated by spaces, and enclosed by quotes. Example below:
"/var/log/asterisk/foo /var/log/asterisk/bar /var/log/asterisk/TEST"
gkrawiec
Fri, 04/01/2011 - 14:09
Permalink
i am trying to add the
i am trying to add the tftpboot folder using the additional files feature,
i tried what jonstar mentioned previously using the quotes, but it still doesnt
work.
I want the tftpboot directory and subdirectories.
so i used
"/tftpboot/"
and i also tried
"/tftpboot"
yet i got errors (from httpd/error_log)
/bin/tar: /tftpboot/ : Cannot stat: No such file or directory
/bin/tar: Error exit delayed from previous errors
and the tared files do not include these additional files.
any tips? how can I include this?
thanks,
-gk
A-Roby
Wed, 04/06/2011 - 15:21
Permalink
not working in 2.9.0rc1.1?
With FreePBX 2.8 the Remote Backup Option of the Backup & Restore module worked.
I could make a change on the master server, and then run the backup job on the slave server, with the “Restore to this server” box checked. The change would appear on the slave.
I did the update to 2.9.0rc1.1 along with all the other updates for 2.9 including Backup & Restore 2.9.0.2 now the process fails.
The error in /var/log/httpd/error_log
is scp: /backups/AsteriskP2-backup/10.0.19.27.20110406.15.44.52.tar.gz: No such file or directory. Expect for the updates nothing else was changed.
p_lindheimer
Wed, 04/06/2011 - 15:34
Permalink
best not to post the same
best not to post the same thing multiple places,
there is an open issue on this, #5021
Philippe Lindheimer - FreePBX Project Leader
FreePBX Training Opportunities - Click Here
Get Official Paid Support - Click Here
ou812
Tue, 05/10/2011 - 08:11
Permalink
Has the option of changing
Has the option of changing the port for the ssh connection ever been addressed as was requested above, (not sure if he put in a feature request) this would be nice to have as I'm sure other people also try to hide there ssh port to the outside world.
Gary.
mbrevda
Tue, 05/10/2011 - 08:21
Permalink
I havnt seen a feature
I havnt seen a feature request for that
Moshe Brevda
VoIPTek
Sun, 06/12/2011 - 11:10
Permalink
Details
Hello All,
Is it ok to leave the public/private key info in /root/.ssh which is the standard place in linux?
Also when you are on the source server can you simply add the public key from the backup server to authorized_key2 ? ( also standard )
Also, how can I tell what bombed when it doesn't execute or for that matter how can I tell that it successfully executed?
I say this because on a fresh install if you make it the backup and tell it to restore, then you will see configuration but what happens when it worked once back in Janurary and hasn't backed up or restored since?
Thanks!