High Availability Backup and Restore

In our never ending quest to make FreePBX the best phone system that doesn’t require money to buy (and even better than most that do cost money…), allow us to introduce you to the latest features in the Backup & Restore module. [url=/news/2010-05-09/preview-the-all-new-directory]Last week[/url] we alluded to a critical server error, but left you guessing as to how we address that. This is a big step in that direction.

Along with the kind assistance provided by some customers of [url=/support-and-professional-services]FreePBX Professional Support[/url] (in the form of time donated for a feature they needed for their business), the boys over at Schmoozecom (disclaimer: including yours truly) have once again surpassed all expectations. As of FreePBX 2.8, (currently in beta – more on that later) the Backup module includes built in support for High Availability clustering!

[img]/files/images/HA-backup.png[/img]

The following is an interview I had with Me (aka myself), elaborating on the new features:

[b]I: So, what is this new HA (High Availability) feature, and why are we so excited with it?[/b] [b]Me:[/b] To ensure that their critical infrastructure maintains a high level of reliability, many business seek out HA solutions of one type of another. To that end, we have introduced an automated capability for a designated server (herein: backup server) to periodically backup a production server (herein: primary server) and to restore said backup on the backup server. This ensures that the backup server always has a fresh copy of the primary server’s settings, and is ready to take over should the primary server fail.

[b]I: Billions of bilious blistering barnacles – that’s not true HA! True HA means the backups in real time and instant hand off between the two servers?![/b] [b]Me:[/b] Actually, according to ( wait for it – you’ll never guess!) Wikipedia, HA is [i]“is a system design … to ensures a certain degree of operational continuity…“.[/i] While there may be some benefit to having both the backups and hand-offs in real-time, such solutions can to be overly complex and way above the requirements of many organizations. Additionally, such solution tend to require a unique and complex setup with additional components not usually installed on FreePBX based distributions. Being OS agnostic, FreePBX is best not left to deal with the configuration and maintenance of such solutions. Instead, FreePBX takes a simplistic approach, removing much obscurity and confusion from the picture. Keep in mind, however, that FreePBX’s HA implementation can be used as part of a greater HA solution, perhaps including real-time fail-over support.

[b]I: So, how does FreePBX backup a server?[/b] [b]Me:[/b] The backup server can perform backups at any requested interval – ranging from every minute (not recommended for most scenarios) to multiple times per hour/day/week/month, etc. It then restores the backup to itself, wiping away any previous configurations, and replacing them with the latest setting from the production server.

Optionally, the backup server can be configured NOT to restore the setting locally, and act as a storage area for backing up MANY primary servers. In this configuration, the backup server keeps copies of backups handy, ready to be restored locally in the event of a failure to any of the primary servers. Please note that in this scenario, the backup server will continue to function as a backup server, even in the capacity of a surrogate to the failed production server.

[b]I: How does production move to the backup server?[/b] [b]Me:[/b] As mentioned previously, on its own – it doesn’t. The sysadmin is required to manually change the ip address of the server so that all peers can find it. This also allows the sysadmin more control of the switching over process, more notification to error’s and an incentive to ensure that the primary server doesn’t fail in the first place!

Additionally, there are services that can do the fail over automatically, but those are beyond the scope of this discussion. Apparently, these can be enabled in certain routers as well.

[b]I: Great! This is the perfect way to backup my CDR’s![/b] [b]Me:[/b] Actually, if you require real-time backups of cdr’s, your probably better off with a master-slave setup for your database (for the cdrs that is).

[b]I: After a backup is restored, what happens to my trunks – will they automatically try to register to my provider and ‘steal’ the incoming calls from the primary server?[/b] [b]Me:[/b] No. FreePBX requires that you click the orange bar at the top of the screen for any changed settings to go in to affect. This can be done manually or, if you’re using other elements as part of you HA solution, programmaticly by calling [code]/var/lib/asterisk/bin/module_admin reload[/code]. If you have traditional PSTN trunks (not VoIP), additional measures will have to be taken to connect those lines to your PBX and assure they are up and running.

[b]I: Bougainvillea! How do I backup the backup server?![/b] [b]Me:[/b] Exposed now in the GUI are many different options that can be run after a backup is complete. For example, copy the backup to an ftp server, ssh the backup to another host, or even have it emailed to you.

[b]I: Doesn’t the ability to pull a backup off a primary server pose a security risk?[/b] [b]Me:[/b] We have gone to great lengths to ensure that your phone system remains safe and secure throughout this process. The backups are all executed over ssh and encrypted with the public key.

[b]I: Public keys rock! (Uh, What are public keys?)[/b] [b]Me:[/b] Public/private keys are a method to encrypt data sent over insecure connections – easily considered one of the securest method in the world. See here for more info.

[b]I: Right, I knew that! But just to make sure, can you ‘remind’ me how to set up ssh keys?[/b] [b]Me:[/b] That’s probably something better left for the wiki/forums, but here is a very, very, very quick primer (designed for Cent OS systems, assuming asterisk is running as Linux user asterisk):
[code] sudo -u asterisk ssh-keygen
hit [enter] hit [enter] hit [enter] [/code] Then copy the public key to the primary server:
[code]sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub root@[/code] Your output should look something like this:
[code] [root@localhost /]# sudo -u asterisk ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/asterisk/.ssh/id_rsa):
Created directory ‘/var/lib/asterisk/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/asterisk/.ssh/id_rsa.
Your public key has been saved in /var/lib/asterisk/.ssh/id_rsa.pub.
The key fingerprint is:
ae:2d:fd:b7:19:d3:e8:34:8a:a9:7d:76:1c:71:c4:a8 asterisk@localhost.localdomain

[root@localhost /]# sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub root@myserver.example.com

The authenticity of host ‘myserver.example.com (21.158.66.3)’ can’t be established.
RSA key fingerprint is 8e:ae:6a:49:bb:1b:1b:91:3f:02:4f:65:ab:e7:5e:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘myserver.example.com,21.158.66.3’ (RSA) to the list of known hosts.
root@myserver.example.com’s password:
Now try logging into the machine, with “ssh ‘root@myserver.example.com'”, and check in:

.ssh/authorized_keys

to make sure we haven’t added extra keys that you weren’t expecting.
[root@localhost /]#

[/code] [b]I: Duh! Now, what do I need to do in the backup module?[/b] [b]Me:[/b] Under Remote Backup Options, put in the hostname or IP address of the primary server, the user name (probably root), and the private ssh key (if you followed the steps above, that should be: [code]/var/lib/asterisk/.ssh/id_rsa[/code]). To restore the backup immediately, check “Restore to this server”.
[url=/files/images/ha-backup-opts.png][img=235×75]/files/images/ha-backup-opts.png[/img][/url] [b]I: Great, I’m all set. One last thing – why don’t I see these options in my backup module?[/b] [b]Me:[/b] These options were included starting with FreePBX 2.8, currently in beta. Feel free to start beta testing 2.8! Have a look [url=/news/2010-05-24/freepbx-2-8-beta2]here[/url] for more info.

[b]I: Thank you so much for sharing your thoughts with me. Is there anything else you would like to add?[/b] [b]Me:[/b] Yes. As of [url=/news/2010-05-09/preview-the-all-new-directory]last week[/url] there were some issues threatening the continuity of Custom Contexts as of FreePBX 2.8. Congratulation and a tip of the hat to all those that step up to [url=/bounties/custom-context]contribute and help resolve[/url] the issue!

[b]Moshe Brevda[/b]

25 thoughts on “High Availability Backup and Restore

  1. I have a standby FreePBX server . The only difference between the primary and secondary is the i/p address .

    Is it possible to have some kind of exclude i/p address at ‘Remote Backup Options’ ?

    That way there would be less to change in the event of a switch over.

    The primary would never get a backup from the remote,

    And when the roles switch [which we do every month to ensure things are working ok] the backup/restore would continue to work.

    Do you know what I mean?

  2. Hello
    the improvements to the backup module are great! However on a fresh install a remote backup/restore failled due to missing /var/lib/asterisk/backups

    so I added this to our Debian FreePBX install script:

    # this is needed in case you want to o a remote backup/restore
    # if it is missing the remote backup/restore will fail as ot 2010-07-12
    mkdir -p /var/lib/asterisk/backups
    chown asterisk:asterisk /var/lib/asterisk/backups

    PS: is there a better place to put suggestions like this?

  3. in a ticket with a few more details.

    I assume what you are saying is that the backup module is installed on the remote system, but was never run.

    Assuming that is the case, it sounds like the backup module does not create the backup directory until it is run locally for the first time, and you are saying that resulted in it failing.

    Assuming my above understanding is true, you should file a bug indicating that the backup module needs to create the needed directory in its install script.

    But just one more question to check for clarification. Was that the only directory you needed to create on the remote system, or did you need to have a subdirectory with the backup set name as well?

    Please provide any of those relevant details in the bug report in the ticket system. Thanks.

  4. when i try to < sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub root@192.168.123.150 > I get Permission denied (publickey,gssapi-with-mic).
    But it works on other box’s .151 & .152 so there must be wrong permissions on box .150 ? not sure were to look or how to fix.

    Gary

  5. Seems like you’re box is configured not to use passwords. You will need to change that momentarily or copy the with some other method. There are a ton of guide out there, this really isn’t the place
  6. type=friend
    username=yournumber
    fromuser=yournumber
    secret=yourpassword
    host=2talk.co.nz
    context=default ; or your own selected context if desired
    dtmfmode=rfc2833
    disallow=all
    allow=ilbc
    allow=gsm
    allow=alaw
    allow=ulaw
    ;allow=g729 ; only if you have licenses to use it
    nat=yes
    canreinvite=no
    insecure=very ; use insecure=invite,port in later versions of Asterisk such as v1.4
  7. Has the option of changing the port for the ssh connection ever been addressed as was requested above, (not sure if he put in a feature request) this would be nice to have as I’m sure other people also try to hide there ssh port to the outside world.

    Gary.

  8. With FreePBX 2.8 the Remote Backup Option of the Backup & Restore module worked.
    I could make a change on the master server, and then run the backup job on the slave server, with the “Restore to this server” box checked. The change would appear on the slave.

    I did the update to 2.9.0rc1.1 along with all the other updates for 2.9 including Backup & Restore 2.9.0.2 now the process fails.

    The error in /var/log/httpd/error_log
    is scp: /backups/AsteriskP2-backup/10.0.19.27.20110406.15.44.52.tar.gz: No such file or directory. Expect for the updates nothing else was changed.

  9. i am trying to add the tftpboot folder using the additional files feature,
    i tried what jonstar mentioned previously using the quotes, but it still doesnt
    work.
    I want the tftpboot directory and subdirectories.
    so i used
    “/tftpboot/”
    and i also tried
    “/tftpboot”

    yet i got errors (from httpd/error_log)

    /bin/tar: /tftpboot/ : Cannot stat: No such file or directory
    /bin/tar: Error exit delayed from previous errors

    and the tared files do not include these additional files.
    any tips? how can I include this?

    thanks,
    -gk

  10. just in case someone else comes looking for error logging info, it can be found in the apache error.log on your system as it is generated by the FreePBX code running in php…. this is true for backups that you run yourself in realtime via the web interface… scheduled jobs are run as a cron not sure where they are logged to.

    The reason i was trying to look for errors was that i was attempting to backup multiple files under Additional Files -> Additional files and folders… I tried multiple methods of separating multiple files (one per line, comma separated, space separated w/ no quotes) and found that each of those methods only backs up the 1st file listed.

    So, it might be good to notate in the context help that the proper way to add “additional files” to be backed up; is on a single line, separated by spaces, and enclosed by quotes. Example below:

    “/var/log/asterisk/foo /var/log/asterisk/bar /var/log/asterisk/TEST”

  11. Love this add on. I have been using this on Trixbox and it works great. The only issues i am having is how do i keep the sip_nat.conf file from restoring to the backup box. I have tried adding it to the Exclude files list, but it keeps restoring the fie. Any thoughts?
  12. The remote backup option works great as long as your SSH is on port 22. However, I am running multiple instances of Asterisk / FreePBX on a sever as virtual machines under Proxmox (single public IP). I am trying to pull a backup to another server on a different IP. I have to port forward to each instance (ie. host SSH port is 65003 -> 22).

    I was able to move the RSA key using ssh-copy-id by specifying the host target as ‘-p 65003 root@myhost.net‘. But I can’t get the remote backup to work.

    Is there any way to force a port number in the “Remote SSH Hostname” field?

    Thanks,

    – Ted

  13. Hi,

    Just implemented this on a live/standby pair of servers. They are pure VoIP platforms and my SIP trunk provider can cope with a primary/secondary IP for the trunks. This means I can just turn off the live server, press the red bar on the standby server and within 30 seconds or so I have the calls going via the standby server.

    I am using Aastra phones so they also have the primary/secondary IPs defined and find the standby server quickly.

    A big thank you!

    Graham

  14. Hello All,

    Is it ok to leave the public/private key info in /root/.ssh which is the standard place in linux?
    Also when you are on the source server can you simply add the public key from the backup server to authorized_key2 ? ( also standard )

    Also, how can I tell what bombed when it doesn’t execute or for that matter how can I tell that it successfully executed?

    I say this because on a fresh install if you make it the backup and tell it to restore, then you will see configuration but what happens when it worked once back in Janurary and hasn’t backed up or restored since?

    Thanks!

Leave a Reply